I love Python. I really love Python. I’m working on a little project that should really obey robots.txt. I hit up google to see if there was any code out there for me to take a look at or if I was going to have to write a little code to handle robots.txt. Of course, one of the first things that I stumbled upon was robotparser, a module that does just that.
Instead of reinventing the wheel, all I have to do is import robotparser and use the code that has already been written for me. Python’s Standard Library absolutely rules.