Protego is a pure-Python `robots.txt` parser with support for modern conventions.

WWW: https://pypi.python.org/pypi/Protego
