[Article] ANALYZING ONE MILLION ROBOTS.TXT FILES
Sven
www.GSA-Online.de
A very interesting article on this very basic thing:
https://intoli.com/blog/analyzing-one-million-robots-txt-files/
https://intoli.com/blog/analyzing-one-million-robots-txt-files/