- Aug 22, 2004
- 107
- 0
- 0
Does anyone know if there is anything like Robotcop for Tomcat? For those who are not familiar with Robotcop, you can read about it here:
http://www.robotcop.org/install_ap13.html
In a nutshell, it extends robots.txt in that it will actually enforce the rules set forth in there, as opposed to doing nothing and hoping the various web spiders/robots/crawlers respect them. I've searched all over google for something like this for Tomcat with no luck.
http://www.robotcop.org/install_ap13.html
In a nutshell, it extends robots.txt in that it will actually enforce the rules set forth in there, as opposed to doing nothing and hoping the various web spiders/robots/crawlers respect them. I've searched all over google for something like this for Tomcat with no luck.