standard for robot exclusion
Technology Dictionary
-> standard for robot exclusion
Search:
standard for robot exclusion
A proposal to try to prevent the havoc wreaked by many of the early World-Wide Web robots when they retrieved documents too rapidly or retrieved documents that had side effects (such as voting). The proposed standard for robot exclusion offers a solution to these problems.
(http://info.webcrawler.com/mak/projects/robots/norobots.html).
(1996-03-23)
©
Art Branch Inc.
SQL Tutorial