Definition of Robots/Robots.txt

The Robots.txt protocol is also known by either the robots exclusion standard or the Robots Exclusion Protocol. It is designed to prevent cooperating or honorable spiders from gaining entrance or access to every part of a website that is partially viewable by the general public. A robot.txt file that is placed on a specific website will put forth a request that clearly delineates which files or directories specified robots should ignore during their search of the site.

It is necessary to set up a separate robots.txt protocol for each sub-domain of a website. Unfortunately, nothing dictates that the spiders or robots must obey the suggestion to ignore certain areas of a website. The robots.txt can clearly indicate which directories and files should be ignored either out of a sense of wanting to keep the information private or the desire to have certain information unavailable since it might be misleading in context with the website. However, nothing dictates that this convention will receive cooperation.

External links:

• Wikipedia explanation of Robots.txt
• Introduction to creating a robots.txt file
• Web Robot Pages
• Robots.txt generator
• Workshoop explanation