Definition of Crawl

In web usage, the term crawl refers to the process of gathering data by search engines. It also refers to the process of visiting a large number of websites at a single sitting through the process of clicking many hyperlinks. This term can be used to refer to the searching of the Web for specific webpages, blogs, articles, pictures, hosts, or other forms of information.

A web crawler is also referred to as a spider, web spider, web robot, bot, ant, automatic indexer, or robot. It is a program that is used by search engines to look for information on the Web. More specifically, web crawlers are programs that browse through the World Wise Web in a slow, methodical manner to access data. Web crawlers can also be utilized to complete automated maintenance tasks for websites. This maintenance can include validating HTML code or checking the validity of links.

External links:

• Wikipedia explanation of Crawl
• Answers.com definition of crawl
• Crawl definition on dictionary.die.net
• Crawler
• How to Have Google Crawl