What is a Crawler?
A crawler, or web crawler, is a bot used to gather information about a webpage. They are often used by search engines.
The process of visiting a webpage to fetch its content is called crawling. Once crawled, the webpage can be added to the search engine index and displayed in search results.
The term crawler is often used for search engine crawlers. Crawlers used by systems other than search engines are called web scrapers.