🇺🇸 4th of July DISCOUNT!


What is Crawl Delay?

Die Crawl-Verzögerungsanweisung gibt an, wie schnell ein Crawler-Bot Seiten auf Ihrer Website anfordern kann.

Crawling uses up a server’s resources. This could slow down the server or even overload it and cause it to go offline.

So, if you notice that crawlers are using up more server resources than you desire, you can set up a crawl delay by adding the below rule to your robots.txt file: 

crawl-delay: 10 

In the above rule, 10 is a placeholder for the time in seconds. You can change it to any time of your choice. 

However, whether the crawler bot recognizes the crawl delay rule is up to its creators. Google and Yandex have clarified that they do not obey the crawl delay rule, while Bing does.

🇩🇪 Deutsch