What is Crawl Delay?
クロール遅延ディレクティブは、クローラー ボットが Web サイト上のページをリクエストできる速度を指定します。
Crawling uses up a server’s resources. This could slow down the server or even overload it and cause it to go offline.
So, if you notice that crawlers are using up more server resources than you desire, you can set up a crawl delay by adding the below rule to your robots.txt file:
crawl-delay: 10
In the above rule, 10 is a placeholder for the time in seconds. You can change it to any time of your choice.
However, whether the crawler bot recognizes the crawl delay rule is up to its creators. Google and Yandex have clarified that they do not obey the crawl delay rule, while Bing does.