What is robots.txt?
Robots.txt は、検索エンジン クローラー ボットやその他のボットに、Web サイト上でクロールが許可されていないページについて通知するファイルです。
The file is named robots.txt and is located in the site’s root folder. It is important to note that search engine crawler bots generally obey the rules in the robots.txt file, but other crawler bots may decide to crawl the links even if you instructed them not to crawl.
You should also avoid using the robots.txt file to hide URLs you do not want search engines to index. Google can discover and index those URLs if linked from another URL.