What is robots.txt?
Robots.txt is a file that tells search engine crawler bots and other bots about the pages they are not allowed to crawl on the website.
The file is named robots.txt and is located in the site’s root folder. It is important to note that search engine crawler bots generally obey the rules in the robots.txt file, but other crawler bots may decide to crawl the links even if you instructed them not to crawl.
You should also avoid using the robots.txt file to hide URLs you do not want search engines to index. Google can discover and index those URLs if linked from another URL.