What is robots.txt?
Robots.txt on tiedosto, joka kertoo hakukoneiden indeksointiroboteille ja muille roboteille sivuista, joita ne eivät saa indeksoida verkkosivustolla.
The file is named robots.txt and is located in the site’s root folder. It is important to note that search engine crawler bots generally obey the rules in the robots.txt file, but other crawler bots may decide to crawl the links even if you instructed them not to crawl.
You should also avoid using the robots.txt file to hide URLs you do not want search engines to index. Google can discover and index those URLs if linked from another URL.