What is a Robots Meta Tag?
The robots meta tag is an HTML meta tag for instructing search engine crawlers on how to crawl, index and display a webpage in search results. It is inserted in the head tag <head> of the webpage.
The robots meta tag is different from the robots.txt file. The robots.txt file tells search engine crawlers how to crawl the entire site. The robots meta tag tells search engines how to crawl a specific webpage.
For example, the below robots meta tag instructs search engines not to index the webpage.
<meta name="robots" content="noindex">