What are robots.txt?

A file that includes certain pages you do not want indexed on search engines.

robots-txt