A file used by webmasters that give search engine robots [like Googlebot] directives on how to crawl a website.