Robots.txt - SEO Best Practices

Add a robots.txt file to every subdomain so you can specify sitemap locations and set web crawler rules. Robots.txt files are always located in the root folder with the name robots.txt. Each robots.txt file only applies to URLs with the same protocol, subdomain, domain and port as the robots.txt URL. For example, http://example.com/robots.txt would be the robots.txt URL for http://example.com but not https://example.com or http://www.example.com. Even an empty robots.txt file is useful to have for cleaning up server logs as it will reduce 404 errors from visiting bots. Keep in mind that if you use a robots.txt file to tell search bots not to visit a certain page, that page can still appear in search results if it’s linked to from another page. To hide pages from search results, use noindex meta tags instead.