Last year, a new project at work called for me to research and familiarize  myself with the robots.txt file. When I first heard about this file, I was a little intimidated. I’ve seen the file in WordPress theme directories, but I never saw it being mentioned as a required file by the developers I learned WordPress development from. Turns out it’s a pretty useful tool that WordPress and web developers alike should consider using.

Well…what is a robots.txt file?

The robots.txt is a text file used to help instruct search engine web robots on which web pages to crawl or not crawl on a website. It is usually placed at the root of a website so that the robots can easily access it when crawling a site. The robots.txt file is a method that is a part of the robots exclusion protocol or robots exclusion standard, which also includes metadata and header directives. Remember to always name the file, robots.txt, otherwise it will not work.

What goes in a robots.txt file? 

A robots.txt file’s structure is actually quite simple. The main directives include the following:

User-agent: This is where the name of the web robot is placed

Disallow: This where the URL that a web robot should not crawl should be placed

Other directives include:

Allow: [optional] This where the URL that a web robot should crawl would be placed (usually used in Googlebot is the user-agent)

Crawl-delay: The time (in seconds) that web robots should wait before crawling a website

Sitemap: This is where the URL of the XML sitemap should be placed (Google, Ask, Yahoo, and Bing are the only search engines that support this command)

The user-agent and disallow directives are enough to be a complete robots.txt file. Multiple sets of user-agent and disallow directives can exist in the same file, as long as each set is separated by a line break. 

Here is an example of a robots.txt file from Google’s Create a robots.txt file article: 

        User-agent: Googlebot

        Disallow: /nogooglebot/

        User-agent: *

        Allow: /

        Sitemap: http://www.example.com/sitemap.xml

Regular expressions can also be used for the robots.txt directives (except for the sitemap directive) in order to account for any patterns that appear in the URLs. There’s a Google Developer article that showcases some examples. The wildcard “*” and the “$” symbols are typically used, and other regex symbols are recognized as well, but not all search engines recognize them. Google and Bing are a couple of examples of search engines that recognize regex symbols.

For more information, check out: https://moz.com/learn/seo/robotstxt.