What Is Robots.Txt? A Startup Guide to the File and Format

robots.txt is a small plain-text file at the root of your domain that tells search engine crawlers which pages and directories they may crawl and which to leave alone. It uses the Robots Exclusion Protocol with simple directives like User-agent, Allow, and Disallow, and it lives at the exact path yourdomain.com/robots.txt -- anywhere else and crawlers ignore it.

It is one of the oldest conventions on the web, dating back to 1994, and it remains the first line of crawl control for every domain -- but it comes with limitations that surprise most first-time users. After you understand the basics, running a full technical SEO audit is the natural next step -- robots.txt is just one piece of a larger crawl-optimization picture.

TL;DR: What Is Robots.Txt?

• robots.txt is a plain-text file at yourdomain.com/robots.txt that tells crawlers which pages and directories they may crawl and which to skip.
• It uses the Robots Exclusion Protocol with four core directives: User-agent, Disallow, Allow, and Sitemap, plus the optional Crawl-delay.
• robots.txt controls crawling, NOT indexing -- a disallowed page can still appear in Google search results if linked from elsewhere.
• The file is publicly visible; anyone can view it at yourdomain.com/robots.txt, so never use it to hide sensitive data.
• Crawl-delay is ignored by Google; Googlebot follows its own crawl-rate algorithms regardless of what you set.
• An empty or missing robots.txt file is not an error -- it simply means "crawl everything," which is the default behavior.
• You can validate your robots.txt file for free using Google Search Console's robots.txt Tester tool in under a minute.

What Is Robots.Txt (and What Does It Actually Do)?

robots.txt is a voluntary compliance mechanism, not an access-control system. When Googlebot, Bingbot, or any well-behaved crawler visits your domain, it checks /robots.txt before fetching other pages. If a path is disallowed, the crawler will skip it. If it is allowed, the crawler proceeds normally. But a malicious bot or scraper is under no obligation to respect robots.txt -- which is why it belongs in the crawling toolbox, not the security toolbox.

The file serves three practical purposes in SEO. First, it directs crawl budget toward important pages by steering crawlers away from staging environments, internal search results, filtered ecommerce URLs, admin panels, and API endpoints. Second, it prevents duplicate-content crawling when the same page is reachable through multiple URL parameters. Third, it points crawlers to your XML sitemap. If you are building your SEO foundation from scratch, robots.txt fits into the broader SEO roadmap alongside structured data, internal linking, and content architecture.

How Does the Robots.Txt Robots Exclusion Protocol Work?

The Robots Exclusion Protocol (REP) was formalized as an Internet-Draft by Google, Microsoft, Yahoo, and Yandex in 2019, though the convention has been in use since 1994. The protocol is simple: a crawler fetches /robots.txt, parses it line by line, and checks whether the URL it wants to crawl is allowed under any matching rule group.

Rule groups are delimited by User-agent lines. Each User-agent directive starts a new group; all Allow and Disallow lines below it apply only to that crawler until the next User-agent line. A User-agent: * group applies to all crawlers that do not have their own specific group. The crawler picks the most specific matching rule for its user-agent token, then applies the longest matching path. This longest-match rule is a critical detail: Disallow: /blog overrides Disallow: / only for paths starting with /blog; everything else falls through to the broader rule.

Wildcards make the protocol more expressive: an asterisk (*) matches zero or more characters and a dollar sign ($) marks end-of-path. Disallow: /*? blocks URLs with query parameters; Disallow: /*.pdf$ blocks URLs ending in .pdf. These patterns are easy to get wrong -- a single misplaced character can open or close your entire site to crawling.

What Is the Correct Robots.Txt File Format?

The format is plain text. Each directive appears on its own line as a key-value pair separated by a colon and a space. Blank lines separate rule groups. Comments begin with # and are ignored by parsers. The character encoding must be ASCII or UTF-8, and the file size must not exceed 500 KiB -- Google ignores anything beyond that limit. Paths are case-sensitive: Disallow: /Admin blocks /Admin but not /admin.

Google ignores the Crawl-delay directive entirely -- it manages crawl rate automatically based on server response times and site authority. If you need to limit Googlebot's crawl rate, you must use the Google Search Console crawl rate settings. Bing and Yandex do respect Crawl-delay, so the directive is functional for those bots if your server is resource-constrained.

How Do You Create a Robots.Txt File?

Creating a robots.txt file takes five minutes once you know the syntax. The work is in deciding what to block, not in the file itself. Follow these steps to create a valid, production-ready robots.txt:

1. Confirm the root path. robots.txt must be served from the root of your domain on the exact host, protocol, and port. For https://www.example.com, the file must live at https://www.example.com/robots.txt. Placed in a subfolder like /blog/robots.txt or on a different subdomain, it is ignored. If your site redirects HTTP to HTTPS, serve robots.txt over HTTPS only.
2. Pick your user-agent groups. Decide whether you need a single User-agent: * group for all bots or separate groups for Googlebot, Bingbot, and others. Most startups need one broad group and optionally a second with tighter rules for less-used crawlers. Googlebot supports robots.txt fully, including pattern matching and the full spec.
3. Write your Allow and Disallow directives. Start with the broadest Disallow rules and use Allow to carve out exceptions. For example, Disallow: /wp-admin/ keeps crawlers out of your WordPress admin, and Allow: /wp-admin/admin-ajax.php lets them through if needed. Put your most specific rules first within each group -- the longest-matching rule wins.
4. Upload to the root of your domain. Create a file named robots.txt (case-sensitive on Linux servers) and place it in your web root. If you use a static site host like Vercel or Netlify, put robots.txt in the public/ or static/ directory. For WordPress, you can create it manually in the root or use SEO plugins like Yoast and Rank Math that generate it for you. Verify it is accessible at https://yourdomain.com/robots.txt.
5. Test in Google Search Console. Open the robots.txt Tester under the Crawl section, paste your URL, and confirm Googlebot can parse every directive. The tester highlights syntax errors and lets you test specific URLs against your rules. Fix any issues before Googlebot discovers them.

What Are Common Robots.Txt Patterns and Examples?

Most robots.txt files fall into a handful of standard patterns. Below are the four you will see most often, from open-to-all to tightly locked down.

User-agent: *
Disallow:

User-agent: *
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

A real-world example for a typical startup site that blocks staging, internal search, and parameterized URLs while pointing to a sitemap:

User-agent: *
Disallow: /staging/
Disallow: /search/
Disallow: /*?
Disallow: /admin/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The Disallow: /*? rule blocks any URL containing a query parameter -- useful for filtering and sorting pages that would otherwise consume crawl budget. The wildcard * matches any characters before the question mark, so /products?color=blue and /blog?page=2 are both blocked. And the trailing Allow: / confirms that everything not explicitly disallowed is eligible for crawling.

Robots.Txt vs. Meta Robots vs. X-Robots-Tag: What Blocks What?

New CMS users often confuse these three mechanisms. They control different things -- crawling, indexing, and serving -- and misunderstanding the differences is the source of the most common SEO errors. The table below maps each tool to what it actually controls.

The meta robots tag and X-Robots-Tag header are the only tools that prevent a page from being indexed after it is crawled. robots.txt blocks crawling, which means Googlebot never sees a noindex tag on a disallowed page -- and therefore can still index that page if external links point to it. This is the crux of the trap. For a deeper look at how indexation signals interact with search appearance, our guide on schema markup and SERP appearance covers the full stack.

Does Robots.Txt Block Indexing on Google?

The most dangerous misconception in entry-level SEO is that Disallow: / in robots.txt removes pages from Google's index. It does not. Disallow stops Googlebot from visiting a page, but if another site links to that page -- or if Google already indexed it before the rule was added -- the page can remain in the index without being crawled. When someone searches for the page's content, Google may display it as a bare URL with the note "A description for this result is not available because of this site's robots.txt."

This happens frequently when a developer blocks /staging/ in robots.txt but the staging site is publicly linked from a blog comment or GitHub README. Google discovers the URL through the link, indexes it, and refuses to show a snippet because robots.txt blocked crawling. The page is visible but looks broken in SERPs -- exactly the opposite of what the team intended.

To actually remove a page from Google's index, you need a noindex signal. The meta robots tag (<meta name="robots" content="noindex">) and the X-Robots-Tag HTTP header both tell search engines to drop the page from the index. Unlike Disallow, Googlebot must crawl the page to see the noindex directive -- so you cannot block the page in robots.txt and noindex it on the same URL. The rule is: first allow crawling, then noindex. Once Google recrawls and processes the noindex tag, the page drops from results.

This same principle underpins broader technical SEO health. When you run a comprehensive SEO audit, you typically start with crawling and indexation, then layer on content, links, and technical signals. robots.txt is the entry point to that crawl chain.

What Are Common Robots.Txt Mistakes That Hurt SEO?

Most robots.txt mistakes follow a pattern: the file looks correct at a glance but quietly breaks crawl access or blocks the wrong paths. These are the ones that cause real SEO damage:

1. Blocking CSS and JavaScript. Googlebot needs to render pages like a browser to understand layout and mobile-friendliness. If your robots.txt disallows /css/ or /js/ directories, Googlebot sees a broken page and your rankings suffer. This advice was common years ago but is now obsolete and harmful.

2. Using Disallow: / on the live site. This blocks crawling of every URL on your domain. It is correct for staging but catastrophic on production. Startups have lost all organic traffic for weeks after pasting in a staging robots.txt by mistake. Always double-check the production file allows crawling.

3. Disallowing pages you want to noindex. The page stays indexed but Google cannot crawl it, so it shows as a URL-only result with no snippet. To remove a page from results, allow crawling and add a noindex tag or X-Robots-Tag header.

4. Duplicate or conflicting rules. When two Disallow rules conflict, the longest-matching rule wins. A bot-specific group takes priority over User-agent: *, so "User-agent: Googlebot: Disallow: /" overrides "User-agent: *: Allow: /". Inconsistent group definitions can accidentally block Google while leaving Bing wide open.

5. Missing or incorrect sitemap URL. The Sitemap directive is optional but valuable -- it tells crawlers where to find your XML sitemap without a separate Search Console submission. A wrong URL (pointing to HTTP when your site is HTTPS, or including a trailing space) silently fails. Always use the full, canonical https:// URL.

These mistakes compound as a site grows. A small blog blocking /css/ loses a few positions; a large ecommerce site doing the same loses thousands of product pages from proper rendering. This is why technical SEO for startups treats robots.txt review as a launch-day checklist item.

How Do You Test and Validate Your Robots.Txt File?

Testing robots.txt is free and fast. Google Search Console provides a robots.txt Tester tool under Settings > Crawl > robots.txt. Paste a URL from your domain, and the tester evaluates it against your live robots.txt file, flagging whether it is allowed or disallowed for each user agent, plus any syntax errors or lines exceeding the 500 KiB limit. Changes must be deployed before testing since the tester reads your live file.

For bulk testing, you can use curl https://yourdomain.com/robots.txt to fetch the live file. Third-party SEO crawlers -- Screaming Frog, Sitebulb, Ahrefs -- also parse robots.txt and flag disallowed URLs in their reports.

Whatever tool you use, test three scenarios: a URL you intend to allow, a URL you intend to disallow, and a URL with query parameters that wildcard rules should match. If any test fails, fix and redeploy before crawlers encounter the broken rule.

A sound robots.txt configuration also supports your internal linking strategy by ensuring that important content paths are crawlable while junk parameters and faceted navigation stay out of the crawl queue. Both pieces -- crawl directives and link structure -- work together to shape how search engines discover your site.

Frequently Asked Questions

What Is Robots.Txt Used For?

robots.txt tells search engine crawlers which pages or directories on your site they are allowed to crawl and which to leave alone. It is a crawl-control signal, not a security or index-removal tool.

Does Robots.Txt Block a Page from Being Indexed?

No. Disallowing a page in robots.txt stops crawlers from visiting it, but if other sites link to that URL Google can still index it without crawling. To actually remove a page from the index, use a noindex meta robots tag or an X-Robots-Tag header.

Where Do I Put My Robots.Txt File?

At the root of your domain on the exact host, protocol, and port -- for example https://www.example.com/robots.txt. Placed in a subfolder or subdomain it is ignored.

What Happens If I Have No Robots.Txt File?

Crawlers assume they may crawl the entire site. A missing robots.txt file is usually fine; it is not an error, though you lose the chance to direct crawl budget or point crawlers at your sitemap.

How Do I Allow All Crawlers in Robots.Txt?

Use two lines: "User-agent: *" on one line and "Allow: /" on the next (or simply "User-agent: *" with an empty "Disallow:"). This permits every crawler to access the entire site.

Key Takeaways

• robots.txt controls crawling, not indexing. A disallowed page can still appear in Google results if linked from elsewhere -- use a noindex meta robots tag or X-Robots-Tag header to remove pages from the index.
• The file lives at yourdomain.com/robots.txt and nowhere else. Subfolder paths, subdomains, and wrong protocols all cause crawlers to ignore it.
• Keep CSS and JS crawlable. Blocking resource directories prevents Google from rendering your pages and directly hurts rankings.
• Validate every change with Google Search Console's robots.txt Tester before deploying. Test allowed URLs, disallowed URLs, and wildcard-matching URLs.
• robots.txt is publicly readable. Never use it to hide sensitive content, authentication pages, or internal data -- it is a crawl advisory, not a security barrier.
• For a complete crawl-and-index health check, pair robots.txt with a full technical SEO audit checklist that covers sitemaps, status codes, canonical tags, and rendering.