A robots.txt file can be only a few lines long, yet one broad rule can change which parts of a site Googlebot, Bingbot, AI crawlers, or advertising crawlers are allowed to request. The difficult part is not writing the file. It is knowing what robots.txt can control, what it cannot control, and where a seemingly sensible rule can create an unintended crawl problem.
Robots.txt is not a ranking switch. It is a crawler-access file. In practical terms, it lets a website tell compliant crawlers which URL paths they may request. That distinction matters because a large share of bad robots.txt advice starts by treating crawl control, indexing, security, and ranking as if they were the same thing. They are not.
For a small public website, you may not need a custom robots.txt file at all. Google says that not having one is fine; it generally means Google can crawl the public URLs it discovers. A custom file becomes useful when you have a clear reason to manage crawling—for example, a large faceted store, duplicate URL spaces, crawler-specific preferences, or a platform section that should not be requested by compliant bots.
This guide covers the mistakes that are actually worth checking on a live site. It also explains the difference
between robots.txt and noindex, how to review your file in Google Search Console, and when a
robots.txt generator
is useful.
Use robots.txt to manage crawling. Use noindex when you want a crawlable page removed from Google Search.
Use authentication or another real access-control method for private content. If Google cannot crawl a page because
robots.txt blocks it, Google cannot read a noindex tag on that page.
Need to build or review a file? The MZTechPro tool creates a conservative draft, includes an existing-file quick check, and keeps advanced crawler controls optional.
Open the Free Robots.txt GeneratorWhat Does a Robots.txt File Actually Do?
Robots.txt is a plain-text file served from the root of a protocol-and-host combination, such as
https://example.com/robots.txt. The standardized core of the Robots Exclusion Protocol is defined in
RFC 9309. A file is made up of crawler groups, and those groups normally contain a User-agent line
followed by Allow or Disallow rules.
The wildcard group above applies to crawlers that use that group. It asks them not to request URLs matching
/internal-search/. The sitemap line points supporting crawlers to an XML sitemap. It does not guarantee
that those URLs will be crawled, indexed, or ranked.
The important part is what the file does not do. Robots.txt is public, so it cannot protect confidential content. It is also not a reliable way to remove a URL from search results. Google can sometimes know about a disallowed URL from links even when it cannot crawl the page itself.
Seven Robots.txt Mistakes Worth Fixing
Blocking the Entire Site Without Realising It
The most obvious high-impact rule is also one of the easiest to overlook during a migration or staging-to-production launch:
For crawlers that follow the wildcard group, this asks them not to crawl any path on the site. That may be intentional in a temporary environment, but it is a serious problem if it reaches a live public site by accident.
Even here, robots.txt should not be your only protection for a staging environment. A staging site can still leak URLs through links, logs, or external references. If the environment must remain private, protect it with authentication or another access-control layer.
Using Disallow as If It Were a Noindex or Security Rule
A robots.txt Disallow rule controls crawling. A robots meta tag or HTTP
X-Robots-Tag: noindex controls indexing for search engines that support it. Password protection,
authentication, or server-side access controls protect content from unauthorised users.
Mixing those jobs creates a common failure: a page is blocked in robots.txt and also given a
noindex tag. Googlebot cannot see the noindex because it cannot crawl the page. Google explicitly
documents that a page must remain accessible to the crawler for a noindex directive to be read.
If the goal is “do not show this page in Google,” do not block Googlebot from the page and then expect it to discover a noindex tag that it is not allowed to fetch.
Copying Generic WordPress, Blogger, or E-commerce Rules Without Checking the Site
A template can be a useful starting point, but it should not decide your crawl policy for you. WordPress plugins, custom themes, Shopify stores, WooCommerce sites, Blogger blogs, SaaS applications, and bespoke platforms all use different URL structures.
Generic lists that automatically block /wp-content/, plugin folders, script directories, tag archives,
search URLs, cart paths, or account URLs can be wrong for a particular site. Google may need CSS, JavaScript, images,
and other resources to render a page properly. A category, tag, filter, or search page may also have genuine user value
depending on the site.
Review the actual URLs first. If you cannot explain why a path should be blocked, leave it alone until you can. Conservative rules are usually easier to audit than a copied file containing 20 assumptions.
Following Path-Matching Myths, Especially the “Trailing Slash Is Required” Rule
A trailing slash is not universally “required.” It changes what a path can match. For example,
Disallow: /admin and Disallow: /admin/ are not identical rules. The first can match URLs
beginning with /admin, while the second is narrower because it includes the slash.
The correct rule depends on your real URL structure. Case also matters in URL paths. Google supports pattern matching
such as * and $, but crawler behaviour should be checked against the documentation of the
crawler you are targeting rather than assumed from an old SEO checklist.
Relying on Unsupported or Non-standard Directives
Google supports the core robots.txt fields it documents, including User-agent, Allow,
Disallow, and Sitemap. It does not support crawl-delay. Crawl-delay is a
non-standard extension that some other crawlers may support, so it should only be used when the crawler you care about
explicitly documents it.
Another outdated pattern is placing noindex inside robots.txt. Google does not support robots.txt
noindex. Use a page-level robots meta tag or an HTTP X-Robots-Tag instead.
Assuming One Robots.txt File Covers Every Subdomain and Protocol
Robots.txt applies to the origin where it is served. A file at https://example.com/robots.txt does not
automatically control https://shop.example.com/, https://images.example.com/, or the HTTP
version of the host.
This matters on sites split across a blog, store, media host, app, or regional subdomain. Search Console's current robots.txt report can show multiple files for the hosts associated with a property, which makes it useful for spotting an old or conflicting file you had forgotten about.
The same care applies to sitemap declarations. Use a valid absolute sitemap URL and make sure it points to the sitemap you actually maintain. A sitemap record in robots.txt is useful, but optional.
Blocking Special-purpose, Advertising, or AI Crawlers Without Understanding the Trade-off
Modern robots.txt files increasingly contain crawler-specific groups. Those rules can have consequences beyond ordinary Google Search, so a user-agent name should not be added just because it appeared in somebody else's template.
For AdSense publishers, Google documents Mediapartners-Google as a special-case crawler used to provide
relevant ads. The global User-agent: * group is ignored for this crawler; crawler-specific preferences
addressed to Mediapartners-Google can affect AdSense crawling.
AI-related crawlers also have different purposes. OpenAI documents OAI-SearchBot for ChatGPT search
discovery and GPTBot as a model-training control. Anthropic separately documents
Claude-SearchBot, Claude-User, and ClaudeBot. Google describes
Google-Extended as a control for certain Gemini model-development and grounding uses and states that it
does not affect Google Search inclusion or ranking.
The practical rule is simple: decide what you want before you block a crawler. “AI bot” is no longer one single purpose.
Robots.txt vs Noindex vs Password Protection
When people say they want to “block a page,” they often mean three very different things. Choosing the right control is more important than adding another robots.txt rule.
| Your goal | Best-fit control | What to remember |
|---|---|---|
| Reduce or prevent compliant crawler requests to a URL path | robots.txt |
The URL may still be known from links; robots.txt is not access control. |
| Keep a crawlable page out of Google Search | noindex meta tag or X-Robots-Tag |
Googlebot must be allowed to crawl the page to read the directive. |
| Keep private content inaccessible to the public and crawlers | Authentication or server-side access control | Do not publish secret URLs in robots.txt and treat that as protection. |
Does Robots.txt Improve SEO or Crawl Budget?
Robots.txt can be useful in technical SEO because crawling is part of how search engines discover and process a site. That does not make robots.txt a direct ranking factor, and it does not mean every site needs an elaborate crawl-budget strategy.
Google has repeatedly said that most small sites do not need to worry about crawl budget. Crawl-efficiency work becomes more relevant on very large sites, fast-changing sites, or sites that create large numbers of duplicate or faceted URLs. On a modest blog or business site, publishing better content, keeping internal navigation crawlable, fixing indexing issues, and maintaining accurate sitemaps will usually deserve more attention than adding dozens of Disallow rules.
If you do manage a large e-commerce platform, marketplace, publisher, or application with many parameter combinations, robots.txt may be one part of a broader crawl-management plan. It should still be based on measured URL inventory and server/crawl data rather than a generic “SEO robots.txt” template.
How to Audit Your Robots.txt File in Google Search Console
Google's current Search Console workflow is different from the old editable robots.txt tester that many older tutorials still describe. Today, the robots.txt report shows the files Google found, fetch status, last crawl, file size, and parsing issues. For a specific page, URL Inspection can help confirm whether Google is blocked from crawling it.
-
Open the live file. Visit the exact robots.txt URL for the host you are auditing, such as
https://example.com/robots.txt. - Read every rule before changing anything. Look for site-wide blocks, old staging rules, CMS paths copied from another site, unsupported directives, and crawler-specific groups.
- Check the Search Console robots.txt report. Review fetch status and any parser warnings or errors. The report also lets you request a recrawl after a critical change.
- Inspect important URLs. Use URL Inspection for the homepage, key category pages, important articles or products, and any URL you suspect is blocked.
- Confirm the indexing control. If a page should stay out of search, verify that you are using noindex or proper access control rather than relying on Disallow.
- Publish the smallest necessary change. Broad robots.txt edits are harder to reason about than narrow, deliberate rules.
- Check again after deployment. Confirm that the live file is the version you intended and that important URLs remain crawlable.
Google processes robots.txt files up to 500 KiB; content beyond that limit is ignored. A normal site should never need a file anywhere near that size. If yours is huge, simplify and consolidate it rather than adding more rules.
How the MZTechPro Robots.txt Generator Fits Into This Workflow
The companion tool is deliberately simpler than the article. Its main screen asks for only three things: your website type, an optional sitemap URL, and an optional preference for selected model-training crawler controls. It then creates a draft you can review before copying or downloading it.
Advanced Options are there when you need more control. The current tool supports custom
User-agent, Allow, and Disallow rules; separate AI search and user-retrieval
crawler controls; optional crawl-delay for crawlers that support it; explicit AdSense crawler access controls; and an
Existing robots.txt Quick Check for several common structural risks.
That last part is useful when you already have a file. Instead of starting over, you can paste the existing text and look
for obvious issues such as a wildcard site-wide block, a robots.txt noindex line, invalid sitemap URLs, an oversized file,
or an explicit block on Mediapartners-Google. It is a quick local check, not a full crawler emulator, so the
live website still needs to be tested after deployment.
Build a new draft or check an existing file.
Use the Free Robots.txt GeneratorPractical Robots.txt Examples
A minimal file that does not block normal crawling
An empty Disallow: does not block a path. You could also omit a custom robots.txt file altogether when no
crawl restrictions are needed and your platform does not require one.
A conservative WordPress starting point
This is a starting point, not a universal WordPress recipe. Check the sitemap location and any custom routes used by your installation before publishing it.
An explicit AdSense crawler group
Google says the global wildcard group is ignored for Mediapartners-Google. You normally do not need an
explicit allow-all group simply because you use AdSense, but adding one can make your intent clear when you are auditing a
crawler-specific configuration.
Frequently Asked Questions
Is it bad to have no robots.txt file?
No. Google says that not having a robots.txt file is fine. For a small public site with no crawl restrictions, you may not need a custom file at all.
Can robots.txt remove a page from Google Search?
Not reliably. Robots.txt controls crawling. If you want Google to drop a crawlable page from its index, use a supported noindex meta tag or HTTP header and allow Googlebot to crawl the page so it can see that directive.
Can robots.txt protect a private admin area?
No. Robots.txt is public and is not an authentication system. Use real access controls for private or sensitive areas.
Do I need a trailing slash on every Disallow rule?
No. The slash changes the path you are matching. Choose the rule that matches your real URLs rather than adding a trailing slash mechanically.
Does Google support Crawl-delay?
No. Google documents crawl-delay as unsupported. Some other crawlers may support the non-standard directive, so check the crawler's own documentation before using it.
Should I always add my sitemap to robots.txt?
A Sitemap record is useful and widely supported, but it is optional. It does not guarantee that URLs will be crawled or indexed. You can also submit sitemaps through search-engine webmaster tools.
Should AdSense publishers add Mediapartners-Google?
Not necessarily. Google documents Mediapartners-Google as a special-case crawler that ignores the global wildcard group. If you create a Mediapartners-Google-specific group, avoid blocking it when you want AdSense crawling.
Can I block AI model-training crawlers but still allow AI search crawlers?
In some cases, yes. Providers increasingly document separate crawler tokens for different purposes. For example, OpenAI separates GPTBot from OAI-SearchBot, and Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User. Review the provider's current documentation before adding rules because crawler names and policies can change.
Official References
This article was cross-checked against primary documentation rather than generic robots.txt templates. These are the main references used for the August 2026 review:
- RFC 9309 — Robots Exclusion Protocol
- Google Search Central — Introduction to robots.txt
- Google — How Google interprets the robots.txt specification
- Google Search Central — noindex guidance
- Google Search Console — robots.txt report
- Google — special-case crawlers including Mediapartners-Google
- OpenAI — publisher crawler guidance
- Anthropic — crawler controls
Final Checklist
Before you publish or replace a robots.txt file, read it as if you were the crawler. Which user-agent group matches? Which paths are actually affected? Is a site-wide block intentional? Are you trying to control crawling, indexing, or access? Does a crawler-specific rule affect Search, AdSense, AI search, or model training?
A good robots.txt file is usually boring. It contains only the rules the site genuinely needs, is easy to read six months later, and does not try to solve problems that belong to noindex, authentication, canonicalisation, or site architecture. That is a better goal than building the longest or most “SEO-optimised” robots.txt file you can find.
Ready to review your own file? Start with the existing-file check or create a clean draft from scratch.
Open the Robots.txt Generator