Q3
Your robots.txt rules are generated and tested in your browser. We don't upload, log, or store your rules, paths, or user-agent values.
Templates are examples, not universally correct rules — review and adapt every path to your own site.
User-agent group
Must be absolute URLs — we never assume or invent one for you.
Generated robots.txt
0.02 KBUser-agent: *
Allow: /
Test a URL path
Robots.txt rule tester — this uses the documented Robots Exclusion Protocol (RFC 9309) matching behavior this tool implements. It does not simulate every crawler-specific implementation, and its result is not a guarantee of what any specific search engine will do.
Robots.txt rules apply to URLs on the site's origin; this tester evaluates only the path (and query string, if any) you provide — a full URL is accepted and its path is extracted locally, nothing is sent anywhere.
What Is Robots.txt?
Robots.txt is a plain-text file, placed at a website's root, that tells well-behaved web crawlers which parts of the site they may or may not request. It's formalized by the Robots Exclusion Protocol (RFC 9309) and read by search engine crawlers, including Google's, before they crawl a site.
What Does Robots.txt Do?
It gives crawling instructions: which user-agents a rule group applies to, which paths they may (Allow) or should not (Disallow) request, and optionally where to find the site's sitemap. It works purely on cooperation, crawlers that follow the protocol respect it voluntarily.
What Does Robots.txt NOT Do?
- It is not an authentication or security mechanism — the file itself is publicly readable.
- Disallowing a URL doesn't guarantee it will never appear in search results, Google's own documentation states a disallowed URL can still be indexed if discovered another way, typically without a snippet.
- It does not remove already-indexed content from search results by itself.
- It cannot force a non-cooperating crawler to obey it.
Sensitive or private content should be protected at the server or application level, with real authentication, never with robots.txt alone.
How to Create a Robots.txt File
- Choose a starting template above, or build your own rule groups.
- Add a user-agent group for each crawler (or all crawlers) you want to address.
- Add Allow/Disallow rules with the paths you want to affect.
- Optionally add your sitemap URL(s).
- Review any warnings, then copy or download the generated file.
- Upload it to your site's root as /robots.txt.
Common Robots.txt Examples
Allow all
User-agent: *
Allow: /Block a directory (example path)
User-agent: *
Disallow: /admin/Block the entire site
User-agent: *
Disallow: /With a sitemap reference
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThese are examples to adapt, not universal rules — the right paths depend entirely on your own site's structure.
Allow vs Disallow
Disallow asks a crawler not to request paths matching the rule; Allow explicitly permits them. Allow is most useful for carving an exception out of a broader Disallow, for example blocking /private/ overall while explicitly allowing /private/public-page. When rules conflict, the documented behavior is that the most specific matching rule, the one with the most characters, wins, and equally specific ties favor Allow.
How Robots.txt Interacts With Sitemaps
A Sitemap line in robots.txt points crawlers to a sitemap file, helping them discover your URLs. Per Google's documentation it must be a fully qualified absolute URL and isn't tied to any specific user-agent group, it applies regardless of which group a crawler matches. Robots.txt controls crawling permission; the sitemap is a separate discovery aid, entirely independent tools worth pairing together.
Robots.txt Mistakes to Avoid
- Accidentally leaving "Disallow: /" in place after testing, blocking the whole site.
- Assuming robots.txt hides sensitive information from the public.
- Relying on Crawl-delay, Host, or Noindex as if they were standard, universally supported directives.
- Forgetting that a disallowed URL can still be indexed if discovered another way.
- Placing the file somewhere other than the site's root.
- Using a relative or malformed sitemap URL instead of a fully qualified absolute one.
Frequently Asked Questions
Explore more
Generate the sitemap you'll reference here with our XML Sitemap Generator, verify a URL's status with the HTTP Status Checker, or check a page's links with the Broken Link Checker. Inspect a page's meta tags with the Meta Tag Checker, or run a full Buzzing Bee website & SEO audit. Explore the rest of our free marketing tools, or see our SEO & organic growth services.
Related tools & resources
Related tools
- HTTP Status CheckerCheck a URL's HTTP status code and full redirect chain, or check up to 20 URLs at once.
- Slug GeneratorTurn titles into clean, readable URL slugs as you type, with Unicode support and a URL preview. Nothing is up…
- Hreflang Generator & ValidatorGenerate and validate hreflang tags for multilingual and international SEO.
Learn more
- SEO GuidesTechnical SEO, on-page fundamentals, and the checklist we run against every client site before it goes live.
- SEO & Organic GrowthMost SEO work stalls after the first few technical fixes because there's no system connecting content, author…
Related articles
- How Long Does SEO Take to Show ResultsSEO usually takes three to six months to show movement and six to twelve to deliver steady leads. Here is wha…
- Technical SEO Checklist for Business WebsitesMost technical SEO checklists run to 150 items written for enterprise auditors. This one is sorted into four…