Skip to content
Q3.Labs, Home

Free Tool

Robots.txt Generator & Tester

Create a robots.txt file for your website and test how your rules handle specific URL paths — entirely in your browser.

  • Free
  • · No signup
  • · Nothing uploaded
  • · RFC 9309-based tester
A robots.txt file being generated and tested against sample URL paths

Your robots.txt rules are generated and tested in your browser. We don't upload, log, or store your rules, paths, or user-agent values.

Templates are examples, not universally correct rules — review and adapt every path to your own site.

User-agent group

Must be absolute URLs — we never assume or invent one for you.

Generated robots.txt

0.02 KB
User-agent: *
Allow: /

Test a URL path

Robots.txt rule tester — this uses the documented Robots Exclusion Protocol (RFC 9309) matching behavior this tool implements. It does not simulate every crawler-specific implementation, and its result is not a guarantee of what any specific search engine will do.

Robots.txt rules apply to URLs on the site's origin; this tester evaluates only the path (and query string, if any) you provide — a full URL is accepted and its path is extracted locally, nothing is sent anywhere.

What Is Robots.txt?

Robots.txt is a plain-text file, placed at a website's root, that tells well-behaved web crawlers which parts of the site they may or may not request. It's formalized by the Robots Exclusion Protocol (RFC 9309) and read by search engine crawlers, including Google's, before they crawl a site.

What Does Robots.txt Do?

It gives crawling instructions: which user-agents a rule group applies to, which paths they may (Allow) or should not (Disallow) request, and optionally where to find the site's sitemap. It works purely on cooperation, crawlers that follow the protocol respect it voluntarily.

What Does Robots.txt NOT Do?

  • It is not an authentication or security mechanism — the file itself is publicly readable.
  • Disallowing a URL doesn't guarantee it will never appear in search results, Google's own documentation states a disallowed URL can still be indexed if discovered another way, typically without a snippet.
  • It does not remove already-indexed content from search results by itself.
  • It cannot force a non-cooperating crawler to obey it.

Sensitive or private content should be protected at the server or application level, with real authentication, never with robots.txt alone.

How to Create a Robots.txt File

  1. Choose a starting template above, or build your own rule groups.
  2. Add a user-agent group for each crawler (or all crawlers) you want to address.
  3. Add Allow/Disallow rules with the paths you want to affect.
  4. Optionally add your sitemap URL(s).
  5. Review any warnings, then copy or download the generated file.
  6. Upload it to your site's root as /robots.txt.

Common Robots.txt Examples

Allow all

User-agent: *
Allow: /

Block a directory (example path)

User-agent: *
Disallow: /admin/

Block the entire site

User-agent: *
Disallow: /

With a sitemap reference

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

These are examples to adapt, not universal rules — the right paths depend entirely on your own site's structure.

Allow vs Disallow

Disallow asks a crawler not to request paths matching the rule; Allow explicitly permits them. Allow is most useful for carving an exception out of a broader Disallow, for example blocking /private/ overall while explicitly allowing /private/public-page. When rules conflict, the documented behavior is that the most specific matching rule, the one with the most characters, wins, and equally specific ties favor Allow.

How Robots.txt Interacts With Sitemaps

A Sitemap line in robots.txt points crawlers to a sitemap file, helping them discover your URLs. Per Google's documentation it must be a fully qualified absolute URL and isn't tied to any specific user-agent group, it applies regardless of which group a crawler matches. Robots.txt controls crawling permission; the sitemap is a separate discovery aid, entirely independent tools worth pairing together.

Robots.txt Mistakes to Avoid

  • Accidentally leaving "Disallow: /" in place after testing, blocking the whole site.
  • Assuming robots.txt hides sensitive information from the public.
  • Relying on Crawl-delay, Host, or Noindex as if they were standard, universally supported directives.
  • Forgetting that a disallowed URL can still be indexed if discovered another way.
  • Placing the file somewhere other than the site's root.
  • Using a relative or malformed sitemap URL instead of a fully qualified absolute one.

Frequently Asked Questions

What is a robots.txt file?

A plain-text file at a website's root that tells well-behaved crawlers which parts of the site they may or may not request, using the Robots Exclusion Protocol (formalized as RFC 9309).

Is robots.txt required for every website?

No. A site with no robots.txt file is treated as having no crawl restrictions at all — it's optional, useful mainly when you want to guide crawler access deliberately.

Where should robots.txt be placed?

At the site's root, for example https://example.com/robots.txt — crawlers look for it at that specific location on each site.

What does User-agent: * mean?

It defines a group of rules that applies to any crawler not specifically matched by a more specific User-agent group elsewhere in the file.

What does Disallow: / mean?

It asks crawlers covered by that group not to request any URL on the site. This tool warns you when you create this rule, since it affects the entire site.

How do I allow all crawlers?

Use "User-agent: *" with either "Allow: /" or simply no Disallow rules at all — both permit unrestricted crawling for any agent matching that group.

Can robots.txt block Google?

It can ask Googlebot and other Google crawlers not to crawl specific paths, and Google's documentation states it follows robots.txt rules for crawling. It's a crawling instruction, not an enforcement mechanism outside of cooperating crawlers.

Does robots.txt remove a page from Google?

Not reliably. Google's own documentation states that a disallowed page can still be indexed and shown in search results (typically without a snippet) if Google finds it another way, like a link from elsewhere. Blocking crawling is not the same as removing a page from search results.

Can robots.txt protect private information?

No. Robots.txt is a voluntary instruction for well-behaved crawlers, not an access-control or security mechanism, and the file itself is publicly readable. Sensitive or private content should be protected with real authentication or server-level access control, never robots.txt alone.

Should I add my sitemap to robots.txt?

It's a common, useful practice. A Sitemap line points crawlers to your sitemap file, and per Google's documentation it must be a fully qualified, absolute URL, not tied to any specific user-agent group.

What is Crawl-delay?

A non-standard directive some crawlers historically used to request a pause between requests. It is not part of RFC 9309 and Google's own documentation states it is not supported by Google Search, so this tool never generates it and flags it clearly if found in a pasted file.

Can I use robots.txt for specific folders?

Yes, a Disallow or Allow rule with a folder-style path (like /admin/) applies to that path and, by default prefix matching, anything beneath it.

What is the difference between Allow and Disallow?

Disallow asks a crawler not to request matching paths; Allow explicitly permits them, most useful for carving out an exception inside a broader Disallow rule. When rules conflict, the most specific (longest) matching rule wins.

Does robots.txt control every search engine the same way?

Every crawler that respects the Robots Exclusion Protocol should follow the documented core semantics, but implementation details and support for non-standard directives can differ between crawlers. This tool implements the documented RFC 9309 and Google-published behavior, not every crawler's private implementation.

Explore more

Generate the sitemap you'll reference here with our XML Sitemap Generator, verify a URL's status with the HTTP Status Checker, or check a page's links with the Broken Link Checker. Inspect a page's meta tags with the Meta Tag Checker, or run a full Buzzing Bee website & SEO audit. Explore the rest of our free marketing tools, or see our SEO & organic growth services.

Keep exploring

Related services

  • SEO & Organic GrowthMost SEO work stalls after the first few technical fixes because there's no system connecting content, author…
  • Digital solutions for BankingBanks compete on trust, clarity and convenience, under strict rules about what they can say and do. We build…
  • Content MarketingMost content programs produce volume without strategy: articles that rank for nothing and convert no one.

Guides & articles

Tools

  • Broken Link CheckerCheck the links on a public page for 404s, redirects, and unreachable destinations.
  • HTTP Status CheckerCheck a URL's HTTP status code and full redirect chain, or check up to 20 URLs at once.

Locations

  • TexasTrusted by growth-stage brands across Texas. Digital marketing, SEO and web development for businesses in Aus…

We use cookies for analytics to understand site usage. Privacy Policy