Robots.txt Generator

What robots.txt controls — and what it does not

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may request. Build your rules here, download the file, and upload it so it is reachable at yourdomain.com/robots.txt. It must live at the root — a file at /files/robots.txt is ignored entirely.

The critical distinction, and the source of most robots.txt mistakes, is that it controls crawling, not indexing. Disallow tells a crawler not to fetch a page. It does not tell a search engine to keep that page out of its results. If other sites link to a disallowed URL, Google can still index it from those links alone — which produces the familiar bare listing with no description and the note that no information is available for this page.

The trap: never Disallow a page you want de-indexed

To remove a page from search results you need a noindex directive, delivered either as a meta robots tag in the page's HTML or an X-Robots-Tag HTTP header. But a crawler has to fetch the page to see that directive.

So blocking the page in robots.txt actively prevents its removal: Google cannot fetch the page, cannot see the noindex, and the URL stays in the index indefinitely. The correct sequence is to allow crawling, serve noindex, wait for the pages to drop out, and only then add a Disallow rule if you also want to save crawl budget.

What actually belongs in the file

Most sites need very little. Blocking too much is far more damaging than blocking too little, and a great many self-inflicted traffic losses trace back to an over-broad rule.

• Worth blocking — internal search result pages, faceted filter URLs that generate near-infinite combinations, cart and checkout paths, admin areas, and staging or print variants of real pages.
• Never block — CSS and JavaScript files. Google renders pages like a browser, and blocking your assets means it evaluates a broken layout, which harms mobile-friendliness and Core Web Vitals assessment. This was a widespread problem when it was still common advice.
• Never block — images you want in Google Images, or your own sitemap.
• Always include — an absolute Sitemap: line pointing at your sitemap URL. It costs nothing and helps discovery.

Directives and how they are actually interpreted

User-agent starts a block of rules for a named crawler; the asterisk matches any crawler that has no block of its own. Note that a crawler obeys only the most specific block that names it — if you write rules for Googlebot and separate rules for the wildcard, Googlebot ignores the wildcard section entirely. Rules are not merged, which surprises people regularly.

Allow creates an exception inside a broader Disallow, which is how you block a directory but permit one file inside it. Where two rules conflict, Google applies the most specific match by path length rather than the order they appear.

Crawl-delay is honoured by Bing and Yandex but ignored completely by Google — control Google's crawl rate through Search Console instead. The obsolete Noindex: and Host: directives are not supported and should not be used.

Before you upload it

Treat robots.txt as a live production file, because it is one of the few files that can remove a whole site from search in a single line. The pattern Disallow: / blocks everything, and it very commonly reaches production because it was correct on the staging server and nobody changed it at launch.

After uploading, fetch the file yourself to confirm it returns a 200 and the expected content, then test specific URLs in Search Console's robots.txt tester. Be aware too that a robots.txt returning a 5xx error is treated by Google as a temporary instruction to stop crawling the site, so an outage on that one file has outsized consequences.