Home › Generators › Robots.txt Generator
Robots.txt Generator
Build a robots.txt file for your website in seconds with this free robots.txt generator. Pick which crawlers the rules apply to, add allow and disallow paths, add your sitemap, and set an optional crawl-delay — the live preview updates as you type, and you can copy the file or download it ready to upload to your site.
Paths start with a slash, like /admin/ or /wp-admin/. A disallow of / blocks the whole site. Leave crawl-delay empty if you do not need it — many crawlers ignore it.
How it works
A robots.txt file is a plain text file that lives at the root of a website, at an address like example.com/robots.txt. Before a search engine crawler visits a site, it fetches this file and reads the rules inside. Each group of rules starts with a User-agent line that names the crawler the rules apply to — an asterisk means every crawler. Under it, Allow and Disallow lines list paths. A disallow rule tells the crawler not to visit matching pages; an allow rule makes an exception inside a disallowed folder. The crawler follows the most specific rule that matches a page, so a narrow allow can open one file inside a blocked folder.
Paths in robots.txt always start from the site root. Disallow: /admin/ blocks every page whose address starts with /admin/, while Disallow: /cart/ blocks the cart pages of a shop. A single Disallow: / blocks the entire site, which is why it is used on staging copies and sites under construction — and why it must never be left on a live site by accident. You can also block a single file, such as Disallow: /private.pdf, or a file type with a wildcard, such as Disallow: /*.pdf$. Wildcards work in most major crawlers, but keep them simple: a handful of clear rules beats a long list of clever patterns that nobody on the team understands six months later.
Two optional lines finish a good file. A Sitemap line points crawlers at your sitemap, for example Sitemap: https://example.com/sitemap.xml, which helps them find new and updated pages faster. A Crawl-delay line asks a crawler to wait a number of seconds between requests, which can protect a small server from being hammered — but note that Google ignores crawl-delay, so it is only a polite request to the crawlers that honour it. This generator builds the file with the standard layout: user-agent first, then the rules, then crawl-delay, then the sitemap lines at the end.
Disallow: /admin/
Allow: /admin/help.html
Sitemap: https://example.com/sitemap.xml
When your file is ready, press Download robots.txt and upload the file to your site's root folder, so it is reachable at yourdomain.com/robots.txt. Then check it in your browser: if you can see the file at that address, crawlers can too. Most search engines also offer a robots.txt tester in their webmaster tools, which is worth a quick check before you rely on the file. Remember that robots.txt is a request, not a lock. It keeps well-behaved crawlers out of the areas you list, but it does not hide a page from people, and it does not reliably keep a page out of search results — for that, use a noindex tag on the page itself.
Worked example
Imagine a small shop at example.com. The owner wants search engines to crawl the shop and the blog, but not the admin area, the cart, or the checkout, and wants crawlers to wait 10 seconds between requests. Here is the file this generator builds for that site.
Step 1: Choose "All crawlers (*)" as the user-agent, so the rules apply to every search engine.
Step 2: Add three disallow rules: /admin/, /cart/, and /checkout/.
Step 3: Add one allow rule: /images/, so product pictures stay crawlable even if they sit in a blocked area.
Step 4: Set crawl-delay to 10 and the sitemap to https://example.com/sitemap.xml.
Step 5: Press Download robots.txt and upload it to the site root.
The finished file reads exactly like this:
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Allow: /images/ Crawl-delay: 10 Sitemap: https://example.com/sitemap.xml
With this file in place, a crawler may visit the shop pages, the blog, and the images folder, but it will stay out of the admin area, the cart, and the checkout, wait 10 seconds between requests, and find the sitemap at the address given. Copy those same settings into the generator above and the live preview shows the identical file.
FAQ
What is a robots.txt file?
A robots.txt file is a small text file that sits at the root of a website and tells search engine crawlers which parts of the site they may visit. It uses simple rules: a user-agent line names the crawler, and allow or disallow lines list the paths it may or may not crawl. Search engines read it before they crawl, but it is a request, not a lock — well-behaved crawlers follow it, and it does not hide pages from people.
Where do I put the robots.txt file?
Upload it to the root folder of your website so it is reachable at yourdomain.com/robots.txt. Most hosting platforms and site builders have a place for it: static hosts let you upload the file directly, WordPress users can add it through an SEO plugin or by uploading it to the site root, and Cloudflare Pages and similar hosts serve a robots.txt file placed in the publish folder automatically.
What does Disallow: / mean?
Disallow: / under User-agent: * tells every crawler not to crawl any page on the site. It is useful for a site that is still under construction or a private staging copy, but it is a serious setting — leaving it on a live site can stop the site from appearing in search results. Use it only on purpose, and remove it before launch.
Does robots.txt keep a page out of search results?
Not reliably. Disallowing a page stops crawlers from visiting it, but if other sites link to it, a search engine may still list its address without a description. To keep a page out of search results, use a noindex meta tag on the page itself instead, or in addition to a disallow rule. Robots.txt controls crawling; noindex controls indexing.
Should I include a sitemap in robots.txt?
Yes, it is good practice. A sitemap line points crawlers to your sitemap file, which lists the pages you want found, so they can discover new and updated pages faster. Add one line per sitemap, for example Sitemap: https://example.com/sitemap.xml, and put the sitemap lines at the end of the file.