Robots.txt Generator
Build crawl rules for search engines without syntax mistakes.
Upload the file to the root of your domain so it answers at /robots.txt. Robots rules are advisory: they guide well-behaved crawlers but do not protect private pages, which need real access control.
About the Robots.txt Generator
A robots file tells crawlers which paths they may fetch. Its syntax is small but unforgiving, and a misplaced slash can hide an entire site from search. This generator assembles a valid file from your choices and explains what each line does.
It also warns about the most damaging mistake: a blanket disallow rule that blocks everything, which is occasionally intentional and far more often an accident.
How to use the Robots.txt Generator
- 1Choose whether crawling is allowed by default.
- 2Add the paths you want to disallow, one per line.
- 3Optionally add allow exceptions and a crawl delay.
- 4Add your sitemap URL if you have one published.
- 5Copy or download the generated robots.txt file.
Features
- Allow-all or block-all starting point
- Multiple disallow and allow rules
- Optional crawl-delay directive
- Sitemap declaration
- Warning when the rules block your whole site
Why people use it
- Keep admin and duplicate paths out of search results.
- Avoid accidentally blocking your own pages.
- Publish a syntactically valid file the first time.
Common use cases
- Hiding a staging path from crawlers.
- Blocking internal search result pages.
- Declaring a sitemap location for a new site.
Privacy and your data
Everything you type or open here is processed by your own browser. No file, text or result is uploaded to a server, stored in a database or shared with anyone, and closing the tab clears it completely.
Frequently asked questions
Related tools
Meta Tag Generator
Build title, description, Open Graph and Twitter tags ready to paste.
URL Slug Generator
Turn any title into a clean, readable URL slug.
Keyword Density Checker
Analyse word and phrase frequency in your page copy.
PDF Merger
Combine several PDF files into one ordered document.
PDF Splitter
Split a PDF into separate files or a chosen page range.
PDF Compressor
Rebuild a PDF to reduce its file size for email and uploads.