Free Robots.txt Generator
This robots.txt generator builds a crawler-rules file for your site root: which paths search engines may crawl, where your sitemap lives, and optional crawl-delay. Tick the rules you need and copy the file to example.co.za/robots.txt. Nothing is uploaded.
How it works
crawlers read robots.txt before crawling. Disallow keeps crawlers out of utility paths; Allow re-includes specific files. The sitemap line points crawlers at your URL list. Note: robots.txt is a request, not access control, and it does not hide pages from people who know the URL.
Example
| Rule | Meaning |
|---|---|
Disallow: /admin/ | Crawlers skip the admin area |
Sitemap: https://…/sitemap.xml | Crawlers find the URL list |
Frequently asked questions
Where do I put robots.txt?
At the site root, e.g. example.co.za/robots.txt. Crawlers only check that location.
Does robots.txt hide pages?
No. It asks well-behaved crawlers not to crawl; anyone with the URL can still open the page. Use passwords for private content.
Should I block CSS and JavaScript?
No. Google needs them to render pages. Blocking them can harm rankings.
What is crawl-delay?
A hint asking crawlers to wait between requests. Google ignores it; some smaller crawlers honour it.
Is my input uploaded?
No. The file is built locally in your browser.
Related tools
- Sitemap Generator, build the file it references.
- Meta Tag Generator, page-level tags.
- Guide: Robots.txt Explained