Robots.txt Explained for Beginners
Robots.txt explained for beginners starts with a simple idea: the file is a set of polite requests that tells search engine crawlers which parts of a site they may visit. Search engines send out automated programs, called crawlers or spiders, that travel the web reading pages. When a crawler arrives at a site, it usually checks for a file named robots.txt first. That file can welcome crawlers everywhere, keep them out of admin areas, or point them to the sitemap. It cannot force anyone to obey, but well-behaved crawlers respect it.
This guide is written for new website owners, bloggers, students and small businesses who manage their own sites. It covers what robots.txt does, how allow and disallow rules work, the common mistakes that damage visibility, and how to check a file safely. If you are still choosing an address for a new project, start with our domain search, then browse all free tools for utilities that support each step of running a site.
What Is Robots.txt?
Robots.txt is a plain text file that lives in the root folder of a website. Its address is always the domain plus /robots.txt, for example https://example.co.za/robots.txt. When a crawler visits a site, it reads this file before crawling pages. The file names which crawlers the rules apply to and which paths those crawlers should avoid. Everything in the file is public: anyone can open it in a browser, including competitors and curious visitors.
The file speaks in short blocks. Each block starts with a User-agent line that names the crawler, where an asterisk means all crawlers. Below that come the rules for that crawler. A Disallow line names a path crawlers should not visit, and an Allow line carves out an exception. A Sitemap line points to the XML sitemap so crawlers can find every page faster. Blank lines separate the blocks for readability.
Here is a small complete example that suits many starter sites:
User-agent: *
Allow: /
Sitemap: https://example.co.za/sitemap.xml
This tells all crawlers they may visit the whole site and shows them where the sitemap lives. Official documentation from Google Search Central on robots.txt describes the same format in full detail, including advanced matching. Beginners should learn the three basic lines above before touching anything more complex.
Why Robots.txt Matters
Robots.txt matters because crawlers have limited time for each site. A large store or forum can hold thousands of near-identical pages from filters, search results and session addresses. When crawlers spend visits on those repeats, they may reach the important product and article pages late or rarely. Sensible disallow rules steer crawlers away from the noise so the valuable pages get attention sooner.
The file also protects fragile parts of a site. Admin dashboards, login pages, cart checkouts and internal search results gain nothing from search traffic, yet crawlers will visit them if invited. Each wasted visit adds load to the server and clutters crawl reports with pages that should never rank. A few clear rules keep crawler traffic focused on public content that earns visitors.
For new sites, robots.txt works together with the launch checklist. Owners who verify ownership, submit sitemaps and check index coverage usually find crawler problems earlier. Our domain search can help confirm domain details during setup, and the full tools collection includes utilities for the checks that follow. Getting the basics right at launch avoids painful rework later.
How Allow and Disallow Rules Work
Rules match the start of a URL path. The line "Disallow: /admin/" blocks every address that begins with /admin/, including /admin/login and /admin/users/list. The line "Disallow: /drafts" blocks /drafts, /drafts/page1 and even /draftsman, because matching looks at the opening characters only. This prefix behaviour surprises many beginners, so test every rule against real addresses before saving the file.
Allow creates exceptions inside blocked areas. Suppose a media folder is closed but one public brochure inside it should be crawled. These two lines handle it:
User-agent: *
Disallow: /media/
Allow: /media/brochure.pdf
When rules overlap, the most specific matching rule wins. Here /media/brochure.pdf is longer and more exact than /media/, so crawlers may visit that one file while skipping the rest of the folder. Length of match decides, not the order of the lines, although keeping exceptions below the rule they modify makes the file easier for humans to read.
Two special values deserve attention. "Disallow: /" with nothing after the slash blocks the entire site, and it appears in many staging templates by default. Launching a site while that line is still present hides every page from polite crawlers. The opposite, an empty "Disallow:" line or "Allow: /", permits everything. Always read the file once more before a launch and confirm which of these two states it sets.
File placement and spelling are strict. The file must be named exactly robots.txt in lowercase and sit at the domain root. A file at /blog/robots.txt will never be read, because crawlers only check the root address. Only one such file exists per domain, including subdomains, so www and blog subdomains each need their own. After editing, open the live address in a browser to confirm the new version loads.
Examples
The table below shows common robots.txt patterns and what each one does. Read each row carefully before copying any pattern, since one extra slash can change the meaning of a whole site.
| Pattern | Meaning | Typical use |
|---|---|---|
| User-agent: * / Allow: / | All crawlers may visit everything | Standard live site default |
| User-agent: * / Disallow: /admin/ | All crawlers skip the admin folder | Dashboards and login areas |
| User-agent: * / Disallow: /search / Disallow: /cart | Skip internal search and cart pages | Stores and forums |
| Disallow: /media/ / Allow: /media/brochure.pdf | Block a folder except one file | Private files with one public item |
| User-agent: * / Disallow: / | Block the entire site | Staging copies only, never live sites |
| Sitemap: https://example.co.za/sitemap.xml | Points crawlers at the sitemap | Almost every site should include this |
Consider a small business site with a blog, a price list and a client login area. A suitable file welcomes crawlers to the public pages, closes the login folder, and lists the sitemap. The owner tests /login and two blog addresses after saving, confirms the login is skipped and the articles are visited, and only then moves on. Ten minutes of testing replaces months of wondering why pages rank slowly.
Now consider the staging accident. A developer builds the new design at staging.example.co.za with "Disallow: /" so Google ignores the copy. At launch the file is copied to the live domain along with everything else, and the block goes live too. Traffic fades over the following weeks because crawlers stop visiting. Checking the root robots.txt address on launch day would have caught the single line responsible.
Common Mistakes
Thinking robots.txt hides pages or keeps them private. This is the most dangerous misunderstanding. Disallowed pages can still appear in results if other sites link to them, and the file itself advertises private paths to anyone who reads it. Real protection needs a password, a noindex tag or removal of the page. Never list truly sensitive addresses in robots.txt and assume they are safe.
Blocking CSS and JavaScript files. Older templates sometimes disallow theme, style or script folders to save crawl budget. Modern crawlers need those files to render pages correctly, and blocking them can leave Google seeing a broken layout. Unless a specialist advises otherwise, keep site assets fully crawlable and spend the budget rules on low-value pages instead.
Copying a full block from a staging site to a live site. Staging files often start with "Disallow: /" for the whole domain. Carrying that line to production blocks every page at once. Treat robots.txt as a launch checklist item: read the live file after every deployment and confirm it permits the pages the business needs.
Using robots.txt to fix duplicate content. Blocking one copy of a duplicated page does not combine ranking signals or pick a preferred version. Canonical tags and redirects solve duplication, while robots.txt only controls visits. Use each tool for its own job rather than forcing one file to do everything.
Guessing at syntax instead of testing. Wildcards, dollar signs for end-of-path matching, and crawl-delay lines behave differently across crawlers, and some crawlers ignore them entirely. After any edit, test affected addresses with the crawler testing feature in Search Console. A rule that looks right but was never tested is only a hope, not a control.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a small text file on a website that tells search engine crawlers which pages they may or may not visit. It sits at the root of the site, for example at example.co.za/robots.txt.
Does robots.txt hide pages from Google?
No. Robots.txt only asks polite crawlers not to visit a page. It does not block people, hide URLs from results in every case, or protect private content. Use passwords or noindex tags for real protection.
What do allow and disallow mean in robots.txt?
Disallow tells crawlers not to visit matching paths, while allow grants an exception inside a disallowed area. Rules are matched from the top down, and the most specific matching rule wins.
Why should I not block CSS and JavaScript files?
Search engines need CSS and JavaScript to render pages the way visitors see them. Blocking these files can cause poor rendering and weaker understanding of the page content.
Where do I put the robots.txt file?
Place one file named exactly robots.txt in the root folder of the domain so it loads at yourdomain/robots.txt. Crawlers do not check subfolders for it.
How do I check that my robots.txt works?
Open the file in a browser at yourdomain/robots.txt, read each rule line by line, and test important URLs with the crawler testing tool in Google Search Console before relying on the file.
Related Tools
- Domain Search, look up domain details when setting up or auditing a new site.
- Character Counter, check the length of titles and descriptions on pages you keep crawlable.
- Word Counter, review thin pages before deciding whether crawlers should spend time on them.
- How DNS works, understand the address system that sits behind every domain you manage.
- All Tools, browse the full collection of free website, SEO and everyday utilities.
Conclusion: Robots.txt is a small public file with real influence: it guides polite crawlers toward valuable pages, away from admin and duplicate areas, and on to the sitemap. Allow and disallow rules match path beginnings, specific rules beat general ones, and the file must live at the domain root under its exact name. It never hides private content and should never block CSS or JavaScript, so test every change before trusting it. For more plain-language explainers, browse our guides.