How to Write a robots.txt File Without Blocking Your Site

robots.txt tells cooperating crawlers which paths they may fetch. It is not a security boundary and it is not the same as noindex. A single mistaken Disallow: / can keep crawlers off the whole site while the pages still open in a browser.

The Haivix Robots.txt Generator builds one User-agent block (*, Googlebot, Googlebot-Image, or Bingbot), Allow/Disallow paths, and absolute Sitemap URLs. You can copy the text or download robots.txt. It does not upload to your host or prove how Google will behave.

Presets and empty Disallow

Presets cover allow-all, block common private paths (/admin/, /account/, /checkout/, /search/), block everything (/), or custom lines. Paths must start with /. Sitemaps must be absolute http: or https: URLs.

If you leave Allow and Disallow empty, the tool emits an empty Disallow: line—the usual allow-all convention for that agent. It warns if you disallow / with no Allow exceptions.

Crawl rules are not indexing rules

Blocking a URL in robots.txt can stop crawling; it does not reliably remove a URL already indexed. Use meta robots or headers when you need noindex—pair this with the Meta Tag Generator. Share previews are separate again: Open Graph Image Preview.

The generator emits a single User-agent section per run—not multiple agents with different rules in one file. Crawl-delay, Host, and Clean-param are out of scope.

Deploy checklist

Open the Robots.txt Generator, choose a preset or custom paths, add sitemaps, then copy or download and place the file on your host.

• File will be served at https://your-domain/robots.txt.
• You did not disallow / unless you meant to block the site.
• Private areas are disallowed; public content remains allowed.
• Sitemap URLs are absolute and already live.
• Auth-protected URLs still need real authentication—not only robots.txt.
• After publish, you fetched /robots.txt yourself and spot-checked a disallowed path.