Robots.txt Generator
Build a robots.txt file with allow, disallow, and sitemap rules
Everything runs in your browser. Your data never leaves your device.
Related tools
How to use
Set the user-agent, add Allow and Disallow paths, and optionally provide your sitemap URL. The robots.txt content appears on the right, ready to copy.
Set the user-agent
Use * to target all crawlers, or enter a specific bot name such as Googlebot or Bingbot. Multiple user-agent blocks can target different bots with different rules.
Add Allow and Disallow paths
Disallow paths tell crawlers not to access those paths. Allow paths explicitly permit access even within a disallowed subtree. Use a trailing slash for directories — /admin/ blocks the whole admin section.
Add your sitemap URL and copy
Paste your sitemap URL in the Sitemap field so crawlers can discover all your pages. Copy the output and save it as robots.txt in the root of your web server.
Examples
Blocking admin and staging paths
Set user-agent to *, add /admin/ and /staging/ as Disallow paths, and leave / in Allow. This tells all crawlers they can index the public site but should not enter the admin or staging sections.
Blocking all crawlers on a staging server
Set user-agent to *, set Disallow to /, and remove all Allow paths. This blocks every crawler from indexing any page — suitable for a staging or development server you do not want appearing in search results.
About this tool
Generate a well-formed robots.txt file by specifying the user-agent, allowed paths, disallowed paths, and an optional sitemap URL. The output updates live as you add or remove rules. Paste the result into the root of your web server or repository. Nothing is uploaded — generation runs in your browser.