robots.txt Generator

Build your site's robots.txt file: pick the user-agent, add Allow and Disallow rules, point to your sitemap and download the file ready to upload.


      

      
      
    


    
🔒 Everything happens in your browser: no data is ever sent to a server.

  


  

    
What a robots.txt file does

    
The robots.txt file lives at the root of your site (e.g. https://www.yoursite.com/robots.txt)
    and tells search engine crawlers which paths they may visit. The Disallow directive
    asks bots not to crawl a path, while Allow carves out an exception inside a blocked
    folder (say, you block /admin/ but allow /admin/public/). One important caveat: robots.txt
    controls crawling, not indexing — a blocked page can still show up on Google if other sites
    link to it. To truly keep a page out of search results, use a noindex meta tag instead.

    
The rules most sites actually use

    
Almost every site blocks private areas such as /wp-admin/ or /admin/, shopping carts and
    internal search pages with query parameters, which only waste crawl budget. Adding a
    Sitemap: line with the full URL helps Google and Bing discover all your pages right
    away. Keep in mind that paths are case-sensitive, and that each block of rules applies to
    the user-agent declared above it: with * you address every bot at once.