Robots.txt Generator

Build a valid robots.txt with a form, no syntax to memorize or get wrong.

Start from a preset, or build your own

Added once at the end of the file, regardless of how many groups you have above.

Generated robots.txt


        


        

            robots.txt controls crawling, not indexing. A disallowed page can still show up in search results (usually without a snippet) if other sites link to it, use a noindex meta tag or X-Robots-Tag header if you actually need a page kept out of search results. Google ignores Crawl-delay entirely, though some other crawlers respect it. This file only works when placed at your domain's root (yoursite.com/robots.txt), not in a subdirectory. After publishing, check it with our own Robots.txt Tester.
        


        

            
A blocked page is easy to miss

            
An accidental Disallow rule can quietly cut off crawling for weeks before anyone notices the traffic drop. Weeklift's weekly digest surfaces pages with falling impressions alongside your real Search Console data.

            Join the Waitlist
            
Launching soon. No spam, just one email when we go live.

        


        

            How do multiple user-agent groups work?
            
Crawlers look for the most specific group that names them by their exact user-agent string (like Googlebot) and use those rules instead of the wildcard * group. If no group names them specifically, they fall back to the * group. Rules from different groups aren't combined, a crawler follows exactly one group's rules, whichever is the best match for it.

        


        

            Other free tools:
            CTR-by-Position Estimator ·
            SERP Snippet Preview ·
            Bulk SERP Truncation Checker ·
            Open Graph Previewer ·
            Robots.txt Tester ·
            XML Sitemap Validator ·
            Canonical Tag Checker ·
            Schema Markup Generator ·
            Schema Markup Validator ·
            Heading Structure Checker ·
            Readability Scorer ·
            Search Console RegEx Generator ·
            Keyword Intent Classifier ·
            Hreflang Tag Checker ·
            Keyword Multiplier