Robots.txt Tester & Meta Robots Decoder

Check whether a page is actually crawlable, in plain English.

Robots.txt Tester

Try "Googlebot", "Googlebot-Image", or "*"

e.g. /blog/my-post or /files/report.pdf

Allowed

Meta Robots Tag Decoder

Paste the value from <meta name="robots" content="..."> or an X-Robots-Tag header.

    This tool mirrors Google's documented matching rules (longest-matching rule wins, Allow beats Disallow on a tie), but crawlers occasionally interpret edge cases differently, and some directives (like crawl-delay) aren't honored by Google at all despite being valid syntax for other crawlers. Always confirm important pages with Search Console's own URL Inspection tool before relying on this alone.

    Catch crawl issues before they cost you traffic

    Weeklift's weekly digest is built from your real Search Console data, so accidental blocks or indexing issues show up alongside your ranking changes, not buried in a separate report.

    Join the Waitlist

    Launching soon. No spam, just one email when we go live.

    How does the matching logic work?

    Rules are grouped by the User-agent lines that precede them. When testing a crawler, this tool first looks for a group that names it exactly (case-insensitive), and only falls back to the wildcard * group if no exact match exists, matching real crawler behavior. Within the matching group, every Allow and Disallow rule is compared against the path, and the longest matching rule wins, regardless of which type it is. If an Allow and a Disallow match with equal length, Allow wins, this is what lets you block a whole folder but carve out one allowed subfolder inside it. Wildcards (* for any characters, $ to anchor the end of the URL) are both supported.

    See how a bad robots.txt rule fits into the bigger picture: 8 common SEO mistakes.

    Other free tools: CTR-by-Position Estimator · SERP Snippet Preview · Open Graph Previewer · Canonical Tag Checker · Schema Markup Generator · Heading Structure Checker · Readability Scorer · Search Console RegEx Generator · Bulk SERP Truncation Checker · Schema Markup Validator · Keyword Intent Classifier · XML Sitemap Validator · Robots.txt Generator · Hreflang Tag Checker · Keyword Multiplier