How to use
Paste the contents of your robots.txt (https://your-domain/robots.txt) on the left, or open the file.
Choose a crawler (User-agent): Googlebot, Bingbot, or AI crawlers such as GPTBot and ClaudeBot.
Enter URLs or paths to test, one per line, to see allowed/blocked and the line number of the rule applied.
Review “Findings” for mistakes and directives Google ignores; “Go to line” selects the line to fix.
Check the overview below to see whether major search engines and AI crawlers can fetch your home page.
What it does
RFC 9309 matching
The longest matching rule wins and Allow wins ties. User-agents are matched case-insensitively and multiple groups for the same crawler are combined. Supports * and $ and percent-encodes non-ASCII paths.
See the rule and line that decided
Every result shows the Allow/Disallow rule applied, its line number and the User-agent group used. Click to select that line.
Catches common mistakes
Rules before any User-agent, misspellings (Dissallow), missing colons, relative Sitemap URLs, Noindex and Crawl-delay that Google ignores, files over 500 KiB and HTML served instead of robots.txt.
AI crawler overview
See whether Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot and other AI crawlers can fetch your home page, next to the search engines.
Use cases
- Make sure admin and internal search pages are blocked before launch
- Find out whether robots.txt is why pages are not being indexed
- Review your allow/deny settings for AI crawlers such as GPTBot and ClaudeBot
- Try out wildcard (*) and end-anchor ($) rules before deploying them
- Check the impact of a robots.txt change before publishing it
Supported formats
Input
Capability matrix
| Feature | Support | Notes |
|---|---|---|
| Allowed/blocked per URL (RFC 9309) | Supported | — |
| Rule and line number applied | Supported | — |
| Syntax and unsupported-directive checks | Supported | — |
| Search engine and AI crawler overview | Supported | — |
| Crawler-specific extensions (e.g. Yandex Clean-param) | Partial | Flagged, not used for matching |
| Fetch robots.txt from a URL automatically | Not supported | Paste the contents instead |
| Check page-level noindex (meta robots) | Not supported | — |
Limits & notes
- robots.txt is not fetched for you. Open https://your-domain/robots.txt in your browser and paste the contents.
- Up to 2,000 URLs per test and files up to 2 MB.
- User-agent matching follows Google (e.g. Googlebot-Image follows the Googlebot group when it has no group of its own). Other crawlers may differ in details.
- robots.txt controls crawling, not indexing. To keep a page out of search results, use noindex on the page.
Privacy & safety
Your robots.txt and test URLs are processed in your browser only. Nothing is sent to or stored on a server.
FAQ
What happens when both Allow and Disallow match?
Does blocking in robots.txt keep a page out of Google?
How do I block ChatGPT or Claude crawlers?
Does Crawl-delay work?
Is what I enter sent anywhere?
Related tools
XML Sitemap Generator & Validator
Paste a list of URLs to create a correctly written sitemap.xml, or paste an existing sitemap to find the errors that stop Google from reading it — with line numbers.
Structured Data / JSON-LD Studio
Build JSON-LD for 13 common types with forms that separate required from recommended fields, then copy it. Paste JSON-LD or HTML to extract every script, @graph and array and see problems from the exact line of a syntax error to missing Google rich result requirements. It is not Google’s official tool.
HTML Inspector
Paste a page’s HTML to review SEO and accessibility basics at a glance: heading structure, meta tags, Open Graph, image alt text, links and structured data — with a preview of how it may look in search and on social media.
Last updated: