Robots.txt Tester
Paste a robots.txt content, enter a URL path + user-agent, see whether the crawler is allowed or blocked. Follows longest-match-wins rule (Allow beats Disallow on tie). Supports * wildcards and $ end-of-line anchors.
GooglebotWhat powers the Robots.txt Tester
Parse robots.txt with multiple UA groups
Test any path + UA combo
Longest-match-wins rule (Allow > Disallow on tie)
* wildcards and $ end-anchor
Sitemap reference detection
Crawl-delay detection
Why the Robots.txt Tester is different
Browser-native
The Robots.txt Tester runs entirely in your browser. Input is processed locally — never uploaded, never logged, never cached anywhere outside your device.
No artificial limits
No daily quotas, no character ceilings, no "upgrade for more" walls. Every feature is the complete feature — the same on the first use as the thousandth.
Production-grade quality
Built to the same engineering bar as paid SaaS tools — accurate algorithms, audited logic, responsive design and accessibility-tested interactions.
Common use contexts
- Pre-deployment verification
- Debugging un-indexed URLs
- Testing bot blocks
- Migration QA for robots rules
Private and secure
Zero upload
All processing happens in your browser. Input is never transmitted, logged or cached.
Works offline
Once the page loads, the tool runs without an internet connection. No network calls happen during use.
No tracking
No accounts, no cookies for tool state. Only aggregate analytics count visits at the page level.