BOTrento LAB · free
Does your website let AI crawlers in?
This is not “indexing in every LLM” — each system has its own crawlers and rules. We show for each one whether it is allowed, blocked or cannot be verified, and keep search crawlers (e.g. OAI-SearchBot) apart from training crawlers (e.g. GPTBot).
What the check measures
- 14 crawlers: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (Claude-SearchBot, Claude-User, ClaudeBot), Perplexity, Google (Googlebot, Google-Extended), Bing, Apple, Common Crawl
- robots.txt rules (most specific group, longest match) and the server's real response to the crawler's User-Agent
- Text in the HTML without JavaScript, obfuscated email (Cloudflare), JSON-LD Organization/LocalBusiness
- llms.txt and llms-full.txt, sitemap with lastmod
What the check cannot tell you
- Whether a system already has you in its index — operators do not disclose that
- Presence in Bing: it cannot be verified honestly without a paid API (the Bing Search API was retired in 2025), so we skip it
- IndexNow: the key can live at any URL, we cannot verify it from outside
- How a CDN treats the real crawler coming from its own IP addresses
Free scope
The given page, robots.txt, llms.txt, sitemap, 12 requests with crawler User-Agents.
Input
Website address.
What the output looks like
priklad-firma.cz
AI crawler access (sample)
OAI-SearchBot search · OpenAI
PerplexityBot search · robots.txt disallows
Claude-SearchBot search · Cloudflare returns 403 for the User-Agent from our address
GPTBot training · your decision, not scored
Method — how we measure
- Allowed = robots.txt allows and the server returns 200. Blocked = disallowed in robots.txt, or the server returns 401/403/429 only to the crawler. Not verified = Cloudflare/CDN blocks the User-Agent from our address but may let the real crawler in from its IPs.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) are not scored: blocking them is your decision and does not directly affect AI search.
- llms.txt is a proposed standard; major AI providers have not confirmed using it — so it has little weight.
1.
Enter the input
Your website address or your numbers. No phone number, no sign-up.
2.
Three findings free
See the result and the three most important findings right on the page.
3.
Report with evidence
We email a detailed report with evidence and a fix plan.
4.
Measure again
Re-run the check after changes, or order an automatic re-check in 30 days.
Fixed-scope fix and monitoring
We configure robots.txt, the CDN and firewall the way you decide (search yes, training your call) and then monitor access regularly. We confirm the price after a consultation.
Frequently asked questions
Should I allow GPTBot?
That is your decision. GPTBot collects data for model training. For appearing in ChatGPT search, OAI-SearchBot and ChatGPT-User matter.
Why are some crawlers “not verified”?
Cloudflare and other CDNs often verify crawlers by IP address. Our check sends their User-Agent from our address, so we see the block but cannot know if the real crawler would pass.
Do I need llms.txt?
No. It does no harm and may help, but no major AI provider has confirmed reading it. Content readable without JavaScript matters more.
Do you check presence in Bing?
No. Without a paid API it would require scraping search results, which breaks Bing's terms. Check it in Bing Webmaster Tools.
More BOTrento LAB tools
Lost enquiries calculator
How much revenue and margin may be lost on enquiries without a quick reply or follow-up. Three scenarios from your numbers.
Follow-up check
How many quotes have no next step and who owns the reminder. A questionnaire, optionally an anonymised export.
Automation ROI calculator
Three payback scenarios after human review, running and implementation costs — and the first suitable process.
SEO check
Technical, content, indexing and speed for the homepage and a sample of sitemap pages.
Google AI Overviews readiness check
Technical readiness for Google's AI Overviews and — separately — observed citations in AI with Google Search.
AI answer visibility check
Do ChatGPT, Perplexity and other AI name you when a customer asks about a service like yours? Exact questions, answers and links.
Try BOTrento on your own mailbox
14 days free, no card required. Nothing is sent without your approval.