Free AI Crawler Checker: Is GPTBot Blocked on Your Site?

ChatGPT, Claude, Perplexity and Gemini can only mention your business if their crawlers are allowed to read your website. Many sites block them by accident, through an old robots.txt rule, a security plugin, or a CDN default. Paste your robots.txt below to see which AI crawlers can reach your homepage and which ones are blocked. It is free, needs no signup, and runs in your browser.

How to use the AI Crawler Checker?

  1. Open yourwebsite.com/robots.txt in a new browser tab.
  2. Select all the text and copy it.
  3. Paste it into the box above, replacing the sample text.
  4. Read the results. Each AI crawler is marked Allowed, Blocked, or Partly restricted, and the tool tells you whether the result comes from a rule written for that bot or from your general rules.
  5. If something is blocked that should not be, use the ready-made robots.txt block under the results and add it to your file.

The checker tests access to your homepage (the root path). If a bot is marked Partly restricted, it can reach your homepage but some folders, such as /admin/, are closed to it.

What is an AI crawler?

An AI crawler is an automated program that visits web pages so an AI company can use the content. Some crawlers collect text to train models. Others fetch pages so an AI search tool can show answers with sources. A third type visits a page only when a person asks the AI to open it. This difference matters. Blocking a training crawler and blocking a search crawler are very different decisions, and mixing them up is the most common mistake we see.

Which AI crawlers does this tool check?

OpenAI

  • GPTBot: collects content for training OpenAI models.
  • OAI-SearchBot: finds pages to show in ChatGPT search results.
  • ChatGPT-User: visits a page when a ChatGPT user asks for it.

Anthropic (Claude)

  • ClaudeBot: collects content for training Claude models.
  • Claude-SearchBot: helps Claude search results.
  • Claude-User: visits a page when a Claude user asks for it.

Perplexity

  • PerplexityBot: builds the index behind Perplexity answers.
  • Perplexity-User: visits a page when a user asks for it.

Google, Apple, Meta and others

  • Google-Extended: a robots.txt control that tells Google whether your content can be used for Gemini. It does not affect your Google Search rankings.
  • Applebot-Extended: tells Apple whether your content can be used to train its AI models.
  • Meta-ExternalAgent: Meta’s crawler for AI training.
  • Amazonbot: Amazon’s crawler, used for services such as Alexa.
  • Bytespider: ByteDance’s crawler.
  • CCBot: Common Crawl, a public dataset that many AI models learn from.

Should you allow or block AI crawlers?

There is no single right answer, but here is a practical way to decide.

Allow AI search and user crawlers if you want visibility. If you want ChatGPT, Claude or Perplexity to recommend your business, cite your articles, or send visitors to your site, their search crawlers must be able to read your pages. For most service businesses, local businesses and agencies, this is the better choice.

Decide separately about training crawlers. Some publishers block training bots to keep their content out of model training, while still allowing search bots so they appear in AI answers. The “Allow AI search, block AI training” option in the tool creates exactly that setup.

Block everything only with a clear reason. Blocking every AI crawler protects your content from reuse, but it also removes you from a growing channel where people ask for recommendations.

One limit to keep in mind: robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it cannot force every bot to comply.

Common reasons AI crawlers get blocked by mistake

  • A blanket rule. A line like Disallow: / under User-agent: * blocks every crawler, including AI search bots.
  • Copied robots.txt templates. Some templates and plugins add AI bot blocks by default.
  • Security plugins and CDNs. Firewalls and bot-protection settings can block AI crawlers even when robots.txt allows them. If the checker shows Allowed but AI tools still cannot read your site, check your security plugin and CDN settings.
  • Old staging settings. A site launched with a “discourage search engines” setting can keep restrictive rules for months.
Scroll to Top