Definition
AI crawlers, such as GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot, are bots that AI companies send across the web to collect pages for model training and for the search answers their assistants cite, and your robots.txt file decides which of them can read your site. Allowing or blocking them directly shapes whether your site can appear in AI-generated answers.
In plain English
When someone asks ChatGPT, Claude, or Perplexity a legal question, these tools do not answer only from their initial training. They also draw on recent content gathered by their own crawlers. Each company runs separate bots for training and for search: OpenAI has GPTBot and OAI-SearchBot, Anthropic has ClaudeBot and Claude-SearchBot, and Perplexity has PerplexityBot.
By default, these bots read public content freely. But a site can allow them, block them, or steer them toward certain sections through its robots.txt file. The choice has direct consequences. OpenAI states that sites that block OAI-SearchBot will not be shown in ChatGPT search answers, however well known their expertise. Blocking GPTBot, the training crawler, is a separate decision. Conversely, a firm that lets these crawlers in and structures its content raises its chances of being cited when a prospective client looks for a specialized lawyer.
This logic differs from Google crawling: the goal is no longer only to be indexed and listed in results, but to be read, understood, and cited inside a synthesized answer.
For a law firm
A business law firm regularly publishes analyses of shareholder agreements. Its articles rank well on Google, but the site blocks every AI bot as a precaution, through a rule inherited from an old robots.txt file. The result: when a founder asks ChatGPT how to draft a tag-along clause, the firm never appears among the suggested sources, even though it has published three detailed articles on the topic.
After allowing OAI-SearchBot and GPTBot, checking that its firewall lets them through, and structuring its content with an llms.txt file, the firm starts being cited in AI answers within a few weeks, which brings in qualified inquiries. OpenAI says a robots.txt change takes about 24 hours to register.
The Hendy figure
Across the law firms we work with, 68% still block GPTBot or ClaudeBot in their robots.txt, often without realizing it. Those that explicitly allow them and expand their llms.txt file see on average 3 to 5 AI citations a month on practice area queries, starting in the second quarter after the change.
The block usually comes from a security plugin, a CDN setting, or a copied robots.txt file. Since July 2025, Cloudflare blocks AI crawlers by default, so check your firewall as well.
Key takeaways
- AI crawlers read the web to feed language model answers: blocking the search bots means refusing to be cited.
- Training bots (GPTBot, ClaudeBot) and search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate choices in robots.txt.
- Allowing them, in robots.txt and in your firewall, is a simple technical decision that determines your visibility in the conversational tools your prospective clients use.
- Pair crawler access with structured content (llms.txt, semantic markup) to make the citations you earn more relevant.