Blocking AI crawlers in robots.txt: does it protect content or cost citations?

A lot of the advice on blocking AI crawlers mixes up two different things. Blocking GPTBot or ClaudeBot mostly stops training and some answer-engine fetching. Google is harder, because Googlebot serves regular search and feeds the AI features too, so you can't opt out of AI Overviews without leaving search. Google-Extended covers Gemini training but not AI Overviews.

For a site that lives on organic leads, I'd keep the search crawlers open and only block training bots if you have a real reason. People tend to skip the measurement cost. Once you block them, you can't see whether ChatGPT or Google's AI cites you, so brand-citation tracking turns into guesswork. Has anyone seen a traffic or citation change after blocking, or is it still mostly theory?

Agreed on the split. The detail people miss is that OpenAI runs three separate agents: GPTBot for training, OAI-SearchBot for ChatGPT's search index, and ChatGPT-User for on-demand fetches when someone asks it to browse. Blocking GPTBot alone shouldn't cost you citations, but blocking OAI-SearchBot will. So if citations matter, write the rules per user agent instead of one blanket "block OpenAI" line.

For measurement, GA4 won't show any of this because bots don't fire the tag. You'd need server or CDN logs filtered by user agent to see what's actually fetching the site.

Anyone can fake a user agent string, so a "GPTBot" line in your logs doesn't prove the request came from OpenAI. Before you make blocking decisions from those numbers, check the IP against the ranges the vendor publishes, or run a reverse DNS lookup for Googlebot.

Verifying matters most when the rule is enforced at the edge. Robots.txt is voluntary, so a spoofer ignores it either way, and a UA check there mostly just tells you who's polite. A firewall or CDN rule that matches on user agent is a different story. If it's loose, a fake "Googlebot" string can slip through, and a sloppy "block anything with bot in the name" rule can catch real crawlers you wanted to keep. So it depends on where the block lives. For robots.txt, I'd put the effort into the per-agent rules Jun mentioned. For WAF rules, verify the IP first.

Agreed, and the edge rule is where I'd be most careful. Crawlers cache robots.txt, so changes can take up to a day to land. The bigger problem is the file's status code. Google treats a 5xx or unreachable robots.txt as a full disallow for the host, and it can pause crawling until the file comes back. So if your CDN or WAF serves a bot challenge or a 503 on /robots.txt to unverified agents, Googlebot could stop crawling your whole site even though the rules in the file are fine. 4xx responses are handled more leniently, so the real failure mode is mostly 5xx.

Before touching any blocking rules, test what the edge returns for /robots.txt with a Googlebot user agent and with a random one, and make sure both get the same 200 body. After that, the per-agent rules Jun described are the only thing deciding what gets crawled.

Agreed, and one detail matters here: hit /robots.txt over plain HTTP with no cookies and no JS challenge, because some WAFs only challenge the first request from a new IP and then wave the rest through. Run the check twice, a few minutes apart, and compare the status codes. A single 200 can hide a challenge that kicks in later.

Agreed on the repeat test. For a local business I'd guess thin listing info hurts citations more than robots.txt does, since AI answers often lean on Business Profile and directory data. Has anyone seen a clinic get cited from its listing while its own site was blocked?