Alli AI analyzed 24.4 million proxy requests across 78,000
pages over 55 days from January to March 2026. The finding: AI crawlers collectively made 213,477 requests versus 59,353 from traditional search crawlers like Googlebot. That is a 3.6x ratio confirmed by real traffic data, not theoretical estimates.
The detail most marketers are missing is that OpenAI runs two completely separate crawlers with opposite purposes. GPTBot is the training crawler. It collects data to improve OpenAI's models. ChatGPT-User is the retrieval crawler. It fetches pages in real time when users ask ChatGPT questions. Blocking GPTBot keeps your content out of future training data but does not affect whether ChatGPT cites you today. Blocking ChatGPT-User makes you invisible in ChatGPT's live answers entirely.
Semrush separately confirmed that LLM bots as a group now crawl 3.6x more than Googlebot across their network. Cloudflare's 2025 analysis showed ChatGPT-User requests surged 2,825% year over year during 2025 alone.
The crawl strategy question is no longer theoretical. It is a business decision with direct revenue implications.
more than almost any content optimization you can do.
Leave a Comment
Your email address will not be published. Comments are moderated before appearing.