AI bots are now 4× bigger than Google.

Period: 9 days on handler.gg. AI traffic share: 81%.

AI bots are not a new channel, they are a new search paradigm that quietly displaces the old one.

The bots broken down

Amazonbot: 3,470 requests (66.8%)

The largest crawler, used for Amazon Q, Alexa and Bedrock training. Every court page crawled exactly 3 times, which points to systematic indexing.

ChatGPT-User + OAI-SearchBot: 1,328 requests

Real-time fetching when ChatGPT users ask questions. Around 147 requests per day, which represents direct prospect interaction moments.

PerplexityBot + Perplexity-User: 147 requests

Builds an independent search index. Users are typically research-minded professionals and students, disproportionately important for B2B markets.

ClaudeBot: 96 requests

Anthropic's training crawler that first follows robots.txt, then content pages.

GPTBot: 82 requests

OpenAI's training bot for future model versions.

Critical pattern

These bots fetch /robots.txt and /sitemap.xml a lot: 67 + 48 hits respectively. These crawlers look for permission and structure, and crawl more effectively when they are welcomed.

The analytics gap

Google Analytics 4 filters out bots by default, which creates a visibility problem. Most marketing teams are unaware that 80% of crawl traffic comes from AI systems.

Measuring it yourself

With server access:

zcat /var/log/nginx/jouwsite.com-access.log.*.gz | \
  cat - /var/log/nginx/jouwsite.com-access.log | \
  grep -ohE "(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Bytespider|Amazonbot|Google-Extended)" | \
  sort | uniq -c | sort -rn

Without server access:

• Cloudflare: AI Crawl Control dashboard
• WordPress: Wordfence Live Traffic logging
• Most hosting providers offer SSH access to nginx/Apache logs

Three strategic choices

Choice 1: Allow or block?

Recommendation: allow most AI crawlers while you block sensitive pages (/admin, /login, /api).

Choice 2: Optimise for LLM extraction

• Use clear semantic HTML structure (h1, h2, h3)
• Make definitive statements over marketing language
• Implement Schema.org markup (LocalBusiness, Product, FAQPage)
• Create an llms.txt file in the root directory

Choice 3: Clean GA4 analytics

Mark AI bot IP ranges and user agents as known bots to prevent data pollution.

Results on handler.gg

Changes made:

1. robots.txt updated to allow AI crawlers
2. llms.txt file created with key pages and coverage
3. Schema.org markup implemented per club page
4. Internal linking restructured per city and type
5. Meta descriptions rewritten from marketing language to factual statements

Result: Citation rate rose from 12% to 38% within 6 weeks for “padel boeken Nederland” queries, a 3× improvement.

Applicability

This pattern applies universally to:

• Location-based content (services, medical practices, restaurants)
• Comparison content (“best X in Y”)
• FAQ and how-to content