Three layers of AI search visibility: retrieving, reading, selecting

Rob Hoeijmakers recently wrote about caching and how AI tools helped him “understand something he had been working around for thirty years.” His technical piece on infrastructure contained a striking observation: “the audience had already changed.”

Site visitors are no longer predominantly human browsers. A growing share consists of machines, crawlers from OpenAI, Anthropic, Perplexity and Amazon. Hoeijmakers concluded that caching functions as “infrastructure for machine readers.” In a follow-up he noted: “On any given day, people are a minority.”

That observation is correct and deserves investigation. Who exactly are these machine readers? What do they do on your site? And most importantly: how do you measure whether what they read comes back in their answers?

Over the past months a three-layer framework has been developed that structures these questions: retrieval, reading, selection. Not for elegance, but because practitioners consistently work on the wrong layer or skip one.

Layer 1: Retrieval

The first layer is about reachability. Can AI bots actually discover and retrieve your site? This is largely established SEO territory, though the players have changed.

Your robots.txt must allow the right crawlers: GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Amazonbot. Each has its own user-agent strings and respects instructions, unlike many scraping bots. They check robots.txt and sitemap.xml first. Server logs show these are the most crawled paths, which indicates that these systems prioritise proper procedure.

llms.txt is an emerging standard that functions as a structured business card for AI systems. This text file in the root sums up key content, services and contact details in a machine-parseable format. It complements sitemaps, aimed specifically at how LLMs process information.

Caching belongs here too. Hoeijmakers' point is precisely this: caching strategies that target only human browsers miss the majority of your audience. AI crawlers make repeated requests. Solid caching provides fast, consistent responses without unnecessary server load.

Basic infrastructure counts: server uptime, response headers, TLS certificates, redirect chains. A 301 that redirects to a 302 that points to the final content requires three crawler requests instead of one. Multiplied across thousands of pages, this wastes irreplaceable crawl budget.

For most sites, layer 1 is well documented. Guides exist for robots.txt configuration, llms.txt implementation, and checklists. The problem is not a lack of information: it is that the discussion typically stops here.

Bots can now get in. What happens after that?

Layer 2: Reading

This layer gets too little attention. We know that AI bots visit our sites, server logs confirm this. But what they do on arrival, how they process content, what they extract or ignore, remains opaque to most site owners.

Let us be precise about what “reading” means for AI bots. There are two fundamentally different types.

First: pre-training crawlers. GPTBot, ClaudeBot, Google-Extended. These harvest web data for future model training. Today's reading becomes tomorrow's model. You do not control processing directly, but you influence what is found. Structured, factual, well-marked-up content processes more cleanly than unstructured slabs of text.

Second: real-time retrieval bots. ChatGPT-User, OAI-SearchBot, Perplexity-User. These fetch content when users ask questions. If someone types “what is AI search visibility” into ChatGPT and the model consults current sources, it sends requests to relevant pages. What it finds partly determines the answer to the user. This is real-time. This is measurable. This is where layer 2 becomes concrete.

A consistent pattern: the most crawled content is about AI search itself. Articles about llms.txt, AI crawlers, structured data, sitemaps, glossary terms.

This is no coincidence, it is self-proving. They write about AI search, so AI bots crawl it. They write about how machines process content, so machines process that content the most. It confirms exactly what layer 2 measures: bots crawl what they understand.

Structured, explicit content with clear headings, definition sentences and semantic markup parses more easily for machines than creative copy that leans on context and implicit references. “Core Web Vitals measure three metrics: LCP, INP and CLS” is directly usable for retrieval systems. “Your site's speed is hugely important and you have to do something about it” contains no extractable facts for machines.

This has direct content-strategy consequences. It is not only about what you write, but how you structure it. Logically nested HTML hierarchy. Schema.org markup that gives machines a factual layer above prose. Opening sentences per section that work as standalone answers. Comparison tables that are extraction-ready without full page comprehension.

The essence of layer 2: if you cannot see what AI bots read, you do not know whether layer 1 works. You cannot steer layer 3. You optimise blind.

The difference between “bots visit my site” and “I know what bots read on my site” is the difference between guessing and strategy.

Layer 3: Selection

Not all content that is read comes back in answers. That is the frustration of AI search visibility: you can be excellent on layers 1 and 2, perfectly reachable with clear structure, and still not be cited. What determines what AI engines select to cite?

This is the hardest, least transparent part. No server logs reveal the internal weighting when models decide which sources to cite. But there are patterns, ever more consistent as more is measured.

Domain authority plays a role, though differently than with Google. Sites that are cited frequently in training data, have backlinks from recognised sources, and consistently publish factual content, receive higher trust scores from retrieval systems. It is not direct “authority scoring” as in traditional SEO, but the mechanism runs in parallel: external validation leads to trust.

Expertise signals carry weight. Content with specific facts, concrete figures, source citations, acknowledged nuance receives more citations than generalised content. Research into Generative Engine Optimization showed that techniques such as adding statistics and citations can raise AI answer visibility by up to 40%, which is not subtle.

Freshness counts, especially for retrieval bots. Recently published pages are given preference over undated content or content unchanged for years at equal relevance. Logical: models try to give current information, and publication dates offer crucial signals.

Distinctiveness is perhaps the most underrated. When ten sites phrase the same information in the same way, the model has no reason to cite you over the others. Content that offers a unique perspective, presents its own data, introduces new frameworks, stands out in retrieval results. Generic “what is AI search visibility” content receives fewer citations than specific observations from original data. Models have enough generic definitions; they look for sources that add value.

Multi-source corroboration counts too. Multiple independent sources making identical factual claims raise model confidence. This means writing factual content that aligns with expert consensus, while adding your own data, perspective and application.

You measure layer 3 with prompt testing. Put business-relevant questions to ChatGPT, Claude, Perplexity. Note who is mentioned, in what order, in what language. For a mortgage adviser in Utrecht: test “beste hypotheekadviseur Utrecht” across three systems, note who appears and with what description. Repeat for “hypotheek voor zzp'er,” “annuïteiten- vs. lineaire hypotheek,” and every question clients ask. This is manual, labour-intensive work, but direct measurement of what works.

Layer 3 depends on layers 1 and 2. No technical reachability, no reading. No readability, no selection. No selection, no visibility in AI answers, however good your content is.