Notes on search theory

Classical search ranks documents; generative systems compose answers and then attribute them. That inserts a second selection step — citation — with its own objective function. A document now competes not just to rank, but to be the source a model quotes when synthesizing an answer. The two objectives overlap but are not identical, which is why pages can rank well yet never be cited, and vice versa.

The observable pattern favors extractability and grounding: claims stated as discrete, attributable propositions; entities that are unambiguous; and information that is not already saturated in the model's training distribution. A model citing a source is effectively offloading epistemic responsibility — documents that make verification cheap appear to be preferred.

Information gain is the idea that a document's value is partly measured by what it adds relative to the corpus already indexed for that topic — not by how well it restates consensus. It reframes content strategy as a contribution problem: the question is not 'is this page good?' but 'does the index become more informative with this page in it?' Generative systems sharpen the incentive, since novel information is disproportionately likely to be quoted.

More a change of emphasis than a new field. The fundamentals — entity clarity, semantic structure, verifiable claims — serve both classical ranking and citation selection. What changes is the unit of optimization: classical SEO optimizes pages for queries, while generative retrieval rewards passages that survive extraction from their surrounding context. Writing for both means every important claim should be locally self-sufficient.

The organization of a site around entities and their relations rather than keyword strings. In this model a topic is a graph — a central entity, its attributes, and its relations to neighboring entities — and a site's content is a partial covering of that graph. Architecture decisions (which pages exist, how they link, what each covers) become claims about the topic's structure, which the engine can either corroborate or not.

A triple is a subject–predicate–object statement — the atomic unit of a knowledge graph. Prose written so that triples can be reliably parsed out of it gives an engine a low-friction path from text to structured knowledge. The practical implication is stylistic: declarative sentences with explicit subjects convert to knowledge-graph entries; pronoun-heavy, context-dependent prose does not.

The working model: authority on a topic is a function of coverage (how much of the entity's attribute space the site addresses), coherence (whether internal links mirror the actual semantic relations of the domain), and corroboration (external signals agreeing the site belongs in that neighborhood). The open question is saturation — past some point, additional coverage adds redundancy rather than signal, and locating that point empirically is genuinely hard.

A query-space partitioning failure. When two pages occupy overlapping regions of query space, the engine must split ranking evidence between them, and neither accumulates enough to win. The resolution is conceptual before it is technical: define a distinct intent territory for each page, then make the partition legible through titles, internal links, and canonical signals.

A search engine allocates finite fetch capacity against an effectively infinite web, so it must estimate the expected value of re-crawling each known URL. Site architecture is how a site communicates that value: link depth signals importance, canonical tags consolidate duplicate claims, and freshness patterns set re-visit expectations. Crawl waste — parameter URLs, faceted dead ends — is mispriced inventory competing with pages that matter.

Indexation is a second, stricter filter: the engine asks whether a crawled page earns a place in the index at all. Thin content, near-duplicates, and pages with no distinguishable purpose get crawled and then declined. Persistent indexation gaps are therefore usually quality-model verdicts rather than technical faults — which is what makes them diagnostically interesting.

Schema markup is an assertion layer: the site states, in machine-readable form, what entities it discusses and how they relate. It does not create authority — the engine still verifies assertions against the visible content and external evidence. Its value is in disambiguation: reducing the engine's uncertainty about what a page is, so downstream systems (rich results, knowledge panels, citation selection) can act on it with confidence.

They operationalize a user-experience claim into measurable proxies — loading (LCP), visual stability (CLS), responsiveness (INP). Theoretically they are interesting as an example of the engine importing non-relevance signals into ranking: a small quality prior layered over a relevance-dominated function. That also predicts their observed behavior — decisive at the margin between comparable documents, negligible when relevance gaps are large.

An endorsement under the citation analogy that founded PageRank: a link is one document vouching for another's relevance. The modern interpretation layer conditions that endorsement on context — topical proximity between the documents, the editorial nature of the placement, and the anchor language all modulate how much endorsement actually transfers. Link value behaves like a conditional probability, not a scalar.

Because they are statistically distinguishable. Organic endorsement produces characteristic distributions — of anchor text, source diversity, and topical spread — and manufactured links concentrate probability mass in unnatural regions of those distributions. An engine does not need to catch each scheme individually; it needs only to model what legitimate endorsement looks like and discount the outliers.

Citation gravity: publications need primary sources to ground their own claims, and original research is scarce relative to commentary. A dataset or novel measurement becomes a Schelling point that many writers independently reference. This is the same mechanism as academic citation — and it suggests link acquisition is best understood as contributing referenceable knowledge, not requesting favors.

Epistemic, not mechanical. Continuous observation of rankings, crawls, and index state turns SEO from anecdote into instrumented measurement — hypotheses about a system can be tested against data sampled at the rate the system actually changes. The design problem is selecting what to observe and at what frequency, which is a measurement-theory question before it is an engineering one.

As interpreters between unstructured evidence and structured judgment. Tasks like classifying query intent from a live SERP or summarizing why a cluster of pages competes are judgment calls over messy context — exactly where LLMs are strong, and where rule-based systems were always brittle. The interesting boundary is calibration: knowing which judgments a model can be trusted to make unsupervised.

Only if the constraint is information gain. Generation without a novelty constraint converges on restating the training distribution — adding redundancy to an index that already holds the consensus view. The theoretically sound role for generation is downstream of research: models structuring and articulating information that was gathered from primary sources, with the novel contribution supplied from outside the model.

It relocates it. When observation and synthesis are instrumented, human attention shifts to the questions machines answer poorly: which hypotheses are worth testing, how ambiguous evidence should be weighed, and when a model's confident output is wrong. The scarce resource was never analysis throughput — it is the quality of the questions asked.