Retrieval and RAG pipelines

Retrieval-augmented generation is a pattern in which the model looks information up at answer time instead of relying only on what it absorbed during training. A retriever searches an index for passages relevant to the question, those passages are placed in front of the model, and the model writes its answer from them. Lewis and colleagues introduced the combination in 2020, pairing a generator with a retriever over a dense index of documents for knowledge-intensive tasks.

Compare the two failure profiles, because that is what decides which pattern to use. A model answering from training alone is fast, has no dependency to maintain, and cannot tell you where an answer came from or notice that the policy changed last week. A retrieval system is slower, carries an index that has to be kept current, and can point at the passage it used. In a regulated setting that citation is not a nice presentational touch. It is the difference between an answer that can be checked and an answer that has to be trusted.

The consequence for a data team is that the index becomes a governance surface in its own right. It holds a copy of source material, it is queried by people whose entitlements differ, it goes stale, and it has to be included in deletion work. Each of those four properties is an obligation that arrives with the index and did not exist before it.

The anatomy diagram above draws the pipeline as a chain that folds back on itself, and it labels every connector, because what passes along each one is a different kind of object. The six stages, and the payload each one hands on, are as follows.

1. Chunk. Documents are cut into units that stand alone. What leaves this stage is passages. A passage is the smallest thing retrieval can ever return, so where the cut falls settles what any answer can contain.
2. Embed. Each passage becomes a vector in one shared space. What leaves this stage is vectors. Every passage and every future query must go through the same model, or the distances mean nothing.
3. Index. The vectors are written to a searchable structure. What leaves this stage is an index. This is the artefact with a refresh schedule, a size, a backup and an access story.
4. Retrieve. The query is embedded and pulls back its nearest passages. What leaves this stage is candidates, usually more of them than the model will be shown.
5. Rerank. A second model scores which candidates actually help answer this question, rather than which merely sit nearby. What leaves this stage is ranked context. Reranking is the cheapest quality improvement in most pipelines because it fixes the ordering that distance alone got wrong.
6. Generate. The model writes the answer from the passages it was given. What leaves this stage is an answer, and it is the only artefact in the whole chain that a reader ever sees.

The return path in the diagram matters as much as the forward chain. Evaluation is drawn as an arrow from generation back to chunking, not as a seventh box on the end. Measuring answers is how a team learns that its passages are cut in the wrong place, which makes chunk size an evidence-driven setting rather than a default somebody picked in week one and nobody revisited.

Because only the last artefact is visible, an unmeasured pipeline gives a team no way to tell a generation problem from a retrieval problem. The discipline that helps is to log, for every answer, which passages were retrieved and which were actually shown. That log is the retrieval equivalent of lineage, and it converts arguments about whether the assistant is any good into a question someone can check.

Chunking is splitting a document into passages before indexing. The original retrieval-augmented generation work split a Wikipedia dump into disjoint fixed-length passages and embedded each one, and fixed-length splitting remains the common starting point. The size and the boundaries decide what a search can return, because a passage is the smallest unit that can come back.

The comparison board above puts one question across the top and draws the same document twice underneath it, cut into nine small units on the left and three large ones on the right. A red bar marks the same stretch of text in both. On the left that stretch straddles three units, so no single passage contains the whole answer. On the right it sits inside one unit, but that unit also carries a great deal of surrounding material.

Two effects move in opposite directions, and this is the entire trade. Small passages match a query sharply, because a short unit is mostly about one thing and its position in the space is not diluted. They also cut answers in half, and they strip the context that made a sentence interpretable, so the model receives a rule without its exception. Large passages keep an argument together and preserve the qualifying sentence, but their position is an average of everything they contain, so the match is blunter and a specific query can fail to surface the document that answers it.

Because the trade cannot be won, it has to be chosen deliberately against the questions the system will actually be asked. A corpus of short procedural steps answers well with small passages. A corpus of contracts, where the meaning of a clause depends on definitions three paragraphs earlier, does not. Cutting on structural boundaries such as sections and clauses rather than on a character count usually beats both extremes, and carrying a small overlap between neighbouring passages recovers some of what a hard boundary destroys. Whichever route is taken, every passage should keep the identifier of its source document and its position within it, because that is what makes the answer traceable and the passage deletable.

Vector search means finding the stored vectors nearest to a query vector. Done exactly, it compares the query against every stored vector and returns the true nearest neighbours. Done approximately, it consults an index and trades some recall for speed. The pgvector documentation is explicit that its default behaviour is exact nearest neighbour search with perfect recall, and that adding an index changes the results you get back.

That single sentence carries a lesson many teams learn late. Approximate search is a quality setting, not only a performance setting. If nobody measures recall against exact results, an index tuned for speed can silently drop the passage that held the answer, and the symptom presents as the model being unhelpful. Measure recall against exact search before deciding the index is good enough.

On where it lives, the vocabulary implies a separate product more often than the architecture requires one. A vector database is a store that holds embeddings and answers nearest-neighbour queries, and that capability is available as an extension to databases an organisation may already run: pgvector adds vector types, operators and indexes to PostgreSQL. Whichever way the choice goes, decide it on governance before benchmarks. Keeping vectors alongside the relational data keeps retrieval inside the access controls, backups, monitoring and lineage the organisation already operates, and it lets a query filter by tenant, classification or date in the same statement that ranks by similarity. A dedicated store earns its place where scale or workload justifies separate operational machinery, and the cost of that choice is a second system with its own permissions and its own backup story.

One more architectural point saves a great many disappointing demonstrations. Hybrid retrieval runs keyword search and vector search over the same content and combines the two result lists, commonly with reciprocal rank fusion or by reranking the merged candidates. Keyword search catches exact names, product codes and rare words; vector search catches paraphrase. A part number or a surname is the query type vector search alone is weakest on, because a rare token carries little of the contextual signal that positioned it, and it is exactly the query an audience reaches for when invited to try the system. The keyword half is what recovers it. The pgvector documentation says to use the extension together with PostgreSQL full-text search for hybrid search, and points at reciprocal rank fusion or a cross-encoder to combine the results.

The first family of failures is freshness. An index is a snapshot, and a snapshot is wrong the moment the source changes. Three variants recur. Stale content, where the policy was updated and the index still holds the previous wording, so the assistant cites an obsolete rule with a confident tone. Orphaned content, where a document was withdrawn at source and its passages remain indexed, so the system answers from material the organisation has formally retracted. And partial refresh, where an incremental job fails quietly and half the corpus is a week older than the other half, which produces answers that contradict each other depending on which passage wins.

None of that is visible from the answer, so it has to be instrumented. Treat the index as a dataset and apply data observability to it: watch the age of the oldest passage, the count of indexed documents against the count at source, and the failure rate of the refresh job. Carry the source document identifier and its last-modified date on every passage so an answer can state how old its evidence is. The Data on the Web Best Practices puts the underlying obligation plainly.

The second family is access. An index flattens a corpus into passages, and passages lose the folder permissions, system boundaries and record-level restrictions that governed the originals. If everything is embedded into one store and the retriever queries it under a single service account, the assistant becomes a route around every access control the organisation set. The fix is to keep entitlement metadata on each passage and filter by the asking user before ranking, which is one of the reasons for keeping vectors where row-level security already applies. The related discipline is to decide what should not be indexed at all: special category records, material under legal restriction, and anything whose lawful basis does not extend to this use.

The third family is deletion. An index is a copy, so it falls within the same obligations as every other copy. Where a person exercises the right to erasure, or where a retention schedule expires, removing the source record while leaving its passages in the vector store leaves personal data in a live system that answers questions about it. Deleting a passage is easy to do and easy to forget, so it belongs in the deletion runbook explicitly, alongside backups and warehouse extracts, with evidence that it happened.

Underneath all three sits a definitional problem that the semantic layer module takes up in full. An assistant that reaches documents but not the agreed meaning of a measure will retrieve two passages that use active customer differently and reconcile them by guessing. Retrieval brings the words. It does not bring the agreement about what the words denote.