Practical notes on RAG pipelines

Retrieval as an explicit objective

A useful way to think about RAG is as an optimisation problem, not just “attach a vector store”. For a query q and document d, we want a retrieval function R that maximises the probability that the context actually contains information needed to answer q.

A simple scoring function is

score(q, d) = sim(f(q), g(d))

where f and g are encoders and sim is usually cosine similarity. Production systems often blend multiple signals (BM25, recency, metadata filters) instead of relying on a single score.

Index and chunking design

Index design is where most practical failures come from:

Chunking too small: the model sees fragments with no narrative structure.
Chunking too large: retrieval is cheap but the context window fills with mostly irrelevant text.
No separation between “reference” and “examples”: explanations and logs pollute the knowledge base.

Scoring: a tiny bit of math

Given a query embedding v and a set of document embeddings d₁,…,dₙ, cosine similarity is:

cos(v, dᵢ) = (v · dᵢ) / (∥v∥₂ ∥dᵢ∥₂)

In practice we pre-normalise vectors, so ranking reduces to a dot product. This is what FAISS and similar libraries optimise.

Pipeline pseudocode

A minimal but production-oriented RAG pipeline can be sketched as:

python

def answer(query, k=8):
    q_vec = encode_query(query)
    doc_ids, scores = vector_index.search(q_vec, top_k=k)
    docs = fetch_documents(doc_ids)

    prompt = build_prompt(query=query, documents=docs)
    raw_answer = llm.generate(prompt)

    grounded = verify_against_docs(raw_answer, docs)
    return grounded

RAG mermaid schema

The structure of the system, independent of vendor choices, can be written as: