RAG Explained: How to Build Your First Retrieval-Augmented Generation App

What RAG is

Retrieval-augmented generation (RAG) gives a language model the right information at the moment it answers. Instead of relying on what the model memorised, you search your own documents for relevant passages and put them into the prompt. The model then answers using that evidence, ideally with citations.

Why use it?

• Models do not know your private or recent data
• Answers can point to sources, which builds trust
• Updating a document is cheaper than retraining a model

The pipeline

1. Ingest: load files such as PDFs, web pages and database rows.
2. Clean: remove headers, footers and junk. Keep titles and page numbers as metadata.
3. Chunk: split text into pieces, for example 200–500 words, often with a little overlap.
4. Embed: turn each chunk into a vector (a list of numbers) that captures meaning.
5. Store: save vectors in a vector index. Postgres with pgvector, Chroma and FAISS are common starting points.
6. Retrieve: embed the user's question and find the closest chunks. Combine with keyword search (hybrid) for better results.
7. Re-rank (optional): reorder the results with a better model.
8. Generate: put the best chunks into the prompt and ask the model to answer only from them.
9. Cite and check: show sources, and say "I could not find that" when evidence is missing.

A short sketch

def answer(question: str) -> dict:
    q_vec = embed(question)                       # turn the question into a vector
    hits = index.search(q_vec, k=5)               # nearest chunks, with their metadata
    context = "\n\n".join(f"[{h.source} p.{h.page}] {h.text}" for h in hits)
    prompt = (
        "Answer using ONLY the context. Cite sources like [file p.N]. "
        "If the context does not contain the answer, say you do not know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    reply = llm(prompt)                           # one model call, kept behind a function
    return {"answer": reply, "sources": [h.source for h in hits]}

This is a sketch: embed, index and llm stand for whatever library or API you choose. Keep them behind small functions so you can change providers.

Choices that matter more than the model

• Chunk size: too small loses context, too large adds noise. Test a few sizes.
• Metadata: store document name, section, date and access level, then filter on them.
• Hybrid search: keyword plus vector search often beats either alone, especially for names, codes and numbers.
• Freshness: re-index when documents change, and remove deleted ones.
• Permissions: only retrieve what the user is allowed to see.

Why RAG apps fail

Problem	Typical cause	Fix
Wrong answer with confidence	Retrieved the wrong chunk	Better chunking, hybrid search, re-ranking
"I don't know" too often	Answer split across chunks	Larger chunks or neighbouring chunks
Outdated answer	Old index	Scheduled re-indexing
Invented details	Prompt allows guessing	Strict instruction, require citations
Leaks private data	No permission filter	Filter by user access before retrieval

How to test it

Do not trust a demo. Create 50–100 real questions with the correct answers and the correct source passage, then measure:

• Retrieval: is the right chunk in the top 5? (recall@k)
• Faithfulness: is the answer supported by the retrieved text?
• Correctness: does it match the expected answer?

Read LLM evals explained for the method.

Your first project

Pick documents you know well (a course syllabus, a product manual, company policies). Build the pipeline above with a simple web page, show citations, then write a README with your eval results and failure cases. Ideas in AI engineer portfolio projects.

Next steps

• Study plan: AI engineer study plan
• Next topic: AI agents explained
• Used in practice by forward deployed engineers