LLM Evals Explained: How to Test an AI App (Golden Sets, Metrics and Common Mistakes)

Why evals matter

Language model apps give different answers each time, and a change that fixes one case can break another. Without tests you are guessing. Evals are repeatable tests that tell you whether your app is good, and whether your last change helped. Many hiring teams treat evaluation skill as a sign of a mature AI engineer.

Step 1: build a golden dataset

A golden dataset is a set of realistic inputs with the answer you expect.

• Start with 50–100 examples taken from real or realistic use
• Include easy, normal and hard cases, plus cases where the right answer is "I don't know"
• For RAG, also record the source passage that contains the answer
• Keep a held-out part you never tune on

Step 2: pick what to measure

App type	What to measure
Classification or extraction	Accuracy, precision and recall per field, and format validity
RAG	Retrieval: is the right chunk in the top k (recall@k)? Faithfulness: is the answer supported by the retrieved text? Correctness: does it match the expected answer?
Agents	Task success rate, correct tool and arguments, number of steps, cost, time
Chat or writing	Rubric scores for relevance, tone and safety

Always track cost and latency next to quality.

Step 3: choose how to score

1. Code checks: exact match, JSON schema validity, a regular expression, or "contains the order number". Fast and reliable. Use them whenever you can.
2. LLM as judge: another model grades the answer against a rubric. Useful for open-ended text, but handle with care.
3. Human review: the gold standard. Review a sample regularly and use it to check your judge.

Care with LLM judges: they can favour longer answers, prefer their own style and be inconsistent. Give a clear rubric, ask for a short reason, and compare their scores with a human-labelled sample.

Step 4: run it after every change

Put the eval in a script and run it before you merge a change to prompts, retrieval or models. Save the results in a table:

Run	Change	Retrieval recall@5	Faithfulness	Cost per query
1	Baseline	0.72	0.81	low
2	Chunk size 300 words	0.79	0.83	low
3	Add keyword search	0.88	0.86	slightly higher

(Illustrative numbers only, to show the format. They are not real results.)

Step 5: analyse errors

Read 20 failures by hand and group them: wrong chunk, ignored instruction, bad format, missing data. Fix the biggest group first. This loop, measure, read failures, fix, measure again, is the real job.

Online monitoring

After launch, log inputs, outputs, tool calls, latency and cost. Collect thumbs-up and thumbs-down, and add real failures to your golden set.

Common mistakes

• Tiny test sets that make noise look like progress
• Tuning on the test set, so scores no longer mean anything
• Trusting an LLM judge without checking it
• Measuring quality but not cost or speed
• Never adding new failures back into the dataset

Practise now

Take your RAG app or agent, write 50 test cases, build the scoring script and report your table in the README. This single habit makes a portfolio stand out. See AI engineer portfolio projects.

Next steps

• AI engineer study plan
• AI engineer interview questions