AI Engineer Interview Questions and Answers: LLMs, RAG, Agents, Evals and System Design
Updated 2 Nov 2026 4 min read
On this page 11 sections
AI engineer interviews usually mix software engineering, applied LLM knowledge and judgement. Use these questions to practise out loud. The answers are short on purpose; add your own project examples.
1. Software and Python
Q: How do you handle a failing API call? Use timeouts, retries with backoff for temporary errors, clear error messages, and a fallback or safe failure. Log the request for debugging.
Q: Why write tests for an AI app? Because behaviour changes silently. Tests and evals catch regressions in code, prompts and retrieval.
2. LLM fundamentals
Q: What is a token and why does it matter? Models read and write pieces of text called tokens. They set cost, speed and the limit on how much text fits in the context window.
Q: What does temperature do? It controls randomness. Low values give more consistent answers, high values more varied ones. Use low for extraction, higher for creative tasks.
Q: Why do models hallucinate, and how do you reduce it? They predict likely text, not verified facts. Reduce it with retrieval and citations, clear instructions to say "I don't know", structured outputs, validation and evals.
Q: When would you fine-tune instead of using RAG or prompting? Fine-tune for consistent style, format or a narrow skill when prompting is not enough. Use RAG for changing or private knowledge. Try prompting and retrieval first.
3. Prompting and structured output
Q: How do you get reliable JSON? Request a schema, validate the result in code, retry with the error message on failure, and keep the schema simple.
Q: How do you improve a weak prompt? Add clear task and format rules, a few good examples, and test changes on a fixed set rather than on one case.
4. RAG
Q: Walk me through a RAG pipeline. Ingest, clean, chunk, embed, store, retrieve (often hybrid), re-rank, generate with citations, then evaluate. See RAG explained.
Q: Answers are wrong. How do you debug? Check retrieval first: is the right chunk in the top results? If not, fix chunking, metadata or search. If yes, check the prompt and the model.
Q: How do you choose chunk size? Test several sizes against your eval set. It depends on document structure and question type.
Q: How do you protect private data in RAG? Filter by the user's permissions before retrieval, and never rely on the prompt alone.
5. Agents
Q: Workflow or agent? Use a workflow when the steps are known. Use an agent when the path varies and mistakes are tolerable. See AI agents explained.
Q: How do you make an agent safe? Least-privilege tools, approval for risky actions, input validation in code, step and cost limits, and full logging.
Q: What is prompt injection? Untrusted text (a web page or document) that tries to give the model new instructions. Treat outside content as data, restrict tools and validate actions.
6. Evaluation
Q: How do you know your change improved the app? Run a golden dataset before and after, compare quality, cost and latency, and read failures.
Q: What are the risks of LLM-as-judge? Bias toward long answers, inconsistency and shared blind spots. Give a rubric and check against human labels.
7. System design practice
Design a customer-support copilot. A good answer covers: users and goals, data sources, retrieval and permissions, the generation step, human approval, evaluation, latency and cost, monitoring and a pilot plan. Also see system design basics.
Design a document Q&A product for a company with 10,000 files. Cover ingestion and updates, chunking, hybrid search, access control, citations, an eval set and failure handling.
8. Cost, latency and deployment
Q: How would you cut cost? Shorter prompts, caching, smaller models for easy steps, batching, and stopping early. Measure before and after.
Q: How do you handle model changes or outages? Keep model calls behind one interface, pin versions where possible, run evals on new versions and have a fallback.
9. Behavioural
Prepare stories about a failure you diagnosed, a trade-off you explained to a non-technical person and a project you improved with measurements. Use the STAR method.
How to prepare in two weeks
| Days | Focus |
|---|---|
| 1–4 | Review Python, APIs and SQL |
| 5–8 | Rebuild a small RAG app and an eval script from memory |
| 9–11 | Practise the system design questions out loud |
| 12–14 | Mock interviews and polishing your project explanations |
Next steps
Found this useful? Share it with a friend.