RAG engineer interview questions: what to ask and what a strong answer sounds like

What should a RAG engineer interview cover?

Four things: whether the candidate has shipped retrieval-augmented generation to real users, how well they understand the retrieval stack, whether they can design a system under realistic constraints, and whether they can evaluate and operate it. The bold line is the question; the note after it is your scoring guide. The RAG engineer job description guide sets out the responsibilities these questions test; if you are still deciding whether the problem needs retrieval at all, read RAG vs fine-tuning first.

Which screening questions separate real experience from a tutorial?

• 1. Describe a RAG system you shipped: the corpus, the users and what broke. A strong answer names the document types and volume, who used it, one failure they diagnosed (wrong chunks, stale content, a hallucinated citation) and the measured result of the fix.
• 2. Why RAG for that system rather than long context or fine-tuning? Look for reasons tied to the use case: the knowledge changed, answers needed citations, access had to be filtered per user. They should also know when retrieval is overkill; Anthropic (September 2024) notes that a knowledge base under about 200,000 tokens can simply go in the prompt.
• 3. Which vector store and embedding model did you use, and why? pgvector when Postgres is already in the stack and the corpus is moderate; Qdrant, Weaviate or Pinecone when filtering, scale or a managed service mattered. The embedding model should have been picked by testing on their own queries, not by leaderboard rank.
• 4. How did you know retrieval was working? The answer must contain a labelled question set and a metric such as recall at k, tracked before and after each change. "We checked a few answers" fails at senior level.
• 5. What would you do differently today? Experienced engineers have regrets: chunking chosen too early, hybrid search added too late, no evaluation set until users complained. No regrets usually means no production exposure.

Which technical questions test retrieval depth?

• 6. How do you decide on a chunking strategy? A strong answer starts from document structure (headings, tables, code, numbered clauses), chooses chunk size in tokens with overlap, and mentions parent-child retrieval, where small chunks are matched but the surrounding section goes to the model. They test alternatives against an evaluation set rather than keeping a library default.
• 7. When do you add BM25 to vector search, and how do you merge the results? Keyword search wins on exact terms that embeddings blur: product codes, error strings, names, clause numbers. Expect reciprocal rank fusion or weighted scoring, and a measurable gain: Anthropic (September 2024) reports that combining contextual embeddings with contextual BM25 cut the top-20-chunk retrieval failure rate by 49 percent, and adding a reranker took that to 67 percent.
• 8. What does a reranker add, and what does it cost? A cross-encoder scores each query and passage pair together: more accurate than embedding similarity, too slow for the whole index. The pattern is retrieve 50 to 100 candidates, rerank, keep 5 to 10. It adds latency and per-call cost, justified when recall at 50 is high but precision at 5 is poor.
• 9. How do you handle vague or multi-part questions phrased differently from the documents? Query rewriting, decomposition, multi-query retrieval, hypothetical document embeddings and conversation-aware rewriting of follow-ups into standalone queries. Each adds a model call and must earn its latency in evaluation.
• 10. How do you keep the index fresh? Incremental re-indexing from change events or a schedule, document versioning so old chunks are retired rather than duplicated, and pinning the embedding model version, since a change means re-embedding everything.
• 11. How do you enforce permissions in retrieval? The only acceptable answer is filtering at retrieval time, using access metadata stored with each chunk and synced from the source system. Filtering after generation has already leaked data into the context window.
• 12. When would you reach for GraphRAG instead of plain vector search? Expect awareness, not advocacy. Microsoft's GraphRAG paper (Edge et al., April 2024) targets global questions about a whole corpus and reports gains in comprehensiveness and diversity over a vector baseline, at a much higher indexing cost.

What design exercise should you set?

13. The exercise. "Design an internal assistant over 50,000 documents (policies, support tickets and contracts) for 2,000 employees with different access rights. Answers must cite sources, the assistant must say so when the documents do not cover a question, and the 95th-percentile response time must stay under three seconds. Walk us through ingestion, retrieval, generation, evaluation and cost." Give 60 to 90 minutes, live.

What to look for: they ask about users and document formats before drawing; ingestion treats parsing, chunking, metadata and permissions as real steps; retrieval is hybrid with a reranker and a permission filter; generation includes citation checking and a refusal path; and they can estimate cost per query and say what they would cut if the budget halved.

Which production and evaluation questions matter?

• 14. How do you build the evaluation set, and what do you measure? Questions from real users, with expected answers and source passages. Measure retrieval (recall and precision at k) separately from generation. The RAGAS paper (Es et al., September 2023) defines faithfulness, answer relevance and context relevance as reference-free metrics; a strong candidate uses these or similar and calibrates any LLM judge against human ratings.
• 15. How do you make the system cite correctly and refuse when it should? Citations tied to chunk identifiers, a post-check that every cited passage was in the context, a retrieval-score threshold below which the answer is "this is not covered in the documents", and a hand-off to a human.
• 16. How do you control cost and latency? Cache embeddings and repeated answers, use prompt caching for the static part of the prompt, run a small model for query rewriting, tune top-k instead of stuffing the context, stream the response. They should know each component's cost per query.
• 17. What do you monitor after launch? Retrieval hit rate, no-answer rate, user feedback, latency percentiles, tokens per answer and drift as the corpus changes, with weekly failure sampling fed back into the evaluation set.
• 18. Walk me through debugging a wrong answer. A trace: was the right passage in the index, was it retrieved, was it ranked high enough, did the model use it? Fix the earliest failing stage, not the prompt.

What are the red flags?

• No evaluation set in any system they describe; quality judged by reading a few answers.
• Treats the vector database as the whole solution and cannot say when keyword search or a reranker is needed.
• Assumes a bigger context window removes the need for retrieval; Liu et al. (2023) showed accuracy drops when relevant information sits mid-context, and cost rises with every token.
• Blames the model for every failure and never mentions chunking, parsing or stale content.

How do you run the interview loop?

Four conversations totalling four to five hours, spread over no more than ten days so good candidates do not drop out: a 30-minute screen (questions 1 to 5) by the hiring manager, a 60-minute technical round (6 to 12) by an engineer who has shipped retrieval, the 60 to 90 minute design exercise (13) with two interviewers, one to drive and one to score, and a 30 to 45 minute production round (14 to 18) with whoever will operate the system.

Score each round on specific evidence, not impressions, and compare notes before anyone sees the others' scores. See also how to vet AI engineers and how to hire AI engineers.

Or skip the loop

If you do not have an engineer who can run the technical rounds, hire someone already vetted on this material. Hyparz matches companies with pre-vetted remote RAG developers, typically within days; about 5% of AI-role applicants pass vetting. Every engagement starts with a one-week risk-free trial, which works well as the final round: set the engineer a slice of your real corpus and keep any work delivered. A dedicated account manager, automated timesheets and source-code protection are included. To scope the project rather than the hire, see RAG implementation services.