LLM engineer interview questions: what to ask and what a strong answer sounds like

Which screening questions separate LLM engineers from prompt tinkerers?

Use these five in a 30-minute screen. Each has a quick, checkable answer.

• Describe one LLM feature you took to production. What did it do, who used it, and what did you measure? A strong answer names the use case, the model, the eval set and at least one number: task pass rate, p95 latency or cost per request.
• What is the difference between JSON mode and structured outputs? JSON mode guarantees valid JSON only; structured outputs constrain generation to a supplied schema, so required keys and enum values cannot be missing or invented, as OpenAI's Structured Outputs guide describes.
• When would you use RAG rather than fine-tuning? RAG when the answer depends on changing or private facts; fine-tuning when you need a consistent style, format or narrow skill the base model lacks.
• How do you choose between a hosted model API and an open-weight model you serve yourself? Expect volume, latency, data residency, fine-tuning needs and operational capacity as the criteria, not a vendor preference.
• What is prompt injection and where does it enter an application? Through user input, retrieved documents, tool results and web content; the fix is architectural (separate instructions from data, limit tool permissions, validate outputs), not prompt wording.

Which technical questions test real depth?

Pick five to seven of these for a 60 to 75 minute technical round. Each line gives the question and what a strong answer covers.

• How do you design a prompt for a task that must return structured data at scale? Stable instructions and reference material first so prefix caching applies, the dynamic part last, a schema enforced with structured outputs, a few diverse examples, and an eval run on every change. OpenAI's prompt caching guide explains why order matters.
• Walk me through tool calling for a model that can query a database and send an email. Tight tool schemas, an allow-list, idempotent retries, confirmation or dry-run for side effects such as sending, argument validation before execution, and tool results treated as untrusted text.
• How would you build an evaluation set for an extraction feature? 100 to 300 real inputs covering normal, edge and adversarial cases; deterministic checks for format and forbidden content; reference comparison where a correct answer exists; an LLM judge with a written rubric for open-ended quality; a version number; and a CI run on every prompt or model change.
• What goes wrong with LLM-as-a-judge, and how do you guard against it? The MT-Bench paper (Zheng et al., 2023) documents position, verbosity and self-enhancement biases and limited reasoning in LLM judges, while reporting over 80% agreement with humans for strong judges. A strong candidate validates the judge against human labels, randomises answer order, and never lets judge scores alone gate a release.
• Explain LoRA and QLoRA, and tell me when you would refuse to fine-tune. LoRA (Hu et al., 2021) freezes the base weights and trains low-rank adapters, cutting trainable parameters by up to 10,000 times for GPT-3 175B; QLoRA (Dettmers et al., 2023) backpropagates through a 4-bit quantised base into those adapters, fine-tuning a 65B model on a single 48GB GPU. Refuse when there is no eval set, when retrieval solves the problem, or when clean examples are too few.
• Compare serving with vLLM to calling a hosted API for a workload of your choice. Strong answers note that vLLM offers PagedAttention, continuous batching, prefix caching, speculative decoding, quantisation down to INT4 and an OpenAI-compatible server, then weigh GPU cost, utilisation and the MLOps time to run it, ending with a break-even.
• An assistant started giving longer, vaguer answers last Tuesday. Nothing in your repo changed. What do you do? Check whether a floating model alias was updated by the provider, compare judge scores and output length across the date, re-run the eval set on the pinned previous version, and pin versions from then on.

What practical exercise should you set?

Give the candidate a one-page brief: a support assistant that answers from a 2,000-document knowledge base, must cite sources, may create a ticket on the user's behalf, and has a budget of two seconds to first token and a fixed cost per conversation. Run it as a 45-minute design discussion or a three-hour take-home that produces a design note plus one working slice, such as the eval harness or the tool-calling layer.

Look for hybrid retrieval with permission filtering, a confirmation step for ticket creation, an eval plan that separates retrieval recall from answer quality, streaming to meet the first-token budget, prefix caching, routing between a small and a large model, tracing on every request, and a clear statement of what they would cut to hit the cost ceiling. Candidates who skip evaluation or treat the knowledge base as trusted have not run this in production.

Which production and evaluation questions matter most?

These five suit the final round, ideally with the engineer who will own the system in production.

• What does a request trace contain in your current or last system? Prompt version, model and version, tokens in and out, latency per stage, retrieved chunk identifiers, tool calls with arguments and results, guardrail decisions and outcome. Bonus for knowing the OpenTelemetry GenAI conventions.
• How do you set and enforce a cost and latency budget? Caps on context, output and agent steps per request; the smallest model that passes evals per stage; prompt caching; batching offline work; streaming; and dashboards with cost per feature and anomaly alerts.
• How do you version prompts and models, and how do you roll back? Prompts, tool schemas and model identifiers in version control with review and an eval run before merge; pinned model versions; canary rollout; previous prompt and model restorable in minutes. The production deployment checklist covers the full release gate.
• Which OWASP LLM risks applied to your last system, and what did you do about each? The OWASP Top 10 for LLM Applications (2025) leads with prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling and excessive agency. Strong candidates map at least four to controls they built.
• How did you find out when quality regressed in production? Sampling conversations for human review, drift alerts on judge scores and output length, user feedback capture, and a path from logged failures to new eval cases.

What are the red flags?

• Cannot name a number from a shipped system: no eval score, no latency, no cost figure.
• Proposes fine-tuning before an evaluation set exists, or to inject facts that retrieval would serve better.
• Trusts LLM judge scores without validating them against human labels.
• Treats retrieved documents and tool results as trusted instructions, or believes a strong system prompt stops prompt injection.
• Describes a weekend demo as production experience.

How should you run the interview loop?

Two rounds in under two weeks is enough. Round one is a 30-minute screen with the five screening questions, run by the hiring manager or a senior engineer. Round two is a 75-minute technical session combining five to seven depth questions with the scenario exercise, run by two engineers, one of whom owns production systems.

Write the scoring notes before the interview so that two interviewers rate the same answer the same way, and ask for specifics whenever an answer stays general: which model, which eval, what number, what changed. The vetting guide gives a rubric that works across AI roles, and the hiring guide covers sourcing and offers.

Or skip the loop

If you do not have two engineers free to run this loop, Hyparz runs it for you. LLM developers on Hyparz are pre-vetted on exactly these areas, about 5% of AI-role applicants pass vetting, and companies are matched typically within days with a one-week risk-free trial, a dedicated account manager and automated timesheets. The LLM application development service covers scoped builds end to end.