How to vet AI engineers: a practical evaluation framework

Why do standard coding interviews miss AI competence?

Standard interviews test whether someone writes correct code under time pressure, but AI failures almost never come from incorrect code. They come from leaked validation data, unevaluated prompts, retrieval that returns the wrong chunk, agents that loop on a failed tool call, and models that silently degrade after a data change. None of that shows in a LeetCode round. You need to test judgment about data, measurement and production: look at shipped work and watch the candidate handle a realistic, messy task.

The four competence areas and what good looks like

Weight the areas by role: ML engineers need depth in the first, LLM engineers in the second, every production hire in the third. Strong seniors are deep in two and literate in the rest.

• ML fundamentals. Good looks like: frames the problem before choosing a model, splits data the way production will use it (by time or customer, not randomly), separates offline metrics from business outcomes, and can explain why a simpler model might win.
• LLM and RAG engineering. Good looks like: builds an evaluation set before tuning a prompt, treats chunking, embeddings, hybrid retrieval and reranking as trade-offs rather than defaults, uses structured outputs and tool calling deliberately, and knows when fine-tuning (LoRA, QLoRA) beats retrieval.
• Production and MLOps. Good looks like: versions data, prompts and models; monitors latency, cost and quality; has a rollback plan; has served models with vLLM or a managed endpoint and can discuss the cost.
• Data. Good looks like: asks where the data came from and what is missing, writes a data contract, and has debugged a pipeline (Airflow, Databricks, Snowflake or plain SQL) that broke quietly upstream.

Portfolio review: shipped systems versus notebooks

Ten minutes per candidate before any call, looking for evidence that something reached users and was measured, not for breadth.

• One system described end to end: data, model or prompt, evaluation, deployment, monitoring, what broke.
• Numbers that moved, with the metric named: held-out accuracy, latency, cost per request, acceptance rate.
• An evaluation artefact: a dataset, a scoring script, a dashboard, or a write-up of a caught regression.
• Code others touched: pull requests, reviews, tests, not a solo notebook.
• A written design note or incident report; remote AI work is mostly writing.
• A "no" story: a time they recommended against a model, or chose a simpler one.

Work-sample tasks that mirror the job

Each task fits two to four hours. Give candidates the data and a clear deliverable, score against the rubric rather than your own preferred solution, and pay for anything longer.

Task 1: Evaluate and improve a small RAG pipeline (LLM/RAG roles)

Provide 50 to 100 messy documents (mixed formats, duplicates, some out of date), 20 questions with reference answers and a baseline pipeline. Ask for a measured improvement and a one-page report. Rubric: built or extended an evaluation harness before changing anything (30 percent); diagnosed the failure mode, such as chunk boundaries, stale documents or embedding mismatch (25 percent); shipped a change with before-and-after numbers (25 percent); explained the next step and its cost (20 percent).

Task 2: Debug a degraded model (ML roles)

Provide a trained model, the training notebook and two weeks of production inputs where accuracy fell, with a planted cause: a shifted feature distribution, a leaked field or a label delay. Rubric: checked data before the model (30 percent); found the cause with evidence (30 percent); proposed a fix and a monitor to catch it next time (25 percent); write-up a product manager could follow (15 percent).

Task 3: Fix and harden a pipeline step (MLOps, data and automation roles)

Provide a small pipeline (an Airflow DAG, an n8n workflow or a plain Python job) that fails intermittently on malformed records with no alerting. Rubric: reproduced the failure (20 percent); fixed it without silently dropping data (30 percent); added validation, retries and an alert (30 percent); documented the change and a runbook (20 percent).

Interview questions by role, and what a strong answer shows

Use four to six questions per role and go deep. The follow-ups matter more than the wording: always ask "how did you measure that?" and "what went wrong?".

Machine learning engineer

• Walk me through how you split the data for your last model. Strong: splits by time or entity to mirror production, names the leakage risk they avoided.
• Describe a model that performed well offline and badly in production. Strong: a specific cause (drift, skew between training and serving features, label delay) and the monitor added afterwards.
• When would you pick gradient-boosted trees over a neural network, or either over an LLM? Strong: structured data, data volume, latency and explainability arguments, with a real example.
• What metric did the business actually care about, and how did it relate to your offline metric? Strong: can name the gap and a time the two disagreed.

LLM and RAG engineer

• How did you build the evaluation set for your last LLM feature? Strong: real user queries, reference answers or rubrics, graded by humans and a model judge, with the judge itself checked.
• A user says the assistant answered from an outdated document. Where do you look? Strong: retrieval logs first, then chunking and metadata filters, then freshness in the index; not "tweak the prompt".
• When is fine-tuning the right call instead of RAG, and which method would you use? Strong: style, format or latency goals rather than knowledge injection; LoRA or QLoRA for cost; still keeps retrieval for facts.
• How do you keep inference cost and latency under control? Strong: caching, smaller models for routing, structured outputs to cut retries, batching or vLLM for self-hosted serving, measured per-request cost.
• How do you handle a prompt change safely? Strong: prompts versioned like code, run against the eval set, shipped behind a flag, with a rollback.

AI agent developer

• Describe an agent you ran in production. What could it do, and what was it not allowed to do? Strong: explicit tool permissions, scoped credentials, human approval for irreversible actions.
• How did you stop it looping or burning budget on a failing tool? Strong: step and cost limits, retry policies, state checkpoints (LangGraph or similar), circuit breakers.
• How do you evaluate an agent rather than a single response? Strong: trajectory-level evals, task success rate on a scenario set, tool-call correctness, regression suites.
• Where do MCP or the OpenAI Agents SDK fit, and where would you not use them? Strong: treats them as integration and orchestration layers, not magic, and knows the operational trade-offs.
• What is the worst thing your agent did, and what changed afterwards? Strong: a concrete incident and a concrete guardrail.

MLOps and automation

• How do you know a model in production is still good? Strong: quality, drift, latency and cost monitors with alert thresholds, and a retraining or re-eval cadence.
• Walk me through a rollback. Strong: versioned artefacts, a real incident, time to recover.
• For a workflow in n8n, Zapier or Make, how do you make it safe to run unattended? Strong: idempotency, error branches, dead-letter handling, alerting, and awareness of each tool's pricing model at volume.

Red flags that predict a bad AI hire

• No evaluation set on any project; quality was "checked by looking at outputs".
• Every project is a notebook or demo; nothing reached real users or ran unattended.
• Cannot name a metric that moved, or names only training accuracy.
• Frameworks before problems: leads with the tool list, cannot say why a simpler approach was rejected.
• No failure stories; production AI always breaks, and a candidate without an incident never ran anything.
• Ships prompt changes without tests.
• Cannot say when not to use AI, or dismisses rules-based and classical approaches.
• Vague about data provenance, privacy or access, especially customer data.

References and the trial week

Ask references two questions: what did this person ship, and would you hire them again for the same work. Then replace the final interview with a paid trial week on a bounded real ticket, such as an evaluation harness for an existing feature or a known retrieval failure fixed with before-and-after numbers. Judge on whether it shipped, whether trade-offs were explained in writing, and whether your team wants to continue. The full process is in how to hire AI engineers.

How Hyparz vets AI engineers

Hyparz applies this framework before you see a profile. Applicants are screened for real-world engineering ability, then tested in their lane with practical work rather than trivia: a modelling problem on a real dataset, a retrieval task over messy documents, or an agent task with tools that can fail. Reviewers look for evaluation discipline, sensible cost and latency decisions, production habits and clear writing. About 5 percent of AI applicants are accepted. You get a short list matched to your role, typically within days, a one-week risk-free trial on your actual work and a dedicated account manager. Start with hire AI engineers, or the specific lanes: ML engineers, LLM developers, AI agent developers and MLOps engineers.