AI agent developer interview questions: what to ask and what a strong answer sounds like

What screening questions filter AI agent developers quickly?

Twenty minutes. Confirm the candidate built an agent that did real work with tools, not a chat demo.

• 1. Describe one agent you built that ran without you watching. What could it do, and who used it? Strong: names the tools, the users, the volume, and what happened when it was wrong.
• 2. Why was it an agent rather than a fixed workflow or a single prompt? Strong: the task needed run-time branching on tool results; they can also name a case where a plain pipeline won.
• 3. Which stack did you use and what did you work around? Strong: a named framework (LangGraph, the OpenAI Agents SDK, a hand-rolled loop) and a limitation hit.
• 4. How did you know it was working? Strong: a scenario set with a success rate, not "we tried a few examples".
• 5. What was the worst thing it did? Strong: a real incident, its blast radius, and the guardrail added afterwards. No incident usually means no production use.

Which technical questions test real depth in agent design?

Sixty minutes with a senior engineer. Pick five or six; "how did you measure that" and "what broke" matter most.

• 6. Walk me through a tool definition you wrote. How did you decide its inputs, outputs and error shape? Strong: narrow, typed parameters with a JSON schema, a description written for the model, error returns the model can act on, and a test outside the agent.
• 7. When would you expose tools through MCP, and what changes operationally? Strong: MCP (Model Context Protocol) gives clients a standard way to discover and call tools on a server, which pays off when several agents share integrations. The cost: each server is a trust boundary whose tool descriptions and results can carry injected instructions, and auth, allow-lists and logging are still your job.
• 8. Planning first or a ReAct-style loop? When does each win? Strong: a reason-act-observe loop suits short tasks where each step depends on the last result; an explicit plan with checkpoints suits long tasks that need review and midway recovery. Good candidates mix them.
• 9. How do you model state and memory in a long-running agent? Strong: separates working state for the current task from durable memory and from document retrieval, and checkpoints state so a crashed run resumes, as with LangGraph's persisted graph state.
• 10. What makes a tool safe to retry, and what do you do with the ones that are not? Strong: idempotent tools accept a request key so a retry cannot create a second order; non-idempotent actions get a confirm step, a dry run, or a "prepare" and "commit" split; retries have backoff and a limit.
• 11. How do you stop an agent looping, burning budget, or taking an action it should not? Strong: hard step and token caps per run, a cost budget that aborts cleanly, a termination condition the model cannot override, scoped credentials per tool, and human approval for irreversible actions.
• 12. A tool returns a document that contains instructions for the agent. What happens? Strong: every tool result is untrusted data; instructions inside results are never executed, sensitive tools are unreachable from a turn that just read untrusted content, and the risky path is a test case. No view on this is disqualifying for senior roles.

What scenario exercise shows how a candidate actually builds agents?

One hour, live, no code. The brief: "Design an agent that handles refund requests for an online store. It can read the order and the refund policy, issue a refund up to a limit, and escalate to a person."

Look for: they ask what the limit is and who approves above it; they define four or five narrow tools with typed inputs and failure returns; they draw the loop and name where it stops (success, escalation, step cap, budget cap); they make the refund tool idempotent and gated; they describe a scenario set of twenty to fifty cases, including fraud attempts and a message that hides instructions; and they say what the trace should contain.

Weak: one giant prompt with every policy pasted in, and "the model will figure it out".

Which production and evaluation questions separate shipped from demoed?

Thirty to forty minutes, weighted heaviest. In LangChain's State of Agent Engineering survey of 1,340 practitioners (fielded November to December 2025), 57 percent reported agents in production and about a third named quality as their main blocker, ahead of latency and cost. Quality is an evaluation problem.

• 13. How do you evaluate an agent rather than a single response? Strong: a scenario set with expected end states, task success rate as the headline metric, trajectory checks (right tools, sensible order, stopped when it should), an LLM judge checked against human labels, run on every change.
• 14. What do your traces contain, and how have you used one to fix a bug? Strong: every run has a trace with each model call, each tool call and result, and latency and cost per step, plus a specific story, such as four identical search calls that led to a loop guard.
• 15. How do you control cost and latency per task? Strong: a cheap model for routing and a strong model only for the hard step; cached tool results; parallel tool calls; summarised history; a cost cap per run; and cost per completed task tracked.
• 16. Where does a human sit in the loop? Strong: approval before irreversible actions, escalation with the agent's draft attached, a queue someone watches, and interrupt-and-resume so the run pauses rather than dies.
• 17. How do you ship a change to the agent safely? Strong: prompts, tool schemas and model versions are versioned like code; the scenario set runs in CI; changes go behind a flag with a rollback; and a model upgrade counts as a change, because it can alter tool-calling behaviour.

What are the red flags when interviewing AI agent developers?

• No agent ever ran unattended; every example is a demo or a chat interface with no tools.
• Quality was "checked by watching it run"; no scenario set, no success rate.
• No step, token or cost limits.
• Every action was reversible in their telling, so they never needed approval gates or idempotency.
• Treats tool results as trusted; no view on prompt injection through retrieved content.
• Leads with framework names and cannot say why a plain workflow was rejected.
• Cannot name a case where they advised against building an agent.

How should you run the interview loop for this role?

Four touchpoints inside one week: a twenty-minute screen (questions 1 to 5) by the hiring manager; a sixty-minute technical interview (five or six of questions 6 to 12) with the engineer who will review the candidate's code; the scenario hour with the same engineer and a product owner; and thirty to forty minutes on production and evaluation (questions 13 to 17) with whoever carries the pager. Score each stage before you discuss.

Skip whiteboard algorithms. To see code, give a paid two-to-four-hour take-home: a small agent with two tools, one failing intermittently. Rubrics are in how to vet AI engineers, the full process in how to hire AI engineers, and the posting in the AI agent developer job description guide.

Or skip the loop: hire pre-vetted agent developers

Hyparz runs this interview before you see a profile. AI-role applicants are screened for real engineering ability, then tested with a practical agent task with tools that can fail, scored on tool contracts, limits, a scenario set and clear writing. About 5% of AI-role applicants pass vetting. You get a short list typically within days, a one-week risk-free trial (you keep whatever is delivered), a dedicated account manager, automated timesheets and source-code protection. Start with hire AI agent developers, or scope the build through AI agent development services.