How to deploy LLM applications to production: a checklist that catches what demos hide

Updated October 17, 2026 · By the Hyparz talent team

An LLM application is ready for production when it has a versioned evaluation set it passes, guardrails on inputs, outputs and tool calls, tracing on every request, a cost and latency budget it stays inside, a serving path chosen deliberately (managed API or self-hosted with vLLM), and a release process with canary and rollback. Prototypes break in production because they skipped those six things, not because the model got worse.

This guide covers each in turn and ends with a checklist and the roles you need to run it.

Key takeaways

Build and version an evaluation set before launch; without it you cannot tell whether a prompt, model or retrieval change is an improvement or a regression.
Most RAG failures are retrieval failures: invest in chunking, hybrid search, reranking, grounding instructions and citations.
Guardrails must cover prompt injection, output handling and tool permissions; OWASP ranks prompt injection as the top LLM application risk.
Decide between managed model APIs and self-hosted serving (vLLM) on the basis of volume, latency, data control and the team you have to run it.
Trace every request with tokens, latency, model and prompt versions and retrieval context, following the OpenTelemetry GenAI conventions.
Treat prompts and model versions as deployable artifacts: version them, canary them, and be able to roll back in minutes.

Why do LLM prototypes fail in production?

Prototypes fail because the inputs change and nothing measures it. A demo runs on a few dozen friendly questions; production brings long messages, pasted documents, other languages, hostile prompts, concurrency and a model vendor that silently updates behaviour. Without an evaluation set, tracing and guardrails, the team only finds out when a customer or a bill does.

The second cause is that an LLM feature is a distributed system with probabilistic parts: retrieval, model, tools and post-processing each fail in their own way, and a failure in one looks like a failure in another. Production readiness means seeing which part failed, for which request, at what cost, and changing one part without re-testing by hand.

How do you build an eval set before you ship?

Collect 100 to 300 real or realistic inputs that cover normal use, edge cases and adversarial cases, and write down what a good answer looks like for each. Score with a mix of exact checks (format, required fields, forbidden content), reference comparisons and LLM-as-judge rubrics, and validate the judge against human labels before trusting it. OpenAI's model optimization guide describes the same loop: build evals first, iterate on prompts, and only then consider fine-tuning.

Version the set alongside the prompts, run it in CI on every prompt, model or retrieval change, and keep adding failures from production logs. For agents, evaluate trajectories as well as final answers: the right tool called with the right arguments, no unsafe actions, and sensible stopping.

How do you make retrieval reliable for RAG?

Evaluate retrieval separately from generation, with questions mapped to the passages that should be retrieved, and measure chunk-level recall and precision before judging answers; a model cannot answer well from the wrong context.

Parse documents properly (tables, headers, PDFs with layout) and chunk by structure, not by fixed character count alone; attach metadata such as source, version, date and access scope.
Use hybrid retrieval, combining vector similarity in pgvector, Qdrant or a similar store with keyword search, and add a reranker; this is the single biggest quality lever for most corpora.
Instruct the model to answer only from the provided context, require citations to chunk identifiers, and return an explicit "not found" when retrieval confidence is low.
Filter by the requesting user's permissions at query time, never after generation.
Re-index on document change and monitor index freshness; stale retrieval is a silent failure mode.

What guardrails does a production LLM app need?

The OWASP Top 10 for LLM Applications (2025) ranks prompt injection first, followed by sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. A production app needs a specific control for each one it is exposed to.

In practice: treat retrieved content and tool results as untrusted input, separate instructions from data in prompts, validate model output before it reaches a browser, database or shell, and never give the model more permissions than the user it acts for. For agents, keep an allow-list of tools, require confirmation for write or payment actions, cap steps per task, and route low-confidence or high-impact cases to a human. Add every guardrail to the eval set so it is tested like everything else.

How do you control latency and cost?

Choose the serving path deliberately

Managed APIs from OpenAI, Anthropic, Google and others are the fastest route and the right default for most teams: no GPUs to run, frontier models, and features such as prompt caching and batch endpoints. Self-hosting pays off when volume is high and steady, when data cannot leave your environment, or when you serve a fine-tuned open-weight model. vLLM is the common choice: its documentation describes PagedAttention for attention memory, continuous batching, chunked prefill and prefix caching, quantization formats from FP8 down to INT4, GPTQ, AWQ and GGUF, and an OpenAI-compatible API server on NVIDIA and AMD GPUs. The hidden cost of self-hosting is the MLOps capacity to run, scale and upgrade it.

Budget tokens like any other resource

Set a maximum context size, output length and step count per request, use the smallest model that passes your evals for each stage, cache repeated system prompts and retrieved context, batch offline work, and stream responses so perceived latency drops. Track cost per request and feature next to latency; a runaway agent loop is both a security and a finance incident, which is why unbounded consumption is on the OWASP list.

What should you observe in production?

Every request should produce a trace that records the prompt version, model and version, input and output tokens, latency per stage, retrieved chunk identifiers, tool calls with arguments and results, guardrail decisions, and the final outcome. The OpenTelemetry GenAI semantic conventions define spans, metrics and events for generative AI clients, agents and MCP, so standard tooling such as Langfuse, Arize Phoenix or LangSmith can consume them.

Alongside traces, log user feedback and sample conversations for human review, and watch for drift: a rise in "not found" answers, longer outputs, more guardrail triggers or a change in judge scores after a provider update. Observability data is also your next eval set, so make it easy to promote a logged failure into a test case.

How do you release safely?

Prompts, retrieval configuration, tool schemas and model identifiers are deployable artifacts and deserve the same discipline as code: version control, review, a changelog and an eval run before merge. Pin model versions explicitly rather than using floating aliases, and schedule upgrades as releases with their own eval pass.

Roll out behind a flag to a small share of traffic, compare eval scores, latency, cost and feedback against the current version, and keep the previous prompt and model ready to restore in minutes; for agents, add a kill switch that disables tool execution.

How do you handle security and data?

Classify what enters prompts: strip or mask personal data where it is not needed, and know which provider or region processes what.
Enforce tenant isolation in retrieval and memory; a shared vector index without a tenant filter is a data leak waiting to happen.
Set retention for prompts, completions, traces and conversation memory, and make deletion requests work end to end.
Store API keys and model credentials in a secrets manager, scope them per environment, and rotate them.
Review the supply chain: model weights, embedding models, third-party tools and MCP servers all run with your data.

The production checklist, and who you need

Use this list as a release gate. Each line should have an owner and evidence, not a tick.

Evaluation

Versioned eval set of 100+ cases covering normal, edge and adversarial inputs
Automated scoring with validated LLM-as-judge or reference checks
Eval run on every prompt, model or retrieval change in CI
Retrieval recall and precision measured separately (RAG)
Agent trajectories evaluated for tool choice and safe stopping

Guardrails and security

Prompt-injection handling for user input, retrieved content and tool results
Output validation before any downstream system consumes it
Tool allow-list, confirmation for write actions, step cap, kill switch
Permission filtering at retrieval time; tenant isolation in indexes and memory
PII minimisation, retention policy, secrets management, supply-chain review

Serving, cost and observability

Serving path chosen (managed API or vLLM) with capacity and failover plan
Per-request limits on context, output and steps; caching and batching in place
Cost and latency per request and feature on one dashboard with alerts
Traces with prompt and model versions, tokens, retrieval and tool calls
Human review sampling and a path from logged failures to new eval cases

Release

Prompts, configs and model IDs versioned and reviewed
Pinned model versions; upgrades treated as releases
Canary rollout with comparison metrics and a rollback tested in advance
Runbook for incidents, including who can disable tools or switch models

Who runs it

A small team covers this: an LLM developer for prompts, evals and application logic, a RAG developer when retrieval quality is the bottleneck, an AI agent developer for tool-using systems, and an MLOps engineer for serving, observability and release engineering. Hyparz matches companies with pre-vetted remote engineers in each of these roles, typically within days, as individual experts or a blended team, with a one-week risk-free trial and a dedicated account manager.

Hyparz at a glance (October 2026)

Vetting pass rate (AI roles)

About 5% of applicants pass

About 5% of the AI engineers who apply to Hyparz pass vetting, which tests real-world ability, system design and communication on practical tasks.

Indicative hourly rates (USD)

Mid-level AI engineer: $60 to $100 per hour
Senior AI / ML engineer: $110 to $190 per hour
MLOps / AI infrastructure specialist: $120 to $200 per hour

Indicative bands for engineers matched through Hyparz; they track the rates Arc.dev publishes for vetted freelance AI talent. Toptal does not publish rates. Your quote depends on specialism, seniority and engagement length. Sources: Arc, freelance AI developers for hire (October 2026): $60 to $100+ per hour; Arc Employer Blog, freelance developer rates 2026 (updated May 2026): AI/ML engineering $110 to $190, MLOps $120 to $200.

Need an engineer who has taken LLM apps to production? Hyparz matches pre-vetted remote LLM, RAG and MLOps engineers typically within days, with a one-week risk-free trial.

Get matched with a production LLM engineer

Frequently asked questions

What is the difference between MLOps and LLMOps?

MLOps covers training, deploying and monitoring models you build yourself: data pipelines, experiment tracking, model registries, drift and retraining. LLMOps applies the same discipline to applications built on foundation models, where the artifacts are prompts, retrieval indexes, tool schemas and model versions, and the metrics are eval scores, groundedness, token cost and guardrail triggers.

How do I evaluate LLM output quality?

Build a labelled set of real inputs with expected outcomes, then score with layered checks: deterministic rules for format and forbidden content, reference comparison where a correct answer exists, and an LLM judge with a written rubric for open-ended quality. Validate the judge against human ratings on a sample. Run the set on every change and feed production failures back into it.

How do I control LLM API costs?

Measure cost per request and per feature first. Then cap context, output and agent steps, route easy requests to smaller models, cache repeated prompts and retrieved context, batch non-interactive work, and stream to cut perceived latency. Alert on spend anomalies the same way you alert on errors.

Do I need a vector database in production?

Only if you do retrieval. For RAG, you need some index: pgvector inside an existing Postgres is enough for many teams and keeps operations simple, while Qdrant or similar dedicated stores suit large corpora, heavy filtering or high query rates. Whatever you choose, pair vectors with keyword search and a reranker, and enforce tenant and permission filters inside the query.

How do I protect against prompt injection?

Assume every piece of text the model reads, including retrieved documents and tool results, can contain instructions. Keep system instructions separate from data, validate outputs before they act on anything, limit tool permissions to what the user could do, require confirmation for high-impact actions, cap steps, and test known injection patterns in your eval set. OWASP ranks prompt injection as the top LLM application risk in 2025.

Flexible Engagement Models