How to choose an LLM for production: a practical framework
Stop choosing models by leaderboard. A practical framework for picking an LLM for production: define the task, build a small eval set, measure quality/latency/cost on your data, and design for swappability.
Most teams choose an LLM the wrong way: they read a leaderboard, pick the highest-ranked name they recognize, and ship. Leaderboards measure aggregate performance on public benchmarks — your product is a specific task on your data with your latency budget. Those are different questions.
Step 1: Name the task precisely
"Add AI" is not a task. "Summarize a support thread into three bullet points with no invented facts," "extract line items from supplier invoices into JSON," "answer policy questions from our handbook with citations" — those are tasks. Precision here determines everything downstream, because model differences are task-dependent: rankings on reasoning benchmarks say little about extraction quality.
Step 2: Build a 50-example eval set before touching an API
Collect 30–100 real inputs and write down what a correct output looks like for each. This is the single highest-leverage hour in the whole project. Without it you are choosing models by vibes, and you'll re-choose by vibes every time a vendor ships an update.
Step 3: Measure the triangle — quality, latency, cost
- Quality: score each candidate on your eval set — exact-match where possible, rubric or LLM-judge with spot checks where not.
- Latency: measure p95 end-to-end, including retrieval if you're doing RAG. A chat UI tolerates 2–4s with streaming; an inline autocomplete doesn't.
- Cost: price per successful task, not per token — a cheaper model that needs retries or human review isn't cheaper.
A common production pattern falls out of this: route the bulk of traffic to a fast, cheap model and escalate the hard cases to a frontier model. Tiered routing frequently cuts spend by 60–80% with no measurable quality loss on the routed traffic.
Step 4: Design for swappability
Whatever you pick will be the wrong choice within a year — prices drop, models improve, vendors change terms. Keep prompts in version control, keep the model behind a thin interface, and keep the eval harness runnable in CI so that swapping models is an afternoon's evaluation, not a rewrite. The eval set you built in step 2 is what makes this cheap.
Common mistakes
- Benchmarking on the demo examples the vendor's docs use, not your ugly real inputs.
- Ignoring rate limits and quota until launch week.
- Choosing open weights for cost reasons, then spending more on GPU ops than the API would have cost.
- Skipping the eval set "for now" — now is exactly when it's cheapest to build.
The framework is boring on purpose. Model choice should be an empirical decision you can re-run, not a bet on a brand name.