LLM features: the distance between the demo and production

Every company has now seen the demo: someone wires a model to internal documents, asks it a question, and the room goes quiet. The demo takes a week. The gap between that and a feature you’d put in front of customers is the actual project, and we build in that gap — so here’s an honest map of it.

Why the demo lies

A demo answers the questions its builder asks it, on the documents they chose, judged by impression. Production answers whatever users actually type, over your full messy corpus, judged by consequences. Every hard problem lives in that difference.

Retrieval is most of the product. For anything grounded in your data, answer quality is mostly determined before the model sees the question — by how documents are chunked, indexed and searched. Garbage retrieval with a frontier model still produces confident garbage. This is data engineering, not prompt magic, and it’s where most of the build time goes.

Evals are the tests you can’t skip. With normal code you write tests; with a probabilistic system you need evaluations — a graded set of real questions with known-good answers, run on every change. Teams without evals ship on vibes, and every prompt tweak is a gamble on regressions they can’t see. Building the first eval set is unglamorous work with the highest ROI in the project.

Failure needs a designed path. The model will sometimes be wrong. The product question is what happens then: Can it say “I don’t know”? Does it cite sources so users can verify? Is there a human escalation for consequential actions? Features fail in production not because the model errs, but because nobody designed for the error.

Cost and latency are product constraints. Tokens are cheap; tokens times retrieval context times every user times every day is a real bill, and multi-step chains can turn a “fast” model into an eight-second wait. Decisions like caching, model tiering (small model for easy cases, big model for hard ones) and context budgets belong in the design, not the postmortem.

Security gets a new attack surface. User input is now instructions. Prompt injection — including through the documents you retrieve — plus PII flowing into prompts and logs means access control has to live outside the model. The model must never be the thing enforcing who can see what.

The staging that works

Ship it in this order and each step pays for the next: internal assistant first (support team, not customers — cheap feedback, contained failures), evals before broadening (built from real internal usage), customer-facing with sources and escape hatches, and only then autonomy over consequential actions, if ever. Teams that invert this — customer-facing agent first — generate incident reports, not product.

When to skip the LLM entirely

If the workflow is deterministic — lookups, calculations, routing on known fields — conventional software is cheaper, faster and doesn’t hallucinate. Some of the best consulting sentences we deliver start with “you don’t need a model for this.” An unglamorous dashboard on trusted data beats an oracle nobody trusts.

If you have a use case and a corpus, we’ll tell you which side of that line you’re on — and if it’s the model side, build the retrieval and eval foundations that make it real.