Production AI Failure Modes

“Our offline metrics keep climbing every week but our users keep saying it feels worse.”

The offline metric improves while user-signal degrades because the eval set was sampled at time T from a distribution that has since moved.

“The retriever finds the right document and the model still gives wrong or hand-wavy answers.”

Retrieval surfaces the relevant passage and generation contradicts or hand-waves past it; the failure is in the LLM-doc interaction, not in retrieval.

“We picked the score threshold because the number looked round, and now it's quietly broken.”

A score cutoff was picked because the number looked round, not because it came from a precision/recall target — it works until the upstream model changes, and then it breaks invisibly.

“Our LLM-as-judge bill is bigger than our production inference bill.”

LLM-as-judge becomes the dominant inference line item because every dashboard refresh re-scores the same (query, doc) pairs against the same teacher.

“Our inference cost has crept up and most of the calls aren't even doing the part the LLM is uniquely good at.”

A frontier LLM is used for work a small specialist could do at roughly 1/100x cost — relevance scoring, faithfulness checking, intent classification — because the work in front of it was never attributed.

“We picked the best embedding model on the leaderboard and can't get past a ceiling on our own data.”

An off-the-shelf embedder hits a ceiling on a domain whose vocabulary and relevance semantics are outside web text; continuing to fine-tune that one model is the wrong move.

“The right document is in the context window and the model ignores it anyway.”

The relevant document is placed in mid-context and the LLM under-attends to it; the fix is order-of-context, not which document was retrieved.

“The reranker is accurate but it's eating our latency budget.”

A cross-encoder reranker's joint attention burns 300 ms or more on the user-visible request; the fix is structural — cascade, batch, hide behind LLM prefill — not model substitution.

“Our distilled judge still passes the held-out eval but stopped behaving in production.”

A distilled judge or scorer holds its agreement with the teacher on training data and silently degrades on the production distribution as that distribution shifts.

“We deployed a cheap-then-expensive cascade. The cost graph looks great. Something feels off.”

A cheap-then-expensive cascade saves money until the cheap stage's confidence calibration itself drifts, and the cascade silently lets through cases it should escalate.