LLM monitoring in production: what to watch, and how failures become tests

LLM monitoring is watching an application built on a language model while real users are on it: whether calls succeed, how long they take, what they cost, and whether the answers and actions are still the ones you tested. The first three are ordinary monitoring with new units. The fourth is the hard one, because a wrong answer comes back with the same status code as a right one.

Most guides to LLM monitoring stop at the dashboard. This one covers the signals worth watching and how each is computed, what it costs to score a sample of live traffic, how to alert on a rate without paging someone over noise, where the OpenTelemetry standard stands, and the tools as their docs and pricing pages read in November 2026. It ends with the step most setups skip: turning a production failure into a test case, so the next change is checked against it.

A disclosure: I build EvalShift, which is not a monitoring tool. It covers the testing half of that last step, and it has a short section near the end. The rest works with any tool.

What LLM monitoring is

Three terms get used for overlapping things, and the tools blur them further.

Monitoring is built on observability data: you cannot chart tool-call errors if tool calls are not recorded as spans. So one product does both, and "LLM observability platform" and "LLM monitoring tool" name the same list of vendors.

Monitoring and offline evaluation feed each other. Monitoring sees real inputs and has no expected answer for them. Offline testing has expected answers and only the inputs someone thought to collect.

Why ordinary APM misses LLM failures

An application performance dashboard watches status codes, latency and throughput. An LLM application can fail while all three look healthy.

A failed response returns 200. Anthropic's stop reason docs ↗ say a refusal arrives as "a normal HTTP 200 response, not an error", with stop_reason set to refusal. Google's API reference ↗ lists more than twenty finish reasons on a successful response, among them SAFETY, MAX_TOKENS, MALFORMED_FUNCTION_CALL and TOO_MANY_TOOL_CALLS. OpenAI's chat completion object ↗ carries a refusal field on the message and a content_filter finish reason. None of these moves an error-rate chart.

A wrong answer has no field at all. A reply that invents a refund policy, or an agent that issues the refund without looking up the order, finishes with a normal stop reason. The only way to count these is to check the content.

Cost is a property of each request. A one-line question and a long agent task hit the same endpoint and cost very different amounts. A loop in an agent shows up on the invoice before it shows up anywhere else.

Behavior changes without a deploy. In September 2025 Anthropic published a postmortem ↗ of three infrastructure bugs that degraded Claude's responses. One misrouted 0.8% of Sonnet 4 requests when it was introduced on August 5, and 16% in its worst hour on August 31. Anthropic wrote that "the evaluations we ran simply didn't capture the degradation users were reporting", and that it would start running quality evaluations continuously on production systems. If the provider's own checks can miss a regression in the model you call, your deploy log will not show it either.

LLM monitoring metrics: what to watch

Five groups. The first two are what an APM tool already does, in different units. The last three are specific to LLM applications.

Operational signals

Split errors by type before you chart them, because the responses differ. A 429 means back off or raise a limit, a 5xx means retry or fail over, and an invalid request after a deploy means your request shape is wrong. Current models reject parameters that older ones accepted, so the same error can arrive with a model upgrade.

Rate limits can be watched before they bite. Anthropic's rate limit docs ↗ list response headers such as anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining, plus retry-after when a 429 carries one. Chart the remaining share at peak hour and you see the ceiling before users do.

For anything streamed to a person, time to first token is the latency users feel. Total duration grows with the length of the answer.

Cost signals

Measure cost per task, not per call. An agent task is several model calls plus tool results fed back in as input, and a cheaper model that takes one more round per task can cost more. The cheapest LLM API works through that arithmetic.

Count cached tokens separately, or the cost chart is wrong. In Anthropic's prompt caching docs ↗, the input_tokens field holds only the tokens after the last cache breakpoint, and the total is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. Cache reads are billed at 0.1 times the base input price or less, and a five-minute cache write at 1.25 times. A prompt edit that moves the cache breakpoint can multiply input cost without changing the request count, and the cache hit rate is the chart that shows it.

Quality signals

Run code checks on every response. They cost nothing and never disagree with themselves: the output parses, it validates against the schema, the order ID from the input appears in the reply. Refusals and truncations come free with the response, so count responses by stop reason too.

User feedback is the signal closest to the truth. Do not rely on thumbs alone: add the implicit signals your product already has, such as the user rephrasing the same question, editing the draft before sending it, or asking for a human.

For everything else, such as groundedness and whether the question was answered, the option is an LLM judge on a sample. That has its own section below.

Agent signals

An agent adds failures that sit between the model call and the reply. A tool execution fails, and the model may recover or may tell the user it worked. A task loops on the same search until it hits the cap.

Chart the tool-call error rate per tool, steps per task at p95, and the count of tasks that hit your step limit. All three need tool calls recorded as their own spans, which is the first thing to check in a tracing library. Agent evaluation metrics has the definitions, and evaluating agent tool calls covers why the reply alone hides these failures.

Change detection

Record the model that answered, not only the model you asked for. Each provider's response names it: OpenAI's model field is "the model used for the chat completion", and Gemini's modelVersion is "the model version used to generate the response". OpenAI also returns a system_fingerprint, which "represents the backend configuration that the model runs with".

This matters because the two can differ. Google's model docs ↗ say a latest alias points to the newest release and is swapped with two weeks' notice by email, and its changelog ↗ records gemini-pro-latest and gemini-flash-latest switching to the Gemini 3 previews on January 21, 2026. The same changelog says that from December 1, 2026, requests to gemini-2.5-pro will automatically route to gemini-3.8-flash, and requests to gemini-2.5-flash to gemini-3.5-flash-lite. Those requests will keep returning 200.

So chart responses by the model named in the response, and alert when a value appears that was not there yesterday. Then put the provider's retirement dates in a calendar. The deprecation schedule lists them for OpenAI, Anthropic and Google.

Online evals: scoring a sample of live traffic

An online eval is a check that runs on production traces after they are logged. Every platform in the table below has the same three controls: a filter for which traces to score, a sampling rate, and a choice between a code evaluator and an LLM judge. They run in the background. Braintrust's docs ↗ say evaluations run "asynchronously in the background without adding latency to your application".

Sampling is the cost control. Arize's online eval guide ↗ suggests 100% for low-volume or critical applications, 10 to 50% for high volume and 1 to 5% for very high volume, starting at 10 to 20% until you trust the evaluator. LangSmith's example ↗ is a rate of 0.1, and Datadog's is 10%.

What a judge on production traffic costs

There are two bills: the judge model's tokens, and whatever the platform charges for holding the result.

For the tokens, Langfuse's LLM-as-a-judge docs ↗ estimate $0.01 to $0.10 per assessment, depending on the judge model and the size of the input. Take 100,000 traces a month, score 10% of them with two judges, and you run 20,000 assessments: $200 to $2,000 a month at that estimate. Score every trace and it is $2,000 to $20,000.

The platform side, as of November 2026:

• +Langfuse counts every score as a billable unit, alongside traces and observations.
• +LangSmith has no per-score charge, but a new online evaluator upgrades the traces it scores ↗ to 180-day retention unless you opt out, at $2.50 per 1,000 on top of the base trace price.
• +Braintrust counts scores: 50,000 are included on Pro, then $1.50 per 1,000.
• +Datadog bills on LLM spans and lists evaluations as unlimited on both plans.

LangSmith also lets you cap an evaluator's LLM spend per week, after which it pauses until the limit resets. If your platform has that setting, use it before the first rule goes live.

Three rules for a judge in production

Give it one criterion with a pass or fail answer. Check its verdicts against 30 to 50 traces you labeled by hand before you alert on it. And filter before you sample: a judge for refund conversations should see only refund conversations. When to trust an LLM judge has the audit procedure, and LLM-as-a-judge covers the known biases.

Alert on a rate with its uncertainty

The alerting built into these tools is a threshold on an average over a time window. LangSmith's alerts ↗ cover run count, cost, errors, feedback score and latency. Braintrust's examples ↗ include "Score drops below 0.8". That works for latency and error counts, where an hour holds thousands of data points. It breaks on sampled quality scores, where the window may hold fifty.

Say the judged pass rate has been 92% and the alert fires below 90%. Today's sample has 100 scored traces and 87 passed. The 95% interval around 87 out of 100 runs from 79.0% to 92.2%, and the baseline sits inside it. Nothing has been shown to change. With 1,000 scored traces and 870 passes the interval is 84.8% to 88.9%, the baseline is outside it, and the drop is real.

Three settings follow from that:

• +Size the window to the sample. Choose the window so it holds a few hundred scored traces, even if that makes it a day and not an hour. A quality alert that needs a day is still faster than a support ticket.
• +Compare with a baseline, not a fixed number. Alert when this window's interval excludes last week's rate. A fixed threshold fires the day traffic shifts toward harder questions.
• +Keep hard failures on counts. Schema failures, 5xx errors, tool exceptions and an unknown model in the response need no statistics. Alert on the first few.

How many eval cases do you need has the same arithmetic for an offline suite.

OpenTelemetry for LLM monitoring: where the standard stands

OpenTelemetry has semantic conventions for generative AI: agreed names for the spans, attributes and metrics an LLM call produces, so that any instrumentation library can feed any backend. They are what lets you switch monitoring vendors without re-instrumenting. They are also not finished.

As read on November 13, 2026:

• +The conventions moved out of the main specification. The pages on opentelemetry.io ↗ now say they are no longer maintained there and point to a dedicated repository, semantic-conventions-genai ↗, created in May 2026.
• +Every page in that repository is marked Development, not Stable. The repository has no tagged release, and its README lists the schema URL as "TODO".
• +The unreleased changelog holds 20 entries marked breaking. One replaces the gen_ai.client.token.usage histogram with separate input and output token metrics. Another renames gen_ai.usage.cache_creation.input_tokens to gen_ai.usage.cache_write.input_tokens.

What the conventions define maps closely onto the table above:

Prompts and completions are off by default. The span conventions ↗ say instrumentations should not capture instructions, inputs or outputs unless the user opts in. That is the right default, and it means a judge has nothing to read until you turn content capture on and decide where the content lives.

Backends already accept the format. Datadog says ↗ it natively supports the GenAI conventions from version 1.37 up. Langfuse receives OTLP ↗ and maps gen_ai.* attributes onto its own data model. Phoenix ↗ is built on OpenTelemetry with its own OpenInference instrumentation.

My advice: instrument through OpenTelemetry, because it is the only portable option, pin the instrumentation library's version, and expect renames until the conventions reach Stable.

LLM monitoring tools

Checked against each vendor's docs and pricing page on November 13, 2026. Prices are list prices for the cloud product.

Notes on reading it:

• +The meters differ. Langfuse counts every trace, observation and score. LangSmith counts traces. Braintrust counts gigabytes and scores. Datadog counts LLM spans only, not tool, workflow or retrieval spans. An agent that makes ten model calls per request is one LangSmith trace, ten Datadog LLM spans and eleven or more Langfuse units. LangSmith alternatives and Braintrust vs LangSmith work through examples.
• +Retention is short. 14 days on LangSmith's base traces and Braintrust's free plan, 15 on Datadog and AX Free, 30 on Braintrust Pro. A failure you mean to turn into a test has to be exported before then.
• +Datadog renamed the product. Its docs and pricing page call it Agent Observability, and the URLs still say LLM Observability.
• +Helicone has changed hands twice. Mintlify acquired it ↗ in March 2026 and said it would continue in maintenance mode. As of mid-November 2026 its site carries a banner saying it is now part of Experiential Labs, with details to come. Check where that stands before you build on it.

How to choose: if traces must stay on your infrastructure, Langfuse, Phoenix or Opik. If you are on LangChain or LangGraph, LangSmith. If you already pay for Datadog, start there. If experiments are the center of your workflow, Braintrust. The best LLM evaluation tools compares the same platforms on their evaluation features.

From alert to test case

Monitoring tells you after the fact: a user saw the wrong answer before the judge scored it. What monitoring can do is make sure each failure happens in production once. That takes five steps, and the monitoring tool does none of them for you.

1. Open the trace, not the chart. Find one failing request and read all of it: the input, the tool calls, the tool results, the reply.

2. Export it before retention deletes it. Copy the input, the conversation so far and the tool results into a file in your repository, with personal data removed.

3. Write the expected behavior by hand. This is the step that gets skipped. The recorded run is what the agent did, and it did the wrong thing. The test case needs what it should have done: the right tool, the right arguments, the fact the reply must contain.

4. Tag it with the failure. refund-without-lookup, wrong-language, loop. The tag is how you know later why the case exists.

5. Run the suite before the fix ships, and on every change after. The fix has to pass the new case without failing the old ones. From then on a prompt edit, a model upgrade or a provider's forced migration is checked against that failure before release. LLM regression testing in CI shows the pull-request version, and eval-driven development is the same loop treated as the way you build.

The loop runs the other way too. When a retirement date appears or an alias is about to move, you have a suite made of your own worst days to run the new model against before the provider switches it for you. Build a golden suite from production traffic covers collecting the ordinary cases that sit next to the failures.

Where EvalShift fits

Disclosure again: I'm the founder. EvalShift does not do production monitoring. It has no dashboards, no alerts and no online evals, so run one of the tools above next to it. What it covers is steps 2 to 5.

The capture SDK records agent runs in production: the input, each model call with its model ID, token counts and latency, and each tool call with its result. It writes them as files to local disk or to a bucket you own on S3, Google Cloud Storage or Azure, and nothing is sent to EvalShift. Redaction runs in process before a capture is written, and EVALSHIFT_SAMPLE_RATE records a fraction of runs.

One command turns a recorded run into a case in the golden suite:

evalshift capture promote cap_3f9a2c --suite support --tag refund-without-lookup

Two details matter for failures. promote refuses a capture whose trace contains an error event unless you pass --allow-errored. And the expected tool calls it writes are marked provenance: captured, which the docs describe as transcribed from the model's own recorded call, where nobody has checked it is right. For a failure it is wrong by definition, so you edit the case file to the correct calls and mark them reviewed. That is step 3.

After that, evalshift compare runs the current model and a candidate over the suite and returns a verdict from the migration policy you wrote, and the GitHub Action does the same on a pull request. With captures.max_age set, evalshift doctor warns when the newest case in a suite is older than that, or when cases were recorded on a model other than the one you run now.

FAQ

What is LLM monitoring? LLM monitoring is watching an application built on a language model while real users are on it: whether model calls succeed, how long they take, what they cost, and whether the answers and tool calls are still correct. The last part needs checks on the content, because a wrong answer returns the same HTTP 200 as a right one.

What is the difference between LLM monitoring and LLM observability? Monitoring answers whether something is wrong right now, with metrics over time and alerts. Observability answers why one request went wrong, with a trace of every model call, tool call and retrieval in it. Monitoring is built on observability data, so one product usually does both.

What metrics should you monitor for an LLM application? Error rate by type, latency and time to first token; tokens and cost per task and the cache hit rate; the pass rate of code checks, a judged quality score on a sample, user feedback and the refusal rate; for agents, tool-call errors, steps per task and loops; and the model named in each response.

How do you monitor LLM output quality in production? Run code checks on every response, count responses by stop reason, collect user feedback, and run an LLM judge on a filtered sample of traces with one pass-or-fail criterion per judge. Validate the judge against 30 to 50 hand-labeled traces before alerting on it.

How much does it cost to run LLM judges on production traffic? Langfuse's docs estimate $0.01 to $0.10 per assessment, depending on the judge model and input size. Scoring 10% of 100,000 monthly traces with two judges is 20,000 assessments, or $200 to $2,000 a month in judge tokens, before any per-score or retention charge from the platform.

What are the best LLM monitoring tools? As of November 2026: Langfuse, Arize Phoenix or Opik if traces must stay on your own infrastructure, LangSmith for teams on LangChain or LangGraph, Braintrust for experiment-centered teams, and Datadog Agent Observability for teams already on Datadog. All of them trace requests, and all but self-hosted Phoenix add online evals with sampling and alerts; for Phoenix those are on Arize AX.

Is OpenTelemetry ready for LLM monitoring? It is usable and not yet stable. As of November 13, 2026, the GenAI semantic conventions live in their own repository, every page is marked Development, there is no tagged release, and the unreleased changelog holds 20 entries marked breaking. Instrument through it, pin the library version and expect attribute renames.

How do you detect that a model changed in production? Record the model named in each response, not only the one you requested, and alert when a new value appears. Gemini returns it in the modelVersion field and OpenAI in the model field. Add the provider's deprecation dates to a calendar, because a rerouted request still returns 200.

Keep reading

• +LLM testing: the types of tests that run before release, and what to rerun when the model changes.
• +Build a golden suite from production traffic: recording real runs and promoting them to test cases.
• +When to trust an LLM judge: auditing a judge before it scores live traffic.
• +LLM model deprecation schedule: the retirement dates to put next to your alerts.