LLM Observability: Knowing What Every Prompt Cost and Did

ai gateway is the discipline of knowing, for every prompt your application sends to a model, which model answered, how many tokens it used, how long it took, what was sent, what came back, and whether it failed. It turns a monthly bill into an audit trail instead of a lump sum. We covered per-request receipts in the product docs, and the rate card for GPT-5.6 Terra shows the per-million-token prices each logged row should reconcile against; this piece is the plain-English case for why per-request logs — not dashboards — are the observability that actually pays for itself.

Here is the uncomfortable truth: the cost that most teams are trying to observe is not a single number. It is the sum of thousands of individual decisions, each one a model call with its own model, token count, latency and failure mode. Aggregate charts hide all of that. A dashboard that tells you "spend was $4,300 this month" is not observability; it is a bill with extra steps. Observability starts one level down, at the level where the money is actually spent — the individual request.

What LLM observability actually is

LLM observability is per-request logging with enough fields to reconstruct what happened, when, and at what price. The minimum useful record looks like this:

• Model: which model ID served the request, because cost and behavior differ wildly across a model catalog.

• Tokens: input tokens and output tokens separately — the two sides of an LLM bill are priced differently, and output tokens are usually the expensive side.

• Latency: time to first token and total duration, so slow generations are attributable to a specific model and prompt.

• Cost: the dollar amount for that exact request, computed from tokens at the list price.

• Prompt and response: what was sent and what came back, so you can reproduce a bad answer.

• Status: success, or the error, timeout, retry or fallback that took its place.

Everything above the request — aggregate spend charts, per-model breakdowns, error-rate trends — is derived from these records. Get the per-request layer right and every other question becomes a query. Skip it, and no dashboard can manufacture the data it needs, because the underlying records were never captured.

Why it matters: debugging, attribution, compliance, security

Four problems only per-request observability can solve. The first is debugging bad outputs. When a user reports a nonsense answer, you need the exact prompt, the exact response, and the exact model that produced it — not a guess about which of your five providers was in the rotation that afternoon. Without the prompt-and-response record, the incident is unreproducible.

The second is cost attribution. A flat monthly bill tells you nothing about who spent what. Once every request is tagged and logged, the bill splits cleanly by team, feature or tenant. That matters the moment finance asks why spend tripled, and it matters again when you need to decide whether an expensive model is earning its keep on a specific feature or quietly leaking money in a rarely used one.

The third is compliance. If you process regulated data, you need to prove what was sent to which model, under which policy, and what happened to it. A request log is that proof; a vendor invoice is not. The fourth is security. Prompt injection, data-exfiltration attempts and runaway retry loops all leave traces — but only if the traces are captured before you need them.

The metrics that matter

Every observability stack should track a small, decision-ready set of metrics. A good starting board looks like this:

The unifying thread: every metric above is a query over per-request logs. Cost per task is just summing the cost field by task tag. p95 latency is just a percentile of the latency field. If the underlying requests were never logged, none of these numbers can be computed honestly — they are estimates at best, invented at worst.

"We got a bill" versus "we can audit every request"

The difference between the two states is exactly one layer of logging. "We got a bill" means a provider sent you an invoice, and you have aggregate truth. "We can audit every request" means that for any prompt in the last month, you can pull the model, the token counts, the latency, the cost and the outcome — and you can reproduce the math from list price to the cents on the invoice.

This is where a router earns its keep. OrcaRouter, for example, treats the request log as a core feature rather than an add-on: every request is recorded with its model, tokens, latency, cost and status, so each row is an auditable receipt [OrcaRouter]. The same logs power budgets and roles, which let you cap and attribute spend per team before the month ends [OrcaRouter]. And because pricing is transparent — provider list prices pass through with no markup added, so a "glass-box" receipt shows you exactly what the vendor charges and what the router itself added, which is nothing [OrcaRouter] — the cost field in the log is vendor truth, not a reseller's estimate.

That last point matters more than it sounds. Per-request cost is only useful if it is accurate. If your gateway or marketplace marks up model prices, every per-request cost field is silently wrong, and your "observability" is measuring your middleman's margin rather than your workload. Zero-markup pricing is what makes the logged cost the real cost.

The takeaway

Build per-request observability before you optimize anything else, because every optimization — cheaper models, caching, tighter prompts — is measured against it. It pays off fastest for teams with multiple models, shared infrastructure, or any need to prove what was sent and what it cost. It's overkill only if your model usage is a single provider, a single feature and a single contract; in that world, the invoice is the log. Everyone else should ask a simple question of every tool they evaluate: can it show me the model, tokens, latency, cost and response for any single request, or does it only show me the average? The first is LLM observability. The second is a bill.

Sourcing note: product facts and the definitions of OrcaRouter's request logs, budgets & roles and zero-markup glass-box receipts are OrcaRouter's own published claims, verified August 22, 2026. Keyword volume and difficulty figures are from Ahrefs (country = US), checked August 22, 2026. All other content is general engineering practice with no third-party vendor data claimed.