AI LabsbyThe Ops ToolboxOps Toolbox
All examples
Live demo

Production Observability for AI Routes

See latency, tokens, matched documents, and rough cost on every request — ready for SRE and finance.

GovernanceQuality checksDocument Q&A
Jump to demo

You're on call reviewing a production request — inspect latency, token usage, and rough cost like SRE and finance ask for in week six.

Ops dashboard

Latency, tokens, document matches, and rough cost per request.

SRE on-callProduction request review
Live
Scenario: External AI policy
Support · Live tracePolicy
Trace ID
tr_8f2a91c
Route
/api/support/copilot
Question
External AI policy
Latency
Pending
User
Tier-1 · session 8821
Case studyArchitecture, governance, and how to adapt this pattern in a pilot

Business use case

Problem

Pilots ship without run logging, when something goes wrong, nobody can answer which model, which retrieval, or what it cost.

Who benefits

  • Platform engineering, SLOs and dashboards per route
  • FinOps, token spend visibility before finance asks
  • Incident response, trace id ties user report to logs

Success metrics

  • 100% of production AI routes emit structured telemetry
  • P95 latency tracked per model and per retrieval mode
  • Weekly review of empty-retrieval rate on RAG paths

Solution

Wrap generateText (with optional seed RAG) and return a telemetry object alongside the answer, pattern for OpenTelemetry export, log drains, or your observability stack.

Technical implementation

Stack

  • AI SDK generateText with usage tokens
  • searchSeedDocuments for retrieval slice timing

Architecture

Every request returns structured telemetry alongside the answer.

How it runs
Drawing the flow…

Outcomes and learnings

  • Log retrieval empties separately from model errors, different fixes
  • Cost estimate is indicative; wire real pricing tables in production
  • Same shape works for agents, batch jobs, and workflows with step spans
Delivery playbookDiscovery → pilot → scale
  1. 1
    Discovery2–4 wks

    Agree mandatory telemetry fields and retention with platform and risk.

  2. 2
    Pilot6–8 wks

    Dashboard P95 latency, token spend, and empty-retrieval rate for one route.

  3. 3
    Scaleongoing

    Export to OpenTelemetry; alert on safety flags and cost anomalies per tenant.

Where else this appliesObservability is what turns a demo into an operated service, finance, SRE, and risk all ask different questions from the same trace.

FinOps chargeback

Token and cost estimates per team, model, and feature flag.

Incident debugging

Support ties a bad answer to retrieval hits and model version within minutes.

RAG quality ops

Alert when empty-retrieval rate spikes after index or taxonomy changes.

Vendor routing reviews

Compare latency and spend when gateway routes change between models.

Emit structured JSON from every AI route; forward to your log drain, OpenTelemetry collector, or your observability stack with consistent traceId propagation.