AI cost controls

Generative AI spend is variable, bursty, and easy to misattribute. A successful pilot can double API bills when adoption rises or when users paste entire documents into chat. Finance needs predictability; engineering needs unified model gateway that do not require emergency outages.

Cost control is not stinginess. It is accountability: who may spend, on which workflows, with what caps, and who approves exceptions. Uncapped pilots erode trust with procurement (production readiness conversation) and encourage shadow API keys (OWASP LLM Top 10).

Treat cost like safety: controls belong in the request path, with telemetry and alerts, not as a monthly surprise in a spreadsheet.

Forecast **cost per successful task (unified model gateway)**, not raw token totals alone. Define "successful" the same way as quality rubrics in the measuring success guide (OpenAI evals).

Provide ranges for pilot vs steady state: low, expected, high scenarios driven by adoption and context length. Explain variance drivers using measuring success metrics.

Show who pays: central platform budget, business unit showback (unified model gateway), or project code. Ambiguous funding creates late political fights.

Include non-token costs: vector index (Azure vector search), embedding refresh, content safety (Azure Content Safety) API calls, log storage, and human review (OpenAI safety best practices) time for HITL workflows.

Spend scales with model tier, input tokens, output tokens, and call count. Long retrieved contexts and chat history dominate many RAG (Azure RAG concepts) bills more than answer length.

Embedding costs accrue on index build and refresh, not only queries. A nightly re-index of a large SharePoint (Azure RAG concepts) corpus can exceed chat spend if unplanned.

Safety and classification calls add overhead per request. Budget Content Safety and Bedrock Guardrails explicitly rather than treating them as negligible.

Educate sponsors: cheaper models reduce cost but may increase rework and human review (OpenAI safety best practices) minutes. Optimise fully loaded cost, not tokens alone.

**model routing (unified model gateway)** sends classify and triage tasks to smaller models and reserves large models for generation that needs reasoning. Route by task type, not globally per app.

Context limits cap retrieved passages, chat history depth, and maximum output tokens. Enforce limits in middleware per RAG guidance, not only in prompt instructions.

Retrieval top-k and chunk size directly affect input tokens. Tune for citation quality vs cost; measure empty-result rate when trimming aggressively.

Caching helps only where answers are stable and privacy policy allows. Never cache personalised or restricted responses without explicit policy review.

Implement per-session, per-user, and per-tenant caps where multi-team platforms serve many units. Tags on every request enable enforcement.

Choose hard blocks vs soft alerts by risk appetite. Hard blocks stop spend; soft alerts notify owners before breach. Production often uses soft alerts for teams and hard caps for sandbox.

Define graceful degradation when caps approach: shorter answers, disable expensive tools, queue to human, or switch to smaller model with user-visible notice.

Never fail silently with empty responses. Users retry and multiply cost. Clear messages reduce thrash.

An AI Gateway (gateway documentation) centralises provider keys, model routes, failover, and spend attribution. Application teams call one endpoint; platform teams change routes without redeploying every app.

Failover to a secondary model avoids outage but may change cost profile. Document failover pricing and quality trade-offs in runbooks.

Use gateway analytics to compare cost and quality by route on the same golden set (OpenAI evals) before promoting a cheaper default.

Avoid per-team API keys (OWASP LLM Top 10) scattered across repos. Keys without gateway telemetry cannot be capped or charged back reliably.

A retailer scales HR policy Q&A from 500 to 8,000 users. Initial spend forecast assumed 3 turns per session; real usage averages 7 turns with long pasted emails.

Platform adds history summarisation after turn 4, reduces retrieval top-k from 8 to 5 with eval monitoring, and routes clarification questions to a small model. cost per successful task (unified model gateway) falls 35% with citation rate (Azure RAG solution guide) stable.

Finance receives weekly showback (unified model gateway) by region. ANZ overspend triggers review; root cause is a champion sharing an uncapped demo link externally. Caps and SSO (Entra conditional access) fix the leak.

Steering approves continued scale with quarterly routing review and mandatory cost line in monthly metrics.

An energy company runs **overnight batch triage (OpenAI function calling)** on 20,000 tickets monthly. Interactive chat caps do not apply; batch job cost dominates.

Team uses batch API where available, smaller model for category suggestion, and large model only on low-confidence cases. Embedding cache avoids re-processing unchanged ticket templates.

Job-level budget cap stops runaway loops when upstream feed duplicates records. Alert pages platform before invoice shock.

Business case compares batch AI cost to contractor hours previously spent on manual sort.

Tag logs early with team, workflow, cost centre, and environment. Retroactive tagging after finance asks is painful and inaccurate.

Showback reports actual spend by tag without internal invoicing; good for pilots. **chargeback (unified model gateway)** assigns cost to business units; needed at scale.

Publish a simple monthly statement: tokens, safety calls, embedding refresh, index storage, by team. Champions use statements to prioritise high-value workflows.

Platform retains central negotiation with providers; business units consume via tags, not separate enterprise agreements unless approved.

Pilots may use soft caps and alerts with weekly finance check-ins. Production requires hard caps, degradation behaviour, and documented approvers for overrides.

Sandbox environments get lower caps and smaller models by default. Prevent accidental production keys in dev via separate accounts or gateway policies.

Before go-live, run a load test at expected peak with cost projection. Peak day after policy announcement exceeds steady state easily.

Align cost bar with production readiness (checklist guide) checklist: caps are a gate item, not optional.

Cost anomalies often indicate bugs, abuse, or **prompt injection (OpenAI mitigations) loops**, not legitimate adoption. Alert on spend rate change, tokens per request spike, and tool loop count.

Correlate cost spikes with deploy events, index refreshes, and marketing announcements driving traffic.

Dashboards should show cost alongside quality metrics. A cheap model that increases escalations may raise fully loaded cost.

Export provider invoices and reconcile to internal logs monthly. Discrepancies reveal untagged keys or wrong region routing.

Procurement (production readiness conversation) will ask for commit discounts, enterprise agreements, and spend predictability. Bring tagged usage history from pilots, not theoretical estimates.

Council should review portfolio spend monthly: which workflows earn continuation, which should be capped or stopped on cost grounds alone.

Publish a one-page cost policy: default caps, tagging rules, approvers, and prohibited patterns (unlogged keys, uncapped public demos).

Pair cost policy with measuring success guide (OpenAI evals) metrics so kill decisions use cost and quality together.

Use this checklist before scaling any pilot beyond the initial cohort.

Common mistakes include caps only on paper, ignoring embedding refresh, and optimising tokens while human rework soars.

What good looks like: finance, platform, and sponsors cite the same cost per successful task (unified model gateway) number in meetings.