Human oversight and approval

Risk and legal teams often hear human-in-the-loop (OpenAI safety best practices) as slowing innovation. In practice, structured oversight unlocks production: supervisors trust proposals they can approve, auditors see decisions, and engineers separate safe read paths from gated writes.

Human oversight spans full manual handling, review of every model output, approve-reject on proposed actions, and periodic quality sampling. Match the level to blast radius and regulation.

Starting read-only is policy, not timidity. Teams that earn automated writes later do so with evidence: low override rates, clear audit trails, and eval coverage on failure modes.

Require human approval before customer-facing writes, financial transactions, HR decisions affecting individuals, legal commitments, and security-sensitive changes per security controls. Regulation and internal policy often mandate this regardless of model confidence.

High-volume read tasks may use sampling oversight instead of per-item approval if quality metrics and stop rules (NIST AI RMF) are strong. Triage suggestions often start at 100% review and move to sample as accuracy proves stable.

Low-risk internal drafts (meeting summaries, internal notes) may auto-save with post-hoc sampling. Do not generalise that pattern to external email without explicit sign-off.

Document oversight decisions in the pilot charter (NIST AI RMF). Changing oversight level is a governance event, not a sprint backlog tweak.

Separate proposal (model or agent generates intent), approval (human or policy gate), and execution (deterministic API call with idempotency). Mixing all three in one model call creates untraceable side effects—see agent vs workflow for separation patterns.

Proposals should be immutable records: what was suggested, on what evidence, at what time. Approvers act on the proposal ID, not a re-generated summary that may drift.

Execution layer handles retries, partial failure, and compensating actions. Workflows excel (AI SDK agents) here; ad hoc agent loops do not.

Rejections should capture reason codes for model and OpenAI evals, not only boolean reject.

Supervisors approve in minutes or the OpenAI safety best practices becomes a bottleneck. Show proposed action, diff against current state, source citations, model confidence, and policy flags on one screen.

One-click approve and reject with optional comment. Bulk approve only for homogeneous low-risk actions with explicit governance exception.

Mobile-friendly queues matter for field managers approving CRM (AI SDK agents) updates. Desktop-only approval UI kills adoption.

Accessibility and language: approvers may not read English fluently. Plain-language summaries alongside technical diffs.

Pending approvals need SLA timers aligned to business process. Procurement (production readiness conversation) approvals may allow 48 hours; chat-assist CRM (AI SDK agents) fixes may need 15 minutes during business hours.

Escalate when queue depth or age exceeds threshold: notify backup approver, route to team lead, or fall back to manual handling without AI.

Measure time-to-approve and queue depth as operational metrics. Rising backlog indicates understaffing or poor proposal quality, not only headcount.

Out-of-hours behaviour should be defined: hold proposals, allow auto-approve for pre-approved low-risk types, or route to on-call.

The person who requests an action should not always approve it. Segregation of duties applies to AI proposals exactly as to manual processes.

Define role matrix: who can propose via agent, who can approve which action types, who can configure tools, who can override safety blocks.

Break-glass override for emergencies requires MFA (OWASP LLM Top 10), reason code, and enhanced logging. Review break-glass usage weekly.

Delegated approval (manager out of office) should follow existing HR delegation rules, not informal Slack consent.

Implement approval as a workflow step with durable state via durable workflow orchestration. Serverless functions alone lose in-flight approvals on timeout unless backed by queue or database.

Agents call read tools freely during proposal phase. Write tools (OWASP LLM Top 10) are disabled or return "pending approval" stubs until execution phase.

Use deterministic validators on execution arguments even after human approve. Approvers trust UI; attackers may tamper API calls—validate with tool calling schemas.

Feature flags can disable all writes globally during incident without taking down read-only RAG Q&A.

An insurer deploys an **ITSM (OpenAI function calling) copilot** that suggests category, priority, and internal notes. Read paths auto-apply suggestions to draft fields only. Changing category on a live ticket requires supervisor approve.

Approval UI shows ticket history, suggested category with similar ticket examples, and policy flag if priority elevated. Median time-to-approve is four minutes during business hours.

After eight weeks, override rate (OpenAI safety best practices) drops below 4%. Council approves auto-apply for category only, with priority changes still gated. Incidents trace to proposal IDs when wrong category applied.

Lesson: narrow auto-write expansion beats full autonomy after one good month.

A government agency uses AI to score vendor submissions against criteria. Model output is advisory only; procurement (production readiness conversation) officer must approve shortlist before RFP progression.

Approval packet includes score breakdown, cited questionnaire answers, and conflict flags. Officers reject proposals when citations missing or criteria weights wrong.

Legal requires all AI recommendations retained seven years with officer sign-off. Execution to ERP (AI SDK agents) creates contract record only after human approval recorded.

No automation bypasses probity rules regardless of model confidence.

A B2B firm generates customer email drafts from CRM (AI SDK agents) context and product docs. Drafts never send automatically. Account manager edits, approves send, and CRM logs sender as human.

Proposal includes retrieved product citations and tone check flags. Content safety (Azure Content Safety) runs before manager sees draft. PII redaction (Azure Content Safety) applied to logged prompts.

Sampling review by team lead on 10% of sent emails monthly. Rising edit distance triggers prompt review.

Customer trust preserved because every external email has human sender accountability.

Some gates should not depend on human judgment every time. **policy-as-code (OWASP LLM Top 10)** blocks proposals above dollar threshold, with restricted countries, or missing mandatory fields before they reach a human queue.

Combine ML proposal with rules engine: model suggests, rules validate, human approves exceptions rules flagged.

Rules change less frequently than prompts; version them separately with audit trail.

Over-reliance on rules without human escape hatch frustrates users. Under-reliance floods approvers with noise.

Track approval rate, rejection rate, time-to-approve, override after approve, and incidents traced to approved actions. Healthy systems show stable approval rate with falling override-after-approve.

Rising rejection rate with reason "wrong citation" indicates retrieval or prompt issues, not approver fatigue alone.

Zero pending queue can mean good automation or users bypassing system. Cross-check with shadow process reports.

Report oversight metrics in the same dashboard as business KPIs from measuring success guide (OpenAI evals).

Teams earn automation by meeting pre-agreed thresholds: e.g. 95% rubric score on golden set (OpenAI evals), override below 3% for 60 days, zero severity-1 incidents, eval CI green.

Council approves oversight level changes with expiry review per NIST AI RMF Govern charter. Auto-write permissions sunset unless renewed with fresh evidence.

Regression triggers step-back: one severity-1 incident may restore 100% review for that action type.

Document the bargain clearly for users: "We remove approver when quality holds, we restore when it does not."

Approvers need 15-minute onboarding: what they are accountable for, what model confidence means, when to reject vs edit, and how to report incidents per evidence pack templates.

Users need guidance on what the agent may propose and what requires their manager. Unclear expectations drive shadow workarounds.

Refresh training when tools, policies, or oversight levels change. Stale training shows up as wrong rejection reasons.

HR and union consultation may apply when oversight affects workload measurement.

Use this checklist before enabling any write tool (OWASP LLM Top 10) in production.

Common mistakes include approval UI that hides the actual write, agents that execute on propose, and approver pools with no SLA coverage.

What good looks like: auditors trace any customer-visible change from user question to human approver to execution log in under ten minutes.