Human Oversight for AI Agents — Control Without Approval Fatigue

COMPANY

Under NDA

DOMAIN

B2B SaaS · AI operations

DURATION

5 months

TEAM

1 designer (me) + PM + ML + Frontend + QA

TYPE

Human-in-the-loop UX · Trust & control

STATUS

Shipped · piloted with 3 customer teams

TL;DR

Autonomy customers actually turn on

The product lets customer-operations teams run AI agents on refunds, order changes, subscription updates and customer replies. The first release asked a human to approve every action that changed anything, and it backfired: reviewers faced ≈280 near-identical requests a day, approved most of them in under two seconds and still missed the bad ones. Customers kept the agents in “suggest only” — the core value never switched on.

I redesigned oversight around risk instead of volume: four tiers from “act and log” to “stop and decide,” consequence-first approval cards, batch review with outlier sampling, undo windows, and a pause that’s always one click away. In a six-week pilot with three customer teams, approval load per reviewer fell by ≈85%, planted errors caught in testing went from 3 of 8 to 7 of 8, and all three teams kept autonomy switched on after the pilot.

RESULTS

What changed.

APPROVAL LOAD PER REVIEWER

−85%

≈280 → ≈40 decisions a day

PLANTED ERRORS CAUGHT

7 of 8

up from 3 of 8 · moderated test

TIME TO PAUSE AN AGENT

<5 s

down from ~2 min

UNAPPROVED HIGH-RISK ACTIONS

0

in 6 weeks of pilot

Metric

Before

After

Approval requests per reviewer / day

≈280

≈40

Agent actions running without pre-approval

≈30%

≈70%

Approvals made in under 2 seconds

61%

6%

Median time on high-impact approvals

≈3 s

≈19 s

Customer-facing incidents per month (pilot teams)

4

1

“I know what the agents are doing” (1–7 survey)

3.4

5.9

Pilot: 3 customer teams, 15 reviewers, 6 weeks · tests: N=8 per round · directional, not statistical.

HOW I MEASURED IT

Three sources, each covering a blind spot of the others. Pilot event logs gave the volume numbers: decisions per reviewer, time on card, undo usage. Two rounds of moderated tests with a planted wrong refund showed whether people actually catch errors — the one thing logs can’t show. A short survey before and after the pilot covered trust. The weak spot is scale: three teams over six weeks, one of them in a quieter season. That’s why the claims stay directional — but all three sources point the same way.

PILOT FEEDBACK

I can finally see what the agents did overnight without asking anyone.

— Ops lead

The batch view is the first time approving felt like checking, not clicking.

— Support specialist

Audit used to take a day of exports. Now it’s a filter.

— Finance & compliance

01 — CONTEXT

Agents act. People answer for it.

Our customers are e-commerce and subscription businesses with large support operations. A typical pilot team runs six AI agents on about 2,000 actions a day — refunds, order changes, subscription updates, address fixes, customer replies, fraud holds — with five people responsible for what those agents do.

OPS LEAD

Sets how much the agents may do on their own and answers for the outcome. Needs the whole operation at a glance and a way to change autonomy without engineering. Decides by “what’s the worst that can happen today?”

SUPPORT SPECIALIST

Reviews and fixes agent work between their own tickets. Needs decisions that take seconds when the case is clear, and real evidence when it isn’t. Decides by “does this match what the customer asked?”

FINANCE & COMPLIANCE

Audits money and data actions after the fact. Needs a trail that explains every action: who or what did it, under which rule, and whether it was reversed. Decides by “can I explain this to an auditor?”

Three persona cards: the ops lead, the support specialist and the auditor — context, needs from the console, what breaks, and what oversight has to guarantee

If every action needs a click, no click means anything.

PROBLEM

The first release was built for safety: every agent action that changed anything waited for approval. In practice, approving became the job. Reviewers cleared queues of near-identical requests in seconds, a €4 shipping refund and a €4,000 bulk credit looked exactly the same, and when something went wrong, nobody could say what the agents had done or why.

Customers responded the rational way — they kept agents in “suggest only.” The product’s value, agents doing the work, never switched on, and renewals depended on it. Turning approvals off wasn’t an option either: one wrong bulk refund costs more than the automation saves, and compliance teams needed a trail for every money action.

02 — MY ROLE

What I owned, what I shared.

LED

Discovery: 12 interviews with ops leads, reviewers and finance at 5 customers, 4 shadowed review shifts · action inventory and risk model · console structure and key flows · UX/UI for seven console screens and the UI kit behind them: tokens, type styles, components · two rounds of moderated testing · pilot success metrics with PM

WITH THE TEAM

Risk thresholds and pilot setup with PM · evidence signals and policy suggestions with the ML engineer · real-time queue and undo windows with frontend · audit requirements with a compliance advisor · pilot teams via customer success

Where reviewer attention went. Before: every action in one approval queue, 61% approved in under 2 seconds. After: four lanes by risk tier, people only where a mistake would hurt

03 — PROCESS

Seven stages, tested twice before the pilot.

01

STEP

Discovery

12 interviews, 4 shadowed review shifts, three months of approval logs.

02

STEP

Action inventory

46 agent actions scored for reversibility, money at stake and customer reach.

03

STEP

Risk model

Four tiers: act & log, act with undo, batch review, stop & decide.

04

STEP

Console structure

One console, three jobs: watch, decide, audit.

05

STEP

Test round 1

New prototype vs. the current queue, N=8, planted wrong refund.

06

STEP

Test round 2

Outlier-first batches, consequence-first cards, N=8.

07

STEP

Pilot

3 customer teams, 6 weeks: a week of refusals, two in shadow mode, then autonomy on.

CORE BET

Oversight should scale with risk, not with volume. People see fewer decisions — and every decision they see is worth their attention.

DIDN’T WORK → REBUILT

Four things broke on the way. They stay in the case because they shaped the final design more than the plan did.

Confidence scores

The model’s confidence numbers were badly calibrated: wrong actions often showed 90%+. Reviewers trusted the number instead of the case. I replaced it with evidence checks — which rule the agent applied, what the customer’s history says.

Batch review

In round 1, 5 of 8 participants approved a batch with a planted wrong item inside. Sampling was rebuilt to always include outliers and mark what differs (“15× the median, different carrier”). Round 2: 7 of 8 caught it.

Friction

Type-to-confirm on every high-impact card annoyed people into copy-pasting amounts. It became a reason picker, with amount confirmation only above a threshold.

Autonomy

Pilot teams refused to switch autonomy on in week one. I added shadow mode: for two weeks the console showed what the agent would have done. After seeing 96% agreement on low-risk actions, all three teams turned autonomy on.

04 — KEY DECISIONS

Seven calls behind the console.

01

KEY DECISION

Risk tiers instead of approving everything

When every action needs approval, approval stops meaning anything.

I mapped all 46 agent actions on two axes — how reversible they are and how much they touch: money, customers, data. Each quadrant got its own mode. Reversible and small: the agent acts and logs. Reversible but visible: it acts, and the console shows the action with an undo window. Hard to reverse but small and repetitive: reviewed in batches. Irreversible or high-impact: the agent stops and asks.

A single approval mode treats a €4 shipping refund and a €4,000 bulk credit the same, and people answer both the same way — with a click. Tiers move attention to where a mistake would actually hurt and turn autonomy into an explicit policy the ops lead can adjust. In the pilot, ≈70% of actions ran without pre-approval, and only 9% stopped for an individual decision.

Policies screen: who decides what, by reversibility and impact — four tiers with their share of today’s agent actions and a suggested policy change

02

KEY DECISION

Show the consequence, not the reasoning

Reviewers don’t need the model’s thoughts. They need to know what will change.

Every approval card leads with the effect: what changes, for whom, how much, and whether it can be undone — as a before/after diff. The evidence (the ticket, the order, the rule the agent applied) sits one level down; the agent’s reasoning is collapsed by default, and the raw confidence score is gone.

The first cards put the agent’s explanation and confidence on top. It looked transparent, but it turned every card into a reading task — and a long, fluent explanation with “94%” next to it reads as right, which is exactly the trap. Consequence-first cards let a reviewer decide in seconds when the case is clear and dig in when it isn’t.

Decide screen, annotated: the consequence as the headline, undo in plain words, evidence one level down, reasoning folded

03

KEY DECISION

One decision for forty identical requests

Repetitive work should be decided once — without hiding a single item.

Small actions that can’t be taken back and follow the same pattern are grouped: “40 delay refunds after the same carrier outage.” The reviewer checks a sample, sees what differs from the rest, and approves or rejects the batch — any item can be pulled out.

The first version sampled randomly, and in testing most people approved a batch with a planted wrong item inside. Outlier-first sampling fixed it: the items that differ most are always in the sample, with the difference highlighted. Catch rate went from 3 of 8 to 7 of 8 between the two test rounds.

Batch screen, zoomed in: the pattern check comes first, the outlier stands apart on the dot plot, and it can be pulled out before approving the rest

04

DECISION

Undo windows instead of asking first

For reversible actions, asking first costs more than fixing later. The agent acts immediately and the console shows the action with a visible undo window — 15 minutes for internal changes, longer for anything a customer will see. A refund can only be compensated, not taken back, so refunds never get an undo window and stay in the review tiers. In the pilot, 2.7% of reversible actions were undone — the window gets used, not ignored.

Trail screen, zoomed in: the undo window sits on the action, refunds never get one, and an undo keeps its reason

05

DECISION

Friction proportional to risk

A one-click approve is fine for a €10 credit and dangerous for a €4,000 one. High-impact cards ask for a deliberate step — pick a reason, confirm the amount above a threshold. The console also notices rubber-stamping: when someone approves cards faster than they could read them, it pauses the queue and says so, without blame. Approvals made in under two seconds fell from 61% to 6%.

Friction scales with risk: one click for low impact, a deliberate step for high impact, and a pause when approvals come too fast

06

DECISION

A stop button that is always one click away

Every agent shows its status and has its own pause; the ops lead can pause all of them from any screen. Pausing doesn’t drop work — queued actions wait, running ones finish or roll back. Time to stop a misbehaving agent went from about two minutes (find the setting, or ask engineering) to under five seconds.

Agent screen, zoomed in: a paused agent with nothing dropped, every agent’s status in the sidebar, and pause all from any screen

07

DECISION

Overrides become policy suggestions

When reviewers keep rejecting or editing the same kind of action, the console proposes a rule change — “move damaged-item refunds over €20 from batch to individual review?” — for the ops lead to accept or dismiss. In the pilot it proposed 11 changes, and ops leads accepted 7. Oversight improves the system instead of only recording its mistakes.

Suggestions screen, annotated: a policy change built from overrides, one limit moving one tier, the impact shown up front, and the ops lead deciding

05 — ARTIFACTS

What exists, what you can see.

01

Interview synthesis

12 interviews, 4 shadowed shifts, approval-log analysis

02

Action inventory + risk model

46 actions scored by reversibility and impact, four tiers

03

Personas

Ops lead, support specialist, finance & compliance

04

Console screens

Watch, Decide, Batch, Trail, Agent, Policies and Suggestions

05

UI kit and tokens

43 color variables, 14 type styles, 53 icons, tier badges, buttons

06

Approval card states

Consequence-first card in every tier and state

07

Test plans + pilot dashboard

Two rounds with planted errors, pilot metrics

06 — KEY INSIGHT

Oversight scales with risk, not with volume.

The intuitive answer to “can we trust the agent?” is a human in front of every action. It feels safe and looks responsible — but people adapt to volume, not to risk. After the hundredth identical request, “approve” becomes a reflex, and the one dangerous action arrives looking exactly like the trivial ones.

The console works the other way round: it spends human attention like a budget. Small, reversible actions run on their own, repetitive ones are decided in batches, and only the irreversible ones stop for a person — framed by their consequence, not by the model’s reasoning.

It’s the same thinking as in my DesignOps case, applied to software instead of a team: move the objective checks into the system, and keep people for the decisions that need judgment.

Human-in-the-loop only works if the loop is small enough for a human to care.

07 — REFLECTION

What I’d do differently.

01

Let customers set the thresholds from day one

We shipped default tiers we’d set internally. Two of three pilot teams rewrote them in the first week — the same refund is trivial for one business and critical for another. A threshold workshop at the start would have saved a sprint.

02

Bring ML into the card design earlier

The confidence-score problem cost three weeks. If the ML engineer had been in the first card sketches, we’d have designed around evidence from the start.

03

Make shadow mode the default path to autonomy

We added it after the pilot had started, to unblock trust. It should be the standard onboarding: see what the agent would do, then let it.

08 — WHAT’S NEXT

Where the console goes from here.

#01

On-call view

A phone-sized view for the one job that can’t wait: seeing what’s running and pausing it.

#02

Policy editor with impact preview

Changing tiers and limits directly, with a preview of how many actions each change would send to people.

#03

Incident replay

Stepping through what an agent did, action by action, after something goes wrong.

NEXT CASE

CASE 01 / 05

Design system case — nine-month timeline of the system built alongside product sprints

02

Building a Design System Without Stopping Product Delivery

A 100+ component design system — tokens, Storybook, Chromatic — built without pausing delivery.

B2B SaaS · 2025–2026