How to design human review for AI work

Decide what needs a person

Not every AI output deserves the same attention. A draft summary for an internal meeting and a payment to a supplier are different jobs. Treat them the same and you either waste your team's time or take risks you didn't mean to.

Ask two questions about each task. First, what does a mistake cost if nobody catches it? Think money, a customer's trust, a legal problem or just a few minutes of rework. Second, can the mistake be undone? A wrong tag on a record can be fixed tomorrow. An email to a client can't be unsent.

Put each task into one of four boxes:

One more filter: does the output leave the building? Anything a customer, regulator or supplier will see deserves a closer look than anything that stays internal.

Choose a review pattern

There are four common ways to involve a person. Most systems use more than one, depending on the task.

• Approve every item. Nothing happens until a person says yes. Right for high-stakes work and the first weeks of any new system. The risk is fatigue: a reviewer approving 200 items a day starts to skim.
• Approve above a limit. Items under a dollar amount, or within normal ranges, go through. Anything bigger or unusual waits for a person. Limits come from your own history, not a guess.
• Check a sample. Everything goes through, and a person reviews a random slice afterward. Good for high volume with cheap mistakes. The sample must be random. If reviewers pick, they pick the easy ones.
• Review the exceptions only. The system sends a person anything it flags: low confidence, a missing field, a value that doesn't match the source. Efficient, but only as good as the flags.

A warning on confidence scores. Many tools attach a score to each answer. Before anyone routes work by it, test it on your own examples. Does a high score really mean the answer is usually right? If not, route by rules you can check, such as “the total doesn't match the line items” or “this supplier is new.”

Design the review screen

The review screen decides whether review is real or theater. If checking an item means opening three tabs, people will stop checking. The careful path has to be the fast path.

Two made-up examples show what that means.

A finance team reviews AI-coded invoices. The screen shows the invoice beside the suggested account and cost center, with last month's coding for the same supplier underneath. A changed account is highlighted. The reviewer approves with one key, or edits in place and picks a reason from a short list. Right items take seconds. Wrong ones stand out.

A service company reviews AI-drafted replies to customer emails. The original message sits next to the draft, anything the system wasn't sure about is marked, and the approve button says what happens next: this sends to the customer. A rejected draft goes back for another try, or to a colleague, so nothing piles up.

What belongs on your screen, which reasons the list offers and where a rejected item goes all depend on the work itself. Getting that right takes time with the people who will do the reviewing, and it is where an experienced builder earns their fee.

Keep reviewers from rubber-stamping

The hardest part of review is human. When a system is right most of the time, people trust it more than they should. Researchers call this automation bias, and have studied it for decades.

In a 1999 study by Skitka, Mosier and Burdick, people working with a very reliable but imperfect automated aid did worse on a monitoring task than people working without it. They missed problems the aid didn't flag, and they followed its advice when other information showed it was wrong.

A review by Parasuraman and Manzey found this complacency shows up when people juggle several tasks at once, appears in experts as well as beginners, and isn't cured by practice alone. A systematic review of 74 studies by Goddard and colleagues listed things that reduced it, including training, a sense of accountability and design choices such as where advice sits on the screen.

Official guidance now names the problem too. NIST's Generative AI Profile describes automation bias as excessive deference to automated systems. The EU AI Act's article on human oversight says people overseeing high-risk systems should stay aware of the tendency to over-rely on the output.

A few things help. Accountability is one: people check more carefully when the approval is visibly theirs. Planted mistakes are another. A few items with known errors go into the queue, and the system tracks whether reviewers catch them. Airport security uses the same idea: a GAO report describes TSA assessing X-ray screeners by projecting fictional threat items onto bag images. Speed is a signal too. If someone approves a two-page contract summary in three seconds, they didn't read it.

Rare mistakes are the hardest to catch. In a study published in Nature, Wolfe, Horowitz and Kenner found people missed targets far more often when those targets were rare. As your system gets better, its remaining errors get rarer, and reviewers become more likely to miss them.

How many planted items, how long a review session should run, and when to ask for the reviewer's own answer before showing the system's: those settings depend on your volumes and your people, and they change as the system matures.

Turn corrections into improvements

Every edit a reviewer makes is free information about where the system is weak. Most teams throw it away.

Kept and sorted by reason, those edits become specific fixes. If “wrong cost center” tops the list for one supplier, the problem is narrow and so is the fix. NIST suggests tracking exactly this. Its AI Risk Management Framework points to collecting data on how often, and why, people overrule an AI system's output.

Corrected items also become a test set, so a change meant to help one supplier can be checked before it quietly breaks another.

Keep a record of who approved what

When someone asks “why did this go out?” six months from now, you want an answer in a minute, not a week. That means keeping the source material, what the system produced, who reviewed it and what they did, and what happened next: sent, paid, filed or returned.

The record should be impossible to edit after the fact, and simple for managers to search. It protects the reviewer as much as the organization. “I approved what I was shown” is easy to prove when the record shows exactly what that was.

Loosen review with evidence

Approving every item forever is expensive. The goal is to review less over time, but only when the numbers say it's safe.

Here is a made-up example. A team starts with a person approving every AI-coded invoice, about 400 a week. After six weeks, reviewers have edited 3% of items, almost all from suppliers seen fewer than three times. The team agrees on a change: invoices from familiar suppliers under a set amount now go through, with a random sample checked afterward. New suppliers and large amounts still need approval. Review time drops, and the risky cases still get a person.

The rules behind a change like that belong in writing first: what counts as enough history, what error rate is acceptable for which mistakes, what sample keeps getting checked, and what triggers tightening again. The answers differ for a payment and a status email, and again for every organization. The sample never drops to zero.

Write down who signed off on each change and why. Loosening review is a decision, and it should leave a record like any other.

What most guides miss

The reviewer isn't always right. People make mistakes too, especially when tired. Have a second person look at a small sample of approved items. You'll sometimes find the AI was right and the edit was wrong.

Speed targets backfire. If you measure reviewers on items per hour, you're paying them to rubber-stamp. Measure caught errors and planted-item catch rates alongside throughput.

Review is a job, so staff it. If reviewing gets squeezed into the gaps between other work, it gets skipped on busy days, which are the days mistakes happen. Put it on someone's calendar.

Reason codes go stale. After a few months, “other” becomes the most common reason. Read those notes, and add or merge codes so the list keeps telling you something useful.

How we do it

When we build AI systems, the review screen gets as much design time as the AI itself: source beside output, one-click decisions and a full record of who approved what. If the bottleneck today is approvals that sit in someone's inbox, see stuck approvals. After launch we stay on to tune the review rules as your volumes and exceptions change, so the system keeps earning trust instead of drifting. A short conversation is the fastest way to work out what review should look like for your work.

Sources

1. Skitka, Mosier and Burdick (1999): Does automation bias decision-making? International Journal of Human-Computer Studies
2. Parasuraman and Manzey (2010): Complacency and bias in human use of automation. Human Factors
3. Goddard, Roudsari and Wyatt (2012): Automation bias, a systematic review of frequency, effect mediators and mitigators. JAMIA
4. Wolfe, Horowitz and Kenner (2005): Rare items often missed in visual searches. Nature
5. NIST AI 600-1: Generative Artificial Intelligence Profile
6. NIST AI RMF 1.0, Appendix C: AI risk management and human-AI interaction
7. EU AI Act, Article 14: Human oversight
8. U.S. GAO-16-704 (2016): TSA should ensure testing data are complete and fully used to improve screener training and operations