Human in the loop: designing oversight that scales

"Human in the loop" is the most reassuring phrase in enterprise AI. It is also one of the least examined. Leadership teams hear it and assume the risk is handled. Often it is not.

A person who approves two hundred agent actions a day is not exercising judgement on each one. They are clicking. The control exists on paper, the cost is real, and the protection is thin. Oversight needs to be designed with the same care as the agent itself.

Three models of oversight

There are three basic ways to keep people involved in what an agent does. Each suits different work.

Approve before act. The agent prepares the action and a person authorises it. Nothing happens without a decision. This offers the strongest control and the least efficiency gain. It suits actions that are costly, irreversible or sensitive: issuing a large refund, changing contract terms, sending a communication that carries legal weight.

Review after act. The agent acts, and a person reviews a sample or the full set afterwards. Errors are caught and corrected, but not prevented. This suits actions that are low in cost and easy to reverse: updating a record, drafting an internal summary, routing a case.

Exception only. The agent acts on its own within defined limits and escalates only when it meets something outside them: low confidence, an unusual request, a customer asking for a person. This offers the most efficiency and relies most heavily on the quality of the limits and the monitoring.

Most organisations will use all three at once. The skill is in matching the model to the work.

Tiering by risk

We suggest scoring each agent action against four factors.

• Cost of an error. Financial, reputational or regulatory.
• Reversibility. Can the action be undone, and how quickly?
• Who is affected. An internal colleague, a customer, or a vulnerable customer.
• How detectable errors are. Will a mistake be obvious, or could it sit unnoticed?

High scores point to approval before action. Low scores point to exception-only handling. The middle ground belongs to review after action, with the sample size set by how much evidence you have.

Consider an agent supporting a sales team in Agentforce Sales. Logging call notes is low risk: exception only. Drafting a follow-up email is moderate: the seller reviews before sending. Applying a discount beyond the standard range is high: a manager approves first. One agent, three tiers.

The tiering should be written down and agreed by the business owner and the risk function. It should also be revisited. An action that needed approval in the first month may have earned review-after status by the sixth, once the evidence supports it. Oversight that never loosens suggests nobody is looking at the evidence.

Avoiding the rubber stamp

A human approval step fails quietly. The approver trusts the agent, the queue is long, and approval becomes automatic. Several design choices help prevent this.

1. Keep volumes honest. If a reviewer cannot give each item real attention in the time available, the tier is wrong or the team is too small. Do not pretend otherwise.
2. Show the reasoning, not only the result. A reviewer should see what the agent relied on: the records it read, the policy it applied. Without that, they can only agree or guess.
3. Make rejection as easy as approval. If approving takes one click and rejecting takes a form, the outcome is predictable.
4. Measure the reviewers. Track approval rates and time per decision. A reviewer who approves everything, at a few seconds an item, is not a control.
5. Test the control. Periodically place known errors in the review queue and see whether they are caught. This is standard practice in quality assurance and applies equally here.
6. Give reviewers authority. The people overseeing an agent should be able to pause it. If they can only raise a ticket, they are observers.

A reviewer who never says no is not a control. They are a delay.

Designing the escalation

Exception-only oversight depends on the handoff. When an agent passes work to a person, three things should be true. The person receives the full context, so the customer does not repeat themselves. The reason for escalation is stated. And the person has the time and authority to resolve the matter.

The hard part is organisational: making sure someone is on the receiving end, with capacity, at the hours the agent operates.

Oversight as an operating model

The broader point is that oversight is not a feature to switch on. It is a set of roles, measures and routines that someone has to own. Who reviews? How many items? Against what standard? Who sees the results, and how often? What triggers a change in tier?

Organisations that answer these questions early can extend their agents with confidence, because they have evidence for each step. We encourage leadership teams to design the oversight model alongside the agent, not after it, and to treat it as the thing that earns the agent its autonomy.