Human oversight in the age of AI agents: Designing for accountability
The oversight illusion
Many organisations believe they have human oversight of their AI systems. They have dashboards. They have exception reports. They have a process for escalating issues. What they often do not have is oversight that is meaningful at the scale and speed at which modern agentic AI systems operate. The gap between the appearance of oversight and its operational reality is one of the most significant governance challenges facing executives today.
The UC Berkeley Agentic AI Risk-Management Standards Profile is direct on this point. It identifies human-computer interaction and loss of control as two of the eight major risk categories unique to agentic systems. Both are fundamentally about what happens when human oversight becomes nominal rather than substantive – when the volume, velocity or complexity of agent actions exceeds the practical capacity of human reviewers to exercise meaningful judgement about them.
The oversight gap
Understanding why the oversight gap exists is the first step to closing it. There are three primary causes.
The first is scale. An AI agent executing a workflow – communicating with suppliers, reviewing contracts, managing customer escalations, analysing financial data – can take thousands of consequential actions in a timeframe that human reviewers cannot match. If oversight is designed as a review of all actions before they are executed, it either becomes a bottleneck that eliminates the productivity benefit of the agent or it becomes a rubber-stamp process that provides the illusion of oversight without the substance.
The second is expertise. As agentic systems are deployed in increasingly specialised domains – legal, financial, clinical, technical – the actions they take can require domain expertise that the designated human oversight function may not possess. A compliance team reviewing AI-generated legal analysis or an operations team reviewing AI-executed technical decisions, may lack the knowledge to identify errors that the agent has made confidently.
The third is anthropomorphism. The Berkeley paper identifies this as a specific risk: AI agents that communicate in natural language, use first-person pronouns and exhibit consistent personality traits can generate user trust and over-reliance that undermines critical scrutiny. People tend to over-trust systems that seem human and under-scrutinise outputs that arrive with apparent confidence.
Designing oversight that works
The NIST AI RMF addresses human oversight requirements across multiple subcategories, with Govern 2.1 requiring that roles, responsibilities and lines of communication for AI risk management are documented and clear and Map 3.5 requiring that human oversight processes are defined, assessed and documented. The Berkeley paper expands on these requirements with specific guidance for agentic systems.
The core design principle is that oversight should be risk-proportionate rather than uniform. Not every action an AI agent takes requires the same level of human review. A well-designed oversight framework distinguishes between actions that can be monitored automatically, actions that should trigger human review when certain conditions are met and actions that require human authorisation before execution.
The Berkeley paper, drawing on research by Kim et al., proposes a three-tier oversight model. The first tier covers routine agent actions that can be monitored by automated systems and flagged for exception review only when anomalies are detected. The second tier covers actions that are unusual, high-stakes or that exceed predefined parameters – these are automatically escalated to human reviewers with relevant expertise. The third tier covers the most critical issues, which may require escalation to a senior oversight committee or equivalent governance body.
This structure allows oversight to scale with the volume of agent activity without becoming either a bottleneck or a formality. The critical design question for each tier is the definition of escalation triggers – the specific conditions that cause an action or pattern of actions to be elevated to the next level of review. These triggers should be documented, tested and reviewed periodically as the system's operational context evolves.
Roles and responsibilities
ISO 42001 requires, under Clause 5 (leadership) and the supporting operational clauses, that roles and responsibilities for AI governance are clearly assigned and understood. For agentic AI, this needs to go beyond the standard designation of an AI governance lead. It requires defining who is responsible for setting the authority limits of each deployed agent, who monitors agent behaviour on an ongoing basis, who has the authority and capacity to intervene or shut down an agent that is behaving unexpectedly and who is accountable for decisions that an agent makes autonomously.
The Berkeley paper is explicit that agentic AI should be treated as a tool under human oversight, not as a peer or subordinate in the workforce. The language matters. When organisations begin referring to AI agents as AI workers or AI employees, they implicitly erode the accountability structures that effective governance requires. The agent does not bear responsibility. The organisation does.
Practical steps for executive teams
For executive teams seeking to strengthen their human oversight frameworks, the following priorities are worth immediate attention. First, conduct an audit of all agentic AI systems currently in operation, documenting the authority granted to each, the oversight mechanisms in place and the escalation procedures that exist. Second, assess whether current oversight mechanisms are substantive or nominal – whether reviewers have the time, expertise and capacity to exercise real judgement or whether oversight has effectively become a process that legitimises autonomous action. Third, establish a clear policy on minimum human control requirements for different categories of AI action, with higher autonomy requiring correspondingly more rigorous oversight design. Fourth, ensure that shutdown and intervention procedures are tested and that the personnel responsible for them are trained and ready.
Human oversight is not a constraint on the value of agentic AI. It is the condition under which that value can be safely realised.
Relevant frameworks: NIST AI RMF (Govern 2.1, Map 3.5, Manage 1.1) | ISO 42001 Clauses 5, 7, 8 | Berkeley Agentic AI Profile: Govern 2.1, Map 3.5, Manage 1.3 (Loss of Control)