The Accountability Gap: Why ‘Human in the Loop’ Is Often Just Theater

Last week, I found myself in Washington, D.C., speaking at the Digital Economist’s day-long event, We The People: Reclaiming Accountability in the Age of Intelligent Systems, held alongside the World Bank and IMF spring meetings. The city was buzzing with energy - central bankers, economists, policy architects,  finance ministers, and half of Congress moving through the same hallways, all grappling with how technology is reshaping power and accountability.

I was asked to address one of the thorniest questions in that space right now: How do we ensure that human oversight of AI stays real, not a formality or a performance for regulators, but genuinely active?

It’s a question I think about constantly in my coaching work with executives and senior leaders, many of whom are integrating AI. And the more I sit with it, the more convinced I am that most organizations are getting this wrong, because of the gap between what oversight looks like on paper and what it requires in practice.

What follows is what I shared in that room, and I think it’s some of the most important thinking I’ve done on AI and leadership.

The Phrase That’s Doing a Lot of Work

“Human in the loop” has become one of the most powerful phrases in AI governance, and one of the most misleading. The phrase implies a safeguard, someone reviewing, overseeing, and capable of catching what the algorithm misses. But when you examine what “human in the loop” looks like in practice, the picture is far less reassuring.

Someone technically signs off. There is definitely a box that gets checked and a log that shows the review happened. And yet, the person reviewing often lacks the technical literacy to interrogate the model’s outputs, the organizational standing to challenge the outcome, and any real consequence if something goes wrong.

When that’s the situation, you have theater, not oversight. 

In any other domain, whether financial controls, IT, or workplace safety, “I didn’t fully understand it” is not an acceptable defense. AI deserves the same standard, and yet we keep accepting a lower bar.

Why This Matters More Than Most Leaders Realize

Consider what happened with Amazon’s AI recruiting tool that systematically downgraded resumes from women. The mechanism that should have caught it upstream, a human reviewer with the knowledge, authority, and accountability to raise a red flag, was not meaningfully in place. That was a leadership accountability failure, not a technical one.

The pattern mirrors where cybersecurity was a decade ago. Executives said “that’s an IT problem” right up until a breach made it a board problem, a reputational problem, and sometimes a personal one. We are in the same window with algorithmic accountability, and the clock is running.

The EU AI Act, which began phased enforcement in 2024, explicitly requires human oversight mechanisms for high-risk AI applications. At this point, every company realizes that oversight is required, but is the oversight you have in place real?  Regulators increasingly know the difference.

Three Things Genuine Oversight Requires

In my DC panel remarks, I argued that active AI oversight requires three things most organizations are not building deliberately. Every senior leader can and should be driving all three.

1. Literacy

Not deep technical expertise, but enough understanding to ask real questions. What data trained this model? What does it optimize for? Where have we seen it fail? Intellectual humility is the prerequisite. Leaders who model “I don’t fully understand this, walk me through it” create cultures where inconvenient questions get asked before they become crises.

Amy Edmondson’s research on psychological safety confirms that teams in high-safety cultures surface problems earlier and adapt more effectively under pressure, and that dynamic applies directly to how we oversee AI.

2. Standing

In most organizations, there is no named person with the authority to pause an AI-driven process when something doesn’t look right. Oversight responsibilities are spread across committees and working groups, which means, functionally, they belong to no one.

The complexity of a multi-agent or supply chain AI system is not an excuse for diffuse responsibility. If anything, it’s a reason to be more precise about who owns what.

3. Consequence

Until there is a person whose professional reputation is on the line for the quality of algorithmic decisions, the same way a CFO’s reputation is on the line for the numbers, you don’t have oversight. You have documentation.

Real accountability follows the decision, not just the process. In my work with executives and boards, I see this pattern consistently: governance frameworks without consequence are governance frameworks without teeth.

On the “Speed vs. Oversight” False Dichotomy

One of the most common objections I hear from leaders is that meaningful AI oversight is too slow, that accountability mechanisms are a brake on innovation. It’s worth noting that the fastest cars have the best brakes. 

I want to push back on that framing: trust is a competitive advantage. Organizations that build traceable, accountable AI systems tend to move faster in regulated industries, attract stronger talent, and face fewer crises. Accountability architecture is infrastructure for speed.

As I’ve written about in the context of board communication and leading through turbulent periods, the leaders who earn lasting institutional confidence are those who communicate risk clearly and early, before it escalates. The same principle applies here. The cost of proactive oversight is always lower than the cost of a preventable failure.

Next Step: Name Someone

The most powerful thing I said in that Washington, D.C. room was also the simplest. When asked what a leader could do on to move from passive to active oversight, my answer was:

“Name someone. Not a committee, not a working group, but a specific person whose job it is to ask inconvenient questions about how your AI systems are making decisions, and who has the standing to pause a process when something doesn’t look right.”

That one act does three things immediately. 

1. It signals to the organization that AI accountability is serious. 

2. It creates a channel for people to surface concerns before they become crises.

3. It forces a necessary conversation about what “reviewing” an AI output means in practice.

   
   

Naming this one person costs nothing and requires one meeting. Most organizations already have the governance language, they’re just missing a named human with actual authority.

From Theater to Accountability

The event in Washington was called We The People: Reclaiming Accountability in the Age of Intelligent Systems. That word - reclaiming - matters. Accountability doesn’t disappear because an algorithm is involved. It gets reassigned, obscured, and diluted. Senior leaders are uniquely positioned to reassign it back.