Opacity and over-reliance

The Decision Nobody Can Explain

Ask why a loan was declined, and a traditional system gives you the rule it applied. Ask why a model declined it, and the honest answer is often that nobody knows. Not that nobody will say. That nobody knows. When we talk about opacity, we aren't talking about a company being secretive or concealing their code. It's structural. In traditional computing, we can find the specific rule. In an AI model, the decision path is often unreadable even to the people who built it.

The Recipe vs. The Chef's Palate

Imagine traditional software as a recipe. It can be read back, step by step, and audited. If the dish is too salty, you can see exactly where it called for too much salt. AI decisions are more like a chef's palate. An experienced chef tastes a sauce and says it 'needs something.' They're usually right, but they can't fully explain the biological chemistry. If you're the manager signing off on the chef's work but you can't taste the sauce yourself, you'll end up approving almost everything. In AI, we're signing off on palates we cannot taste.

Structural Opacity

It's critical to distinguish between 'will not' and 'cannot.' We're used to companies being secretive about proprietary code. But with AI, the opacity is structural. The decision path is unreadable even to the developers. When a system tries to explain itself, it's often just a guess made after the fact. This reconstruction might not reflect what actually drove the output. You're looking at a plausible story, not the actual mechanics.

Takeaways: Understanding Opacity

Before we move to how this impacts human behavior, let's recap. Opacity isn't necessarily a sign of bad intentions; it's a structural reality of the 'black box.' Because we can't see the reasoning, we can't easily challenge the result.

Over-reliance and Automation Bias

Opacity is the primary reason over-reliance happens. If you can't see the reasoning, you can't meaningfully challenge it. This leads to automation bias—our natural tendency to accept a machine's answer over our own judgment. It's important to realize that expertise does not protect you. Under high volume and time pressure, even senior professionals start deferring to the system. It's a rational response to a system that's right most of the time—until it isn't.

Human in the Process vs. Control

Most firms claim they have a 'human in the loop.' But there's a critical distinction. A human 'in the process' is just a person hitting 'approve' on a screen. They are a rubber stamp because they lack the time or information to disagree. A human 'in control' is different. This is someone who genuinely evaluates the decision, has the authority to say no, and has the incentive to use that authority.

Real-World Incident: Medicare Advantage Allegations

A powerful example of this decay in oversight can be seen in the UnitedHealthcare nH Predict litigation. It is important to state clearly these are allegations in live proceedings and are unproven (STAT News, 2023). Families allege the insurer used an algorithm to cut post-acute care. Only about 0.2% of patients appealed a denial. However, for those who did appeal, they succeeded roughly nine times out of ten. The core takeaway isn't just the error rate; it's the silence. The output was almost never challenged by the humans in the process.

Takeaways: Over-reliance and Control

We've seen how opacity disarms the reviewer. Expertise doesn't protect us from automation bias, especially when volume pressure turns a 'human in the loop' into a human in the process.

The Oversight Checklist

When you're looking at a business process, use this checklist to see if 'human oversight' is actually a rubber stamp. Click each category to see what to look for. First, Information. Can the reviewer see why the system made this specific choice? Without the 'why,' they can't disagree. Next, Time. Does the reviewer have enough time per case to actually think, or are they measured solely on speed? Authority is key. Is the reviewer allowed to say 'no' without filing a mountain of paperwork or facing pushback? Finally, Incentive. Is the reviewer rewarded for finding errors, or just for keeping the queue moving?

The Explicit Stop

Prising these systems open, and designing oversight that actually works, are both real disciplines and both belong to the next course. Carry two questions out of here instead: can anyone show why, and has the reviewer ever said no?