Why AI Can't Always Explain Itself — and Why That Matters in Regulated Work

An AI system flags a claim for denial. Or surfaces a case as high-risk. Or nudges toward one diagnosis over another. The recommendation might be exactly right — but sooner or later, someone with authority asks the question that matters: why? And in a surprising number of AI tools, the honest answer is, "we can't fully tell you."

In casual settings, that's a shrug. In law, medicine, finance, and government, it's a wall. When you have to justify a decision to a regulator, a court, a patient, or a board, an answer you can't explain is an answer you can't use — even when it happens to be correct.

What "Black Box" Actually Means

Modern AI doesn't reach an answer by following a rulebook you can open and read. It works through millions of internal numerical weights, tuned over vast amounts of data, that add up to a result no one wrote down in plain language. The system can be reliably right without being able to show its work the way a person would. This isn't the vendor hiding something — much of the time, the reasoning is genuinely opaque by construction.

That's the uncomfortable truth beneath the marketing: raw capability and clear accountability are two different things, and buying the first does not automatically get you the second.

Fine Sometimes, Disqualifying Others

Not every task needs a paper trail. When AI drafts an email or summarizes a call, nobody's going to demand the reasoning, and asking for it would be silly. The stakes decide. The moment the output feeds a decision you'll have to stand behind — one that affects someone's money, health, freedom, or rights — "the AI said so" stops being an acceptable answer, no matter how good the AI is.

So explainability isn't a nice-to-have you bolt on later. In regulated work it's a gating requirement: if you can't trace how a conclusion was reached, you can't defend it, and if you can't defend it, you can't use it.

An Explanation Isn't the Same as Explainability

Here's the trap. Ask some AI tools to explain themselves and they'll happily produce a fluent, confident paragraph describing their "reasoning." It sounds like an answer. But often that story is generated after the fact to satisfy you — a plausible narrative, not the actual basis for the decision. It can even be wrong about the very system that produced it.

Real explainability is different and more boring: the genuine sources an answer was drawn from, the factors that actually drove it, and an auditable trail you could hand to someone who needs to check it. A confident rationale you can't verify isn't transparency. It's a black box wearing a nicer coat.

The One Question to Ask

To tell traceability from theater, ask the vendor to prove the basis, not describe it:

A vendor who built for accountability can show you: here are the documents this answer drew from, here's what drove it, here's the record you could put in front of an auditor. A vendor who can only offer a generated rationale — or who assures you the model is "very accurate" and changes the subject — has told you their tool can't be traced, which in regulated work means it can't be trusted with the decisions that count.

Designing for Traceability

The AI worth using in serious work is built so its answers can be traced back to real, checkable ground — drawn from your own sources, with the basis surfaced rather than hidden, and a trail you could defend. That's the standard we build to at EMDELLE, and part of what BAG, our behavioral engine, is designed to support. (One mention, as always — this post is here to be useful, not to pitch.)

So the next time an AI gives you an impressive answer, ask the unglamorous follow-up: can you show me why — for real? In the work where being wrong has consequences, an answer you can trace beats a smarter one you can't, every time.

See you next Friday — one more thing to learn.

— Tia Lake, Founder & First Steward