Can You Defend That Decision in a Courtroom?

Most organizations treat AI explainability as a policy question. They write a responsible AI policy. They describe their commitment to transparency. They ensure their AI systems are, in principle, explainable.

None of that helps when a customer, regulator, or attorney asks why this person, in this situation, received this outcome.

Explainability isn't a policy. It's an operational capability. And most organizations haven't built it.

I've written about the three questions every board should be asking about AI governance. This piece goes deeper on the second of them: whether the organization can explain and demonstrate a specific decision, not just describe how the system works in general.

The Gap Most Organizations Don't See

There are two kinds of AI transparency. Most organizations are reasonably good at one of them.

The first is system transparency: the ability to describe how the AI works in general terms. Here's the model. Here's the training data. Here's the intended use case. Here's our responsible AI policy. This is what most organizations mean when they say their AI is explainable.

The second is decision transparency: the ability to explain a specific decision to a specific person. Why was this applicant declined? Why did this customer receive this price? Why was this claim flagged? Why was this candidate not selected? Not how the system works. Why this person, this outcome, this moment.

Cigna ran into this in 2023. A ProPublica investigation found that over two months in 2022, the company's medical directors denied more than 300,000 claims using an automated flagging system, spending an average of 1.2 seconds on each one, signing off in batches without reviewing individual patient files. Cigna could describe how the system worked: it flagged mismatches between diagnoses and acceptable procedures. What it couldn't produce for any specific patient was a clinical judgment that accounted for their individual circumstances. Congressional investigations followed. The case is still in court.

System transparency is valuable. Decision transparency is what you need when someone with standing challenges an outcome. Most organizations have built the first and assumed it covers the second. It doesn't.

What Defensibility Actually Requires

Defensibility isn't a legal strategy. It's an operational standard. It means the organization can answer for specific decisions, at the moment they're challenged, with documentation that was created when the decision was made. Not assembled afterward. Not reconstructed for a courtroom. Created at the time.

Four things have to exist for that to be true.

Decision logs. Every consequential AI decision needs to be recorded: what inputs were used, what the AI output was, and when it happened. Not a summary. Not an aggregate. The specific record for this decision.

Preserved decision logic. The model that made a specific decision needs to be preserved, not just the current version. Models are updated. The model running today may not be the model that made the decision being challenged. If the organization can't produce the logic that applied at the moment of the decision, it can't explain the decision.

The explanation chain. There needs to be a traceable path from input to outcome for any given decision. Not a description of how the model works in general, but a specific account of what factors applied, what weight they carried, and what the model determined. For some model types this is straightforward. For others it requires deliberate design choices made before deployment.

Retention policy. The documentation needs to exist long enough to be useful. Regulatory challenge timelines and litigation timelines don't align with standard data retention defaults. The organization needs a deliberate policy on how long decision records are kept, not a default that deletes them before they're needed.

The Retrofit Problem

The most common response when a challenge arrives is to try to reconstruct the documentation. It almost never works.

The model may have been updated since the decision was made. The inputs may not have been logged. The audit trail was never built. The organization can describe how similar decisions are made today, but not how this decision was made then. The response to the challenge becomes a general description of the system rather than a specific account of the decision.

That's the position Cigna was already in. When a specific patient challenged a denial, there was no individual analysis to reconstruct. The documentation didn't exist because the individual decision never happened. The system had logic. The decision didn't. It's the gap that turns a defensible outcome into an indefensible one, not because the decision was wrong but because the organization can't show its work.

Documentation retrofitted after a challenge is almost always too late, too incomplete, or too obviously constructed for the purpose of the challenge to carry weight. Defensibility has to be built in at deployment. It can't be added afterward.

The Board's Question Is Specific

Most boards ask the wrong version of this question. "Is our AI explainable?" almost always gets a yes. Every organization with a responsible AI policy will say yes.

The right question is: can you show me the documentation for a specific decision this AI made?

Pick a decision. A credit application declined last quarter. A hiring decision from six months ago. A claim flagged as potentially fraudulent last year. Ask to see the decision record. Ask what inputs were used. Ask what the model determined and why. Ask how long that record will be retained.

If the answer requires assembling information from multiple systems, or involves rebuilding what the model would have considered, or results in a general description of the process rather than a specific account of the decision, the organization doesn't have operational defensibility. It has the aspiration of it.

The board's question isn't whether the organization has a transparency policy. It's whether they can demonstrate a specific decision to someone with standing to challenge it. The difference between those two things is where liability lives.

What This Looks Like Built Right

Organizations that have built this well don't think of it as a legal preparation. They think of it as operational hygiene. Documentation is built into deployment. The infrastructure for explaining a decision is designed before the model goes live, not assembled after the challenge arrives.

The test isn't whether the organization can pass a responsible AI audit. It's whether the person accountable for a specific decision can walk into a room with a customer, a regulator, or a lawyer and show their work. Not the system's work. The decision's work.

That's a different standard. Most organizations haven't set it yet. The board's job is to ask whether they have.