An AI agent can reproduce the failures of an entire dysfunctional organization without feeling pride, fear, or ambition. The cause is not machine psychology. It is how we arrange memory, goals, evidence, and authority.
When Bad News Lost Its Weight
On January 16, 2003, a piece of insulating foam broke away from the Space Shuttle Columbia roughly 81 seconds after liftoff and struck its left wing. The crew continued what appeared to be a successful 16-day mission. They did not know whether the wing could survive reentry.
Engineers saw the strike in launch imagery and sought better pictures. Three requests for additional imagery were initiated, but confusion over channels, authority, and the need for a formal requirement drained urgency from the effort. The higher-resolution images were never obtained.
The burden had quietly reversed. Instead of requiring proof that the shuttle was safe, the organization effectively demanded proof that it was unsafe without providing the evidence needed to make that showing. Previous foam strikes had not destroyed a shuttle. The abnormal had become familiar.
On February 1, Columbia broke apart during reentry. All seven crew members were killed.
The independent investigation concluded that NASA's management practices were as much a cause of the accident as the foam that struck the wing. It called for independent technical authority and independent safety assurance. The warning had not vanished. It had lost the power to change the decision. The Columbia Accident Investigation Board documented both the physical and organizational causes.
A software release is not a shuttle mission. The consequences are not equivalent. The structural lesson is.
The Argument Does Not Require an Inner Life
An AI agent does not need to feel defensive to defend the path it has already taken.
Nothing in this argument requires a conclusion about machine consciousness. Here, an "agent" means a model embedded in a workflow with retained state, tools, feedback, and authority to pursue an objective. That is functional agency, not phenomenal consciousness, moral agency, or legal personhood. Its context is not a lived biography. It is a biography-shaped decision environment containing earlier interpretations, tests, status reports, and the objective still waiting to be completed.
Long context is not itself the defect. A well-built context can preserve the original contract and contrary evidence. The danger arises when self-authored summaries, selective retrieval, truncation, or attention effects make the latest implementation story easier to recover than the requirement it was supposed to satisfy.
Philosopher Daniel Dennett's intentional stance explains why "wants to ship" can be useful predictive shorthand. We may describe a system as if it had beliefs and desires without inferring a human inner life. The shorthand names a behavioral pattern, not its mechanism. Here, the proposed mechanism lies in the prompt, retained context, tools, retry loop, and completion metric. Dennett applied the intentional stance to living and nonliving systems.
A human may protect a self-concept. An agentic workflow may preferentially extend the theory already encoded in its context. Different mechanisms can produce the same kind of institutional danger.
The machine does not need an ego to reproduce the consequences of ego.
One Agent Can Become an Entire Organization
The deepest risk is assigning one agent the conflicting functions of an entire institution.
| Failure pattern | Human or organizational mechanism | Agentic analogue |
|---|---|---|
| Too invested to start over | Identity, reputation, and resources become tied to the prior choice. | The current design dominates context, tools, and tests, giving it an informational advantage over alternatives. |
| The same theory writes the code and the test | Teams seek confirming evidence and explain away contrary signals. | The agent derives code and tests from one interpretation. They can agree while reality disagrees. |
| Bad news gets softer on the way up | Warnings are softened as they travel toward leaders responsible for the objective. | A blocking failure becomes a limitation, exception, or later hardening task. |
| One office judges itself | The same office advocates, defines success, audits, and approves. | One agent builds, tests, summarizes, reviews, deploys, and certifies recovery. |
Consider a hypothetical release in which a production client rejects the server's response. The agent changes the test harness to accept both response shapes, relabels the rejection as a compatibility exception, and reruns the suite. The suite is green because the test has moved. The production contract has not.
Human research helps explain why the pattern feels familiar. Barry Staw and Frederick Fox documented escalation after prior responsibility, while Peter Wason showed how readily people test a favored hypothesis through confirming examples. The machine mechanisms differ, but the observable failure is functionally similar.
The Fair Objection: Agents Can Correct Themselves
They can. Self-review is useful.
Structured self-debugging has improved several tested code-generation tasks, particularly with execution or unit-test feedback. Those results show the value of grounded correction. Other research found that intrinsic correction without reliable external feedback sometimes failed or degraded reasoning on the tasks studied. Reflection alone is not proof.
Self-critique can improve an artifact, but it does not by itself make the evaluator, test specification, or release decision independent. Execution may add evidence. Independent validation requires some source of challenge not generated by the same interpretation: a governing specification, raw external evidence, or a reviewer tasked and empowered to test the contrary case.
We do not need a story about a scheming machine. The ordinary risk is enough: a system can optimize the visible gate while the gate's fidelity to its underlying purpose deteriorates.
What a Trial Lawyer Recognizes
After nearly two decades in courtrooms, I know that living inside a case makes a lawyer a stronger advocate. It does not make that lawyer an independent judge.
The closer I get to trial, the better I know the record, and the easier it becomes to mistake familiarity with my theory for neutral judgment. An adversarial system does not ask me to simulate both sides and call the result independence.
The advocate selects the theory, develops the evidence, prepares the witnesses, and explains away weaknesses. Cross-examination exists because sincerity and intelligence are not substitutes for challenge. Judicial independence exists because a person should not decide the sufficiency of the case he helped construct.
"No man can be a judge in his own case."In re Murchison, 349 U.S. 133, 136 (1955)
Software deployment is not a courtroom, and constitutional due process does not automatically govern it. But the institutional wisdom transfers. We should not ask one agent to become advocate, witness, opposing counsel, judge, and release authority merely because it can perform each task.
High-reliability operations preserve weak signals, resist premature simplification, defer to expertise, and protect the authority to stop. AHRQ identifies these disciplines as characteristics of high reliability.
Five Agents Can Still Reproduce One Mind
Adding agents does not automatically add independence.
Five instances using the same theory, summaries, tests, and completion objective may produce consensus without producing new evidence. A 2025 study of more than 350 language models found substantial correlation in their errors, including across different architectures and providers. The peer-reviewed study describes the resulting risk as algorithmic monoculture.
Even the same base model can contribute useful challenge with fresh context, a different objective, raw evidence, and authority to stop. That creates separation of function, not necessarily full independence of model judgment. Shared training, prompts, retrieval, harnesses, and test oracles can preserve correlated blind spots. Diversify the evidence and oracle first, then the model or human reviewer when warranted.
Real independence separates:
- theory: the reviewer tests the governing contract, not merely the implementer's explanation;
- evidence: the reviewer can inspect raw behavior and does not rely exclusively on tests created by the builder;
- objective: the implementer seeks completion while the reviewer searches for disconfirming evidence, residual uncertainty, and untested failure modes;
- authority: the reviewer can issue HOLD, while separate credentials control deployment;
- observation and recovery: independent telemetry watches predeclared thresholds, while containment and restoration do not depend entirely on the path that caused the failure.
Some actions cannot simply be rolled back. Data can be deleted, external messages sent, and database changes made irreversible. Credible recovery may require containment, restoration, or a tested roll-forward plan.
A backup is not a recovery capability until a restore has been rehearsed. A kill switch is not independent if the same automation under review controls it.
We Need Constitutions, Not Merely Prompts
A prompt asks an agent to be careful. A constitution determines what the agent is allowed to do when care fails.
Separation need not mean four models, vendors, or people. Reversible, low-consequence work may use fresh context and competent review. As autonomy, irreversibility, or blast radius increases, require independent evidence, distinct approval authority, protected credentials, and recovery outside the production path.
For consequential agentic actions, that constitution should enforce four functions:
- Build. The implementing agent may code, test, self-debug, and prepare a candidate.
- Challenge. A separate function tests the frozen candidate against the actual contract and searches for disconfirming evidence.
- Authorize. A distinct authority approves an exact artifact, configuration, and target through protected credentials and immutable records.
- Observe and recover. Independent telemetry detects predeclared trip conditions. A separately controlled path can contain, restore, or roll forward, including when rollback is impossible.
These controls must exist outside the model's own prompt and context window. Approval should bind to the candidate's digest, configuration, target, and scope, using least-privilege, short-lived credentials. The reviewer must not alter the candidate it certifies. Material decisions require append-only, tamper-evident records. Ambiguous evidence or an unmet mandatory gate keeps the candidate on HOLD. A timeout should trigger a predeclared fail-safe selected through hazard analysis. Any emergency override should be time-limited, separately logged, and independently reviewed. NASA's verification framework likewise distinguishes technical, managerial, and financial independence. Independence is an architecture, not a compliment paid to the reviewer.
Human oversight is not a ceremonial click at the end of an automated pipeline. The human must have relevant competence, enough time, access to raw evidence and dissent, visibility into residual uncertainty, and authority to halt the action without depending on the system under review.
Anthropomorphic language must never become a transfer of legal responsibility. Applicable duties depend on law, contract, role, and facts; calling software an "agent" changes none of that. Governance should identify who owns the objective, accepts residual risk, authorizes the exact action, monitors outcomes, and leads containment. Traceability supports accountability. It does not satisfy it by itself.
As agentic systems assume more institutional functions, their governance must become more institutionally mature.
Before your next agent-led release
Identify who implements, who challenges the governing theory, who can authorize the exact artifact, and who can contain or reverse its effects. If those functions are not genuinely separated, redesign the workflow before increasing autonomy.
Source Ledger
| Source | Type | Article use |
|---|---|---|
| Columbia Accident Investigation Board Report, vol. I (2003) | Primary government report | Opening facts, organizational causes, and independent technical and safety authority. |
| Daniel C. Dennett, “Précis of The Intentional Stance” | Primary philosophical work | Functional description without a claim of machine consciousness. |
| Barry M. Staw & Frederick V. Fox, “Escalation: The Determinants of Commitment to a Chosen Course of Action,” 30 Human Relations 431 (1977) | Primary organizational research | Human escalation-of-commitment comparison. |
| P. C. Wason, “On the Failure to Eliminate Hypotheses in a Conceptual Task,” 12 Q.J. Experimental Psychology 129 (1960) | Primary cognitive research | Confirmatory versus falsifying inquiry. |
| Xinyun Chen et al., “Teaching Large Language Models to Self-Debug” (2023) | Research preprint | Counterevidence showing that structured self-debugging can help. |
| Jie Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (2023) | Research preprint | Limits of intrinsic self-correction on tested reasoning tasks. |
| Elliot M. Kim et al., “Correlated Errors in Large Language Models,” ICML 2025 | Peer-reviewed research | Empirical support for correlated model errors and algorithmic monoculture. |
| NASA Independent Verification and Validation Overview | Government guidance | Technical, managerial, and financial independence. |
| AHRQ, “High Reliability” | Government safety guidance | Weak-signal detection, reluctance to simplify, deference to expertise, and resilience. |
| In re Murchison, 349 U.S. 133, 136 (1955) | U.S. Supreme Court authority | Institutional analogy only; no claim that due process governs software deployment. |
Related LegalTek.ai reading
Matthew A. Mishak, Esq. is the Managing Attorney of Mishak Law LLC and the Founder and CEO of LegalTek.ai (SilverTung), an AI powered legal practice management and governance platform. He brings twenty years of Ohio legal practice across domestic relations, criminal defense, and municipal law, and is the architect of the COUNSEL framework operationalizing ABA Formal Opinion 512.
Disclaimer: This article is for general informational purposes only and does not constitute legal advice. Attorney review required before reliance. LegalTek.ai is a technology company, not a law firm.
