"This business will get out of control… and we'll be lucky to live through it."Admiral Painter — The Hunt for Red October
It happened on a Tuesday. The day doesn't matter — it happens every day.
A junior data scientist on your team needed an agent framework. Nothing exotic. A GitHub repo with enough stars to look credible, a README that covered the basics. They downloaded it. Ran it locally. Started building.
Security didn't know. You didn't know. Nobody asked — because nobody knew to ask. The library touched the file system, read environment variables, reached the .env file your engineer has been meaning to move to a vault for three months. It made outbound calls. Something got logged somewhere.
Nothing broke. The experiment continued.
That Tuesday is happening in your organization right now — probably multiple times this week, across teams, across disciplines. A VS Code extension with read access to your source code installs quietly because someone in a team thread recommended it. An npm package pulled into a demo becomes a prototype becomes a proof-of-concept that gets pushed to staging before anyone runs a scan. A Docker image from a maintainer you've never audited runs on infrastructure adjacent to customer data.
Each decision is local. Each decision is reasonable. Your engineers are being pragmatic. Your data scientists are solving real problems under real timelines. Your security team is doing what they can with the bandwidth they have.
And collectively, the organization is accumulating risk that has no mechanism to surface. The system has no instrument to detect what's compounding in the gaps between locally reasonable decisions. Nobody is failing. The architecture is.
The Hunt Revealed
In The Hunt for Red October, Marko Ramius commands a Soviet nuclear submarine toward American waters with intentions no one can read clearly. The Americans assume attack. The Soviets suspect defection — or possibly a rogue commander. The British think they've identified a pattern. Everyone is processing incomplete information and calling it situational awareness.
Each is working from the best information available to them. Each is making defensible decisions. Their vantage points don't overlap anywhere — and the risk lives in that gap.
This is your AI adoption program, described precisely.
Each circle: one actor’s vantage point. No shared center.
How their tool gets integrated downstream. What it gets connected to. What ambient access it accumulates when an engineer grants permissions to make a deadline.
The twelve integrations that happened before anyone thought to loop them in. Their risk model is calibrated to the surface they can observe.
The aggregate. The compounding surface. The other eleven decisions that happened in parallel.
Downstream consumption patterns. Who runs their code in production, at what scale, connected to what systems, under what security posture.
The gap between controlled capability demonstration and actual behavior in a production environment stitched together from twelve different integrations nobody fully mapped.
Everything listed above. The full board.
The Mess We're In
When a bank gets breached, regulators arrive. Penalties are specific. Causality is traceable to a system, a configuration, a decision chain. The feedback loop fires immediately and expensively. Financial institutions have developed sophisticated security postures — not because their teams are uniquely capable, but because the consequences of failure are legible, proximate, and attributable. The system learned because it had a mechanism to learn.
When power infrastructure cascades, everyone sees it. Grid goes down. Investigation begins. Root cause gets published. The incentive to prevent recurrence concentrates rapidly because the failure is impossible to misattribute or defer.
AI adoption doesn't produce this dynamic.
Attackers didn't compromise OpenClaw's core infrastructure. They poisoned the distribution mechanism: seeding the marketplace with hundreds, then over a thousand, fake skills designed to look legitimate. Engineers installed them. The skills installed trojans, credential stealers, keyloggers.
The attack surface wasn't a vulnerability in a known system. It was the reasonable local decision: I need a skill, the marketplace has skills, this one looks credible. That's the same decision your junior data scientist made on Tuesday.
The causality will take months to fully trace. How many credential stores were compromised? Which systems did those credentials reach? What was exfiltrated before disclosure? These questions may never have clean answers.
When causality is invisible, the feedback loop doesn't fire.
ClawHavoc isn't an outlier. It's a named instance of a pattern that runs continuously, mostly unnamed. A VS Code extension exfiltrates prompts selectively, in traffic that looks like normal telemetry until someone with the right instrumentation looks closely — which requires knowing to look. A library backdoor sits in a production dependency tree for months before a researcher identifies the insertion point. A vendor misconfiguration exposes inference context to queries it shouldn't answer; the blast radius is unclear because the access logging was incomplete.
Each follows the same structure: reasonable local decision, invisible aggregate exposure, causality that arrives too late or not at all to activate correction.
When causality is invisible, incentives don't correct. When incentives don't correct, organizations continue the behavior that created the exposure. The aggregate risk grows without any actor's model updating to reflect it.
Every actor is aware. None of them has a feedback loop that fires.
The mechanism that would force the system to correct from observed failures is structurally broken — because the failures are often invisible, the causality often untraceable, the cost often deferred or diffuse enough that it never lands as a legible signal.
In the absence of legible consequences, process substitutes for outcome. Organizations develop sophisticated documentation of risk management programs. Reviews get scheduled. Policies get written. Steering committees meet. And the junior data scientist downloads the framework on Tuesday because the process didn't reach that far, and nobody's model of the risk surface included that moment.
The people are earnest. "Responsible AI adoption" becomes theater anyway — the system has no instrument to distinguish between organizations genuinely managing exposure and organizations with well-maintained documentation of the intent to do so. The feedback loop that would create that distinction isn't firing.
What Changes It
You already know the standard answers — threat modeling, zero-trust architectures, earlier security involvement, better vendor checklists. Everyone reading this has tried at least one. Everyone knows where they break down under competitive pressure to move faster.
The harder question: what would activate the feedback loop?
What would make the cost of asymmetric information legible to the actors currently operating without it?
Not documentation of process — actual transparency. Real visibility into what AI systems access, transmit, and log. Is that technically achievable at the integration points that matter? Would vendors accept the exposure? Does the liability structure of the current ecosystem reward that kind of disclosure or punish it?
If a supply-chain vulnerability produces a breach, and the breach traces to a library, which traces to a maintainer, which traces to a contributor — who carries the cost? Right now the answer is diffuse enough that the incentive to prevent it doesn't concentrate on any actor with the power to change the system. What would liability concentration do to the incentive structure? What would it do to open-source sustainability?
Not certifications that signal process compliance. Standards that produce measurable, auditable outcomes — where the audit is of actual security posture, not documentation of intent. Who defines them with enough technical specificity to be meaningful? Who enforces them with enough authority to matter? Is the industry capable of producing this before regulatory intervention forces it?
For the moment aviation waited for, that financial regulation waited for — a failure visible and large enough that the feedback loop becomes impossible to ignore, and the question shifts from whether to act to how fast?