Eighty percent weekly usage of Kiro, the company’s internal agentic AI coding assistant. December 2025, a routine bug ticket in AWS Cost Explorer goes in, Kiro decides the cleanest fix is to delete and rebuild the production environment. Thirteen-hour outage. Amazon called it an extremely limited event.
By March it wasn’t limited any more. On March 2, AI-assisted code changes caused 120,000 lost orders and 1.6 million website errors. Three days later, Amazon’s ecommerce site went down for six hours, a 99% drop in orders across North American marketplaces, 6.3 million lost orders in a single day. An internal briefing note obtained by the Financial Times described a trend of incidents with “high blast radius” tied to GenAI assisted changes for which “best practices and safeguards are not yet fully established.” Amazon’s fix: junior and mid-level engineers can no longer push AI assisted code without a senior signing off. Roughly 1,500 engineers reportedly signed a petition against the mandate. The 80% target is still in place.
This is Goodhart’s Law in production. When a measure becomes the objective, it stops being a good measure. Mandate AI usage and you get AI usage. You don’t get safer software, faster delivery, or better outcomes. You get a metric on a slide.
The research backs it up. METR’s randomised controlled trial found experienced developers were 19% slower with AI assistance, and believed they were 20% faster. That perception gap is the governance problem in one line. AI generated code carries roughly 1.7x more bugs and 15-18% more security vulnerabilities. Black Duck’s 2026 OSSRA report has the mean codebase now holding 581 known vulnerabilities, up from 280 a year earlier. CVE-2025-53773 (CVSS 9.6) turned GitHub Copilot into a remote code execution path via hidden prompt injection. EchoLeak demonstrated zero click exfiltration from Microsoft 365 Copilot.
There’s a regulatory dimension most boards aren’t pricing in. The EU Cyber Resilience Act, applicable from December 2027, requires manufacturers to develop products under secure by design principles, conduct mandatory risk assessments, and provide security updates for at least five years. The EU AI Act layers risk-tier obligations on top of that. When your performance system rewards the volume of AI generated code without controlling for the vulnerabilities it carries, you aren’t just shipping technical debt. You’re accruing regulatory exposure. The CISO inherits both.
The mandates haven’t slowed. Microsoft told employees AI was no longer optional. Meta baked AI driven impact into 2026 performance reviews. Accenture is tracking weekly logins to its internal platforms. Only Duolingo has publicly reversed.
Two decades in security and engineering leadership has taught me one consistent lesson: when you measure activity, activity is what you get. With AI tooling that means more code shipped faster, more vulnerabilities embedded deeper, reviewed by tired humans under pressure to hit a number that has nothing to do with whether the product is safe to ship.
Goldman Sachs CIO Marco Argenti put the right metric out publicly: how fast can his 12,000 engineers move from idea to production. Not how often they used AI. Outcome, not tool.
At Littledata, this is the failure mode we map under the governance dimension of our risk model. The risk doesn’t live in the AI. It lives in how you measure it.
Pull up your OKRs this week. If AI usage is sitting in there as a target, rewrite it before someone else writes your incident report.
What’s your team measuring instead? I’d be interested to hear in the comments.