Every Fortune 500 CEO investing in AI right now faces the same brutal math. They’re spending $590-$1,400 per employee annually on AI tools while 95% of their corporate AI initiatives fail to reach production.
Meanwhile, employees using personal AI tools succeed at a 40% rate.
The disconnect isn’t technological—it’s operational. Companies are struggling with a crisis in AI measurement.
Three questions I invite every leadership team to answer when they ask about ROI from AI pilots:
- How much are you spending on AI tools companywide?
- What business problems are you solving with AI?
- Who gets fired if your AI strategy fails to deliver results?
That last question usually creates uncomfortable silence.
As the CEO of Lanai, an edge-based AI detection platform, I’ve deployed our AI Observability Agent across Fortune 500 companies for CISOs and CIOs who want to observe and understand what AI is doing at their companies.
What we’ve found is that many are surprised and unaware of everything from employee productivity to serious risks. At one major insurance company, for instance, the leadership team was confident they had “locked everything down” with an approved vendor list and security reviews. Instead, in just four days, we found 27 unauthorized AI tools running across their organization.
The more revealing discovery: One “unauthorized” tool was actually a Salesforce Einstein workflow. It was allowing the sales team to exceed its goals — but it also violated state insurance regulations. The team was creating lookalike models with customer ZIP codes, driving productivity and risk simultaneously.
This is the paradox for companies seeking to tap AI’s full potential: You can’t measure what you can’t see. And you can’t guide a strategy (or operate without risk) when you don’t know what your employees are doing.
‘Governance theater’
The way we’re measuring AI is holding companies back.
Right now, most enterprises measure AI adoption the same way they do software deployment. They track licenses purchased, trainings completed, and applications accessed.
That’s the wrong way to think about it. AI is workflow augmentation. The performance impact lives in interaction patterns between humans and AI, not solely on tool selection.
The way we currently do it can create systematic failure. Companies establish approved vendor lists that become obsolete before employees finish compliance training. Traditional network monitoring misses embedded AI in approved applications such as Microsoft Copilot, Adobe Firefly Slack AI and the aforementioned Salesforce Einstein. Security teams implement policies they cannot enforce, because 78% of enterprises use AI, while only 27% govern it.
This creates what I call the “governance theater” problem: AI initiatives that look successful on executive dashboards often deliver zero business value. Meanwhile, the AI usage that is driving real productivity gains remains completely invisible to leadership (and creates risk).
Shadow AI as systematic innovation
Risk doesn’t equal rebellion. Employees are trying to solve problems.
Analyzing millions of AI interactions through our edge-based detection models proved what most operating leaders instinctively know, but cannot prove. What appears to be rule-breaking is often employees simply doing their work in ways that that traditional measurement systems cannot detect.
Employees use unauthorized AI tools because they’re eager to succeed and because sanctioned enterprise tools succeed in production only 5% of the time, while consumer tools like ChatGPT reach production 40% of the time. The “shadow” economy is more efficient than the official one. In some cases, employees may not even know they’re going rogue.
A technology company preparing for an IPO showed “ChatGPT – Approved” on security dashboards, but missed an analyst using personal ChatGPT Plus to analyze confidential revenue projections under deadline pressure. Our prompt-level visibility revealed SEC violation risks that network monitoring completely missed.
A healthcare system recognized doctors using Epic’s clinical decision support, but missed emergency physicians entering patient symptoms into embedded AI to accelerate diagnoses. While improving patient throughput, this violated HIPAA by using AI models not covered under business associate agreements.
The measurement transformation
Companies crossing the “GenAI divide” identified by MIT, whose Project Nanda identified the remarkable struggles with AI adoption, aren’t those with the biggest AI budgets; they’re those who can see, secure, and scale what actually works. Instead of asking, “Are employees following our AI policy?” they ask, “Which AI workflows drive results, and how do we make them compliant?”
Traditional metrics focus on deployment: tools purchased, users trained, policies created. Effective measurement focuses on workflow outcomes: Which interactions drive productivity? Which creates genuine risk? Which patterns should we standardize organization-wide?
The insurance company that discovered 27 unauthorized tools figured this out.
Instead of shutting down ZIP code workflows driving sales performance, they built compliant data paths preserving productivity gains. Sales performance stayed high, regulatory risk disappeared, and they scaled the secured workflow companywide—turning compliance violation into competitive advantage worth millions.
The bottom line
Companies spending hundreds of millions on AI transformation while remaining blind to 89% of actual usage face compounding strategic disadvantages. They fund failed pilots while their best innovations happen invisibly, unmeasured and ungoverned.
Leading organizations now treat AI like the biggest workforce decision they’ll make. They require clear business cases, ROI projections, and success metrics for every AI investment. They establish named ownership where performance metrics include AI results tied to executive compensation.
The $8.1 billion enterprise AI market won’t deliver productivity gains through traditional software rollouts. It requires workflow-level visibility distinguishing innovation from violation.
Companies establishing workflow-based performance measurement will capture productivity gains their employees already generate. Those sticking with application-based metrics will continue funding failed pilots while competitors exploit their blind spots.
The question isn’t whether to measure shadow AI—it’s whether measurement systems are sophisticated enough to turn invisible workforce productivity into sustainable competitive advantage. For most enterprises, the answer reveals an urgent strategic gap.
The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune.
