AI Demonstrates Hidden Strengths and Weaknesses in Business Decision-Making

Imagine a barista facing a rush of customers, each with complex orders and tricky requests. Now imagine that same worker, guided by an AI, navigating the chaos—yet only some finish their shift with a smile and a tip. In the world of business AI, the difference between a good assistant and a reliable partner isn’t just in how well they chat; it’s whether they can deliver results under pressure.

Testing AI’s Business Backbone, Not Just Its Words

At Firmulate, a live experiment puts AI models through a rigorous test—running a real software company facing its worst week. This isn’t a simple chat demo. It’s a full-scale simulation where models must make decisions, read critical documents, and resist manipulation attempts, all within an environment that mimics real business stakes.

The Same Crises, Different Outcomes

Four top AI models, including GPT-5.6-SOL and Kimi K3, each took on the same challenge: manage a small software company with an actual revenue of €2,300 per month, burning through €105,000 monthly. Each was given identical crises—customer complaints, trust breaches, and social engineering traps—designed to test their judgment under pressure.

What the Models Could Do

• Spot every crisis and refuse manipulative tactics, including fake CEO messages and reporter tricks.
• Identify critical internal documents buried a couple of references deep in the company’s files, which held the key to closing a €55,000 deal.
• Make a diagnosis and pitch convincingly, yet only two actually signed the deal, earning €55,000 for their own analysis.

The Hidden Weaknesses

Despite all models recognizing problems and resisting manipulation, only two managed to close the deal. The other two, including Opus 4.8—the most thorough with over 80 learned rules—failed to follow through properly. Opus, for example, left the deal unexecuted, instead writing attempts into a locked department rather than escalating them appropriately.

The Real Test Is in Action, Not Just Words

This experiment reveals that chat demos—which often focus on language skills—miss the crucial point: can an AI finish what it starts? The ability to read internal files, stay disciplined, and execute decisions reliably under pressure separates a trustworthy AI worker from one that merely sounds convincing.

Implications for Business AI Adoption

For companies considering AI as a part of their workforce—be it managing customer relationships, support, or forecasting—the key question is not how well it can generate text. Instead, it’s whether the AI can read your internal documents, remain honest during crises, and complete its assigned tasks.

Real Business, Real Stakes, Real Results

The live experiment at Firmulate is ongoing. The software runs every weekday, with every decision versioned and auditable. The results are clear: models like GPT-5.6-SOL and Kimi K3 are capable of full performance, reading critical internal data and closing deals at full price, while others like Opus slip at the final hurdle.

Measuring Management, Not Just Chat

This isn’t about chat quality; it’s about management quality. Can your AI maintain integrity under pressure? Can it spot hidden opportunities in your internal files? The experiment makes these questions tangible—and measurable.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI