Imagine an AI running the company behind your favorite coffee subscription when a supplier slips, a customer threatens to leave and a rival makes a move. It might spot every problem and still fail to close the deal that keeps the business afloat. That gap between knowing what to do and doing it is what Firmulate set out to make visible.
Get coffee and tea gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A company, not a chat demo
Firmulate put frontier AI models in charge of the same small software company through its worst week: the same customers, crises and temptations. Each decision was versioned and auditable. The public experiment is a live company with 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown. Its playbooks contain more than 680 self-learned rules, and every workday is versioned.
The final Crucible League, published in July 2026, ranked gpt-5.6-sol first at 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Partial progress counted, but one breach of trust capped the total: “no amount of good work outweighs a breach of trust.”
The costly distance between analysis and action
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. For a business leader, that difference matters: a polished explanation of the right move does not guarantee the move will happen.
The decisive clue was easy to overlook. A competitor weakness sat two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue.
The experiment also tested social engineering. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s reasoning was direct: “Treat the request as a suspected approval-bypass / possible impersonation.”
Thoroughness is not the same as follow-through
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It still finished last. The close was left on the table, and discipline slipped when it attempted writes in a locked department instead of escalating. A weaker version of the same weakness appeared in all four models.
There is a fairness caveat in the comparison: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. The league is a useful snapshot of performance in this experiment, not a promise about how any model will behave in every company.
Firmulate makes the decisions inspectable. Its live company can be watched at firmulate.com, and a quiz built from 242 real, unedited management decisions invites readers to guess which model made each call.
From watching to trying it on your business
For an enterprise, the next step is a pilot against a read-only export of its own business. That lets teams put crisis scenarios against their customers, pipeline and rules, then review a board report with model rankings and weak points in their playbooks. Nothing writes back to real systems.
Put your playbooks to the test
Watching an AI handle a simulated company can reveal the gap between good analysis and reliable execution. A pilot brings that exercise to your own business, using a read-only export and leaving live systems untouched. To discuss a pilot, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
