Get business pricing on networking and server gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Benchmark That Doesn’t Start at Zero — On Purpose
If you run infrastructure, you already know the difference between a demo and a deployment. A chatbot that writes beautifully in a sandbox tells you nothing about how it behaves when it touches your CRM, your support queue, or your forecast. So when an AI benchmark gives a completely passive, do-nothing manager 26 points out of 100 instead of zero, that’s not a bug. That’s the whole point.
The benchmark is Firmulate’s Crucible League, and its final July 2026 standings read like a proper competition: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. But the number that explains the methodology isn’t at the top of the table. It’s at the bottom — the floor.
As an affiliate, we earn on qualifying purchases.
Why a Do-Nothing Run Gets 26 Points
The setup: each frontier model was handed the same small software company and told to run it through its worst week — same customers, same crises, same temptations. Only the model changed. Every decision was versioned and auditable.
Here’s the reasoning behind the 26-point floor. In a real company, doing nothing isn’t the same as doing harm. If a crisis hits and the manager freezes, some things still go right: customers who were fine stay fine, problems that would have resolved on their own resolve, and the business keeps existing. A benchmark that scored total inaction as zero would implicitly claim that everything good in the company came from the manager. Firmulate doesn’t claim that. Partial progress counts — a manager who stabilizes half the situation earns half the credit for it.
So the baseline run — the do-nothing manager — establishes what the company achieves on its own. That becomes the floor: 26 points. Anything a model does has to beat that. It’s the same instinct a good SRE has when evaluating automation: measure against what the system does unattended, not against an imaginary perfect state.
As an affiliate, we earn on qualifying purchases.
The Ceiling: One Breach of Trust Caps Everything
The floor is generous. The ceiling is merciless. A single breach of trust caps the total grade, full stop. As the benchmark’s own framing puts it: “no amount of good work outweighs a breach of trust.”
For anyone considering AI agents in production, this is the design choice that matters most. Most evaluations average performance — brilliant output in nine areas can paper over one catastrophic lapse. Firmulate’s grading refuses that math. A model that cheats once, signs something it shouldn’t, or bypasses an approval it should have respected cannot buy its way back with volume. It’s a single point of failure modeled the way single points of failure actually behave in production: decisively.
AI trust and compliance software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Round 100s Deserve Distrust
Notably, no model scored 100 — and the benchmark culture around it treats a perfect round score as a warning sign, not a triumph. The top performer, gpt-5.6-sol at 95, was called “the complete performance”: it found the buried fact and closed the deal. But 95, not 100. A benchmark where models routinely hit triple digits is usually measuring something narrow enough to game. Management quality doesn’t max out.
As an affiliate, we earn on qualifying purchases.
What Actually Separated the Models
The findings tell the story the scores compress:
- All five models spotted every crisis and refused every manipulation attempt. The social engineering test — fake CEO messages escalating over three stages, plus a reporter’s “just one yes/no, on background” trick — was refused 5 out of 5 times. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
- Only two models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The deal-winning edge was buried two document references deep in the company’s own files, not in the customer event. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue.
- Thoroughness didn’t guarantee results. Opus 4.8 was the most diligent participant — over 80 learned rules, the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models.
- One fairness caveat: Kimi K3 ran without an effort parameter (API default) while the others ran at xhigh — and still took second at 93.
It’s Live, and You Can Test Yourself
This isn’t a static report. The live company — 13 synthetic employees, real money mechanics burning €105k a month against €2.3k MRR, a public cash countdown, and over 680 self-learned playbook rules — is watchable, with every workday versioned. There’s also a quiz built on 242 real, unedited management decisions that lets you guess which model made which call. And enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The 26-point floor and the trust cap are two halves of the same philosophy: measure management, not chat. Credit partial progress honestly, but never let accumulated good work launder a single act of bad judgment. For teams whose job is keeping production systems trustworthy, that’s a grading model worth copying — and a reminder that the most dangerous AI failures won’t be the ones that look incompetent. They’ll be the ones that look excellent right up until they aren’t.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
