Pilots compare aeroplanes the way other people compare weather: constantly, affectionately, and with strong opinions about stall speed. Software folks do the same thing with their testing strategies, and if you have been hearing the word TestOps around engineer friends — short for operating your automated tests like a service, so every pass/fail verdict arrives fast and can be believed — you may be wondering which flavour is actually best. This piece lines up the main contenders the way a fair comparison should: what each one assumes, what it costs, and who it suits. No winners declared without a flight test.
Contest one: TestOps shapes — pyramid, honeycomb, trophy
Three shapes dominate the argument about what a test suite should contain, and each is really a bet about where defects hide cheapest.
- The pyramid — Mike Cohn’s drawing from around 2003-04, and still the default answer. Hundreds of tiny fast checks at the base, fewer around service boundaries, a handful of full-journey tests at the top. Its bet: most bugs are cheapest to catch close to the code.
- The honeycomb — a newer shape from teams whose products are mostly integrations. It loads the middle: lots of tests around the seams where your system talks to other systems, fewer deep in internals, thin on user-interface journeys. Its bet: in integration-heavy products, the seams are where fires start.
- The trophy — popular in front-end circles. Weight sits on tests that render components and exercise user flows in memory, above unit checks, below browser-driving end-to-end runs. Its bet: users experience the interface, so test where the user lives.
So which should a small team pick? Ask what the product mostly is. A data-crunching service leans pyramid. A payments hub glued to six external APIs leans honeycomb. A customer-facing web app leans trophy. Mission first, diagram second.
Here is the verdict the comparison-minded should tattoo somewhere: the practitioners who run enormous suites — the people catalogued in the Ops Atlas TestOps guide — mostly agree the shape matters less than a pair of numbers: what it costs to run each layer, and whether that layer returns the same answer twice in a row. A cheap, deterministic suite in any shape beats a fashionable one that flips answers overnight. Choose like you choose an aeroplane for a mission, not for the ramp photo.
Contest two: Google’s yardstick versus Microsoft’s
When two of the world’s heaviest test-running organisations describe how they classify checks, they reveal two genuinely different philosophies, and comparing them is instructive.
Google sizes by resources consumed. A small test touches memory only; a medium one may use a single machine; a large one may roam. The sizes map directly to timeouts in their build system, Bazel: 60, 300 and 900 seconds respectively. The philosophy is economic — every check pays rent proportional to what it uses.
Microsoft’s older scheme, L0 through L4, classifies by scope and environment instead. L0 checks are tightly scoped unit tests — the guidance targets averages under 60 milliseconds with none exceeding two seconds — and the ladder climbs to L4, which is testing against production itself. The philosophy is architectural: know which layer of the world you are inspecting, and keep the fast layers fat.
Neither is wrong. If your pain is CI spend, the resource view shows you exactly what to shrink. If your pain is slow feedback, the layered view shows you which tiers to fatten. The aviation equivalent is the difference between filing an aeroplane by useful load and filing it by what systems it carries — one lens answers “what can this haul?”, the other answers “what can this do?”, and an owner eventually needs both. Most teams benefit from asking both questions of every test they own: what does this consume, and which layer does it actually inspect? Keep the answers in one spreadsheet and the comparisons stop being philosophy and start being arithmetic.
Contest three: adopting TestOps now versus waiting
The friendliest thing a comparison can do is talk you out of a purchase. So, two lists.
Signals that the TestOps work is worth starting — if two or more describe you, begin:
- Your quick pipeline stage drifts past ten minutes — beyond where DORA’s delivery research puts the limit of feedback anyone will sit still for.
- Re-running failed jobs before reading them has become muscle memory on your team.
- The main branch goes red weekly and the cause turns out to be noise, not bugs.
- More than one team shares the suite, and collisions are routine.
- Someone outside engineering has begun asking what all this compute costs.
Signals that you would be solving the wrong problem — any one of these and hold off:
- The entire suite completes in less than five minutes. There is no operations problem here yet.
- When you investigate a failure, it is almost always a real defect. Your checks are honest; trust them.
- The genuine gap is behaviour nobody ever wrote a check for. That is a coverage project, not an operations one.
Choosing like an owner, not a shopper
Notice the thread through all three comparisons: the winning choice is the one matched to your mission, measured in your own numbers, and re-evaluated on a schedule. That is exactly how this site approaches ownership questions elsewhere — whether that is the safety checks every light-aircraft pilot should know, the discipline of keeping maintenance records that matter later, or building a maintenance schedule you can actually hold to. Software teams call the same habit TestOps when the machine being maintained is the test suite itself: measure the verdicts, budget the stages, and retire what no longer earns its place on the ramp. The shape you fly is your business; the trustworthiness of the answer is the maintenance standard.
