← All posts

---

Arena4Ai points two coding agents at the same brief and lets a judge model score what they actually built. The centrepiece is an isometric arena where each agent gets a floor, and every block on that floor is a file the agent wrote. Watch it run and you see two cities go up at different speeds, in different shapes.

The picture is the product. Which means the picture has to be true.

01The thing we got right

Early on we hit a problem that has no clean answer: the agent CLIs do not all report their work the same way. Some tell you every file operation. Some tell you a subset. Some tell you almost nothing.

So the renderer draws every block in one of three states. Measured — solid, we watched it happen. Inferred — dashed cap, we know the file exists but not how many times it was touched. Unavailable — hatched, this CLI structurally cannot tell us.

The important decision is what height an unavailable block gets. The intuitive answer is a short one, or none at all. That answer is wrong, and it is wrong in a way that matters:

So unavailable blocks are drawn at full height, hatched. There is a test that pins it, and the test is the whole argument in one line:

targetHeight(n, 'unavailable') === targetHeight(0, 'measured')

We also made the renderer draw that legend onto the canvas itself, on by default, so the caveat travels with the picture rather than living in whatever page happens to embed it. A host can switch it off — our own launch video does, because a dense legend is unreadable in a ten-second shot — but switching it off means taking on the job of saying the same thing another way.

We were, we thought, being careful.

02The thing we got wrong

Then we cut a launch video, and one scene put a real competition on screen: Claude against Codex, same brief, same clock. Under the floor went a caption with the numbers on it.

It is a great line. Four times the files, the same score — the judge scores the work, not the volume. That is genuinely what the product is for, and here was a real competition saying it out loud.

Before it shipped, we checked the numbers against the database. Not because anything looked wrong, but because it was a public claim about a competitor’s model, and those get checked.

source	claude	codex
caption on screen	17 files	4 files
file events in the stream	18 (17 distinct)	4
files actually delivered	17	17

Codex wrote seventeen files. The floor was never lopsided. The caption was false, the argument built on top of it was false, and it was about forty minutes from being the hero video on this website.

03Corroboration from somewhere else entirely

One number disagreeing with another number is not proof of which one is wrong. What settled it was a third source with no connection to either: the judge’s own written commentary.

The judge reads the workspace, not the event stream. Its notes on the Codex submission discuss extract.mjs, setup-guide.md, manifest.json, a service worker, and an IndexedDB offline queue. That is already more than four files, described by something that had looked at the actual result.

When two independent sources disagree with a third, the third is the one to doubt.

04The root cause was not the one we assumed

The obvious explanation was that the Codex CLI under-reports — that it simply does not emit an event for every file, and this is a limit we have to live with and label. That explanation was wrong, and it is worth being precise about why, because the wrong version would have sent us to build the wrong fix.

Codex applies edits through apply_patch and emits them as unified diffs. Every one of those seventeen files was in the event stream the whole time, sitting in the diff headers as +++ b/<path> lines. Our normalizer threw them away: one branch emitted a single event and then skipped the remaining diff lines, another suppressed lines beginning with apply_patch.

We extracted every distinct path from those diff headers and set-compared it against the delivered files:

comparison	count
matched	17
only in the diff headers	0
only in the delivered files	0

An exact bijection. The data was never missing. We were discarding it on the way in.

That reframes the whole thing. It is not a limitation of what Codex tells us. It is a defect in what we read — which means it is fixable, and which means every past competition involving Codex has been quietly understated in our own arena.

05What actually went wrong, though

Here is the part worth keeping, and it is not the parser bug.

We had already built the safeguard. Three states, full-height hatching, a test pinning the exact reasoning, a legend welded to the canvas. All of it designed around one insight: never let a gap in logging render as a claim about a model.

And then we wrote a caption, in a different file, in a different language, that made exactly that claim — in plain text, where no hatching and no legend can reach it.

There was a second lesson underneath it. The first fix proposed was simply to correct the number: say seventeen against seventeen. But the floor still had four blue blocks on it, under a headline reading EVERY BLOCK IS A FILE. A true caption under a false picture is still a false scene. The fix had to be the picture too.

06What we changed

The caption no longer makes a per-team file comparison at all. It says who competed and what the judge scored, because the scores come from the judge and not from the event stream. There is a comment at the site of the change explaining why no file counts may be added back.

The floor now draws all seventeen Codex blocks — the thirteen recovered from the diff headers marked inferred rather than measured. Not hatched: hatching means we do not know, and here we have an exact match against the delivered files. What we genuinely cannot see for those files is how many times each was edited, and inferred is precisely what that already means.

The parser is being fixed at the source, with that seventeen-path bijection as the test oracle. And a reconciliation check is going in at data-load time rather than in a capability table — comparing what the stream says against what was actually delivered, and downgrading confidence when they disagree. A static table saying “this provider is incomplete” would itself become a lie the moment the parser is fixed.

---

None of this was caught by a test. Every test was green. It was caught by checking a number against the database before putting it in front of strangers, because the claim was about somebody else’s model.

The arena exists to measure what coding agents actually did, rather than what they said they did. It would be a poor advertisement for that if we were not willing to point the same instrument at ourselves.

Arena4Ai is free and self-hosted. It runs the agent CLIs already installed on your machine — there is no API key. The repo is here.