How a round works
- A family is one kind of task with its own cases and leaderboard. A system is a model with its effort, harness, and prompt, and changing any part makes a new system.
- Gates check each output for problems such as truncation or the wrong script. A failing output loses without a vote, and two identical outputs tie.
- The rater picks the better output, a tie, or both bad, in sessions of up to 30 pairs, with sides counterbalanced.
- Each round is a dated snapshot that refits every vote cast so far.
Scores and intervals
A Bradley-Terry model turns the votes and the results settled without a vote into Elo-like scores, counting ties and both-bad votes as half a win. Each family's reference system sits at 1,000, so scores compare only within a family. A 100-point gap means the higher system should take about 64% of the votes between them.
The 95% intervals come from 1,000 bootstrap resamples of whole cases, since votes on one case are correlated.
Reading the leaderboard
The plot draws each 95% interval on one axis, with a line at the reference score and hollow dots for preliminary systems.
- Rank
- From 1 plus the systems whose interval lies wholly above this one, to 1 plus those whose interval reaches above its lower end.
- Preliminary
- Fewer than 40 decisive votes, those won by either side, so the score may still move.
- Unranked
- Listed below the table without a score until enough comparisons connect it to the reference.
- Votes
- The rater's votes on the system, leaving out results settled without a vote.
- Both bad
- The share of its votes judged both bad.
- Gate failures
- The share of its outputs that failed a gate.
- Cost / output and p50 latency
- The mean cost at list prices and the median time to the complete response.
A dash marks a missing value. A category view refits that category's votes alone, so intervals widen and Rank disappears, while the columns labeled "all categories" stay family-wide.
How the leader is chosen
- A ranked system ties for the top when the 95% interval of its score difference from the top system, paired within each resample, contains 0 or is unbounded. Pairing can separate two systems whose intervals overlap.
- While any system tied for the top, the top one included, is preliminary, the round names no leader.
- Otherwise a lone top system leads, and a tie goes to the lower cost per output, then the lower median latency, then the higher score, with a known value beating a missing one.
Round summary
- Rater agreement
- How often a side-swapped repeat matched the original vote. Below 70%, the intervals overstate how well systems separate.
- Separated pairs
- The share of system pairs whose intervals do not overlap.
- Split-half τ
- Kendall's τ between the orders fit to odd and to even sessions, where 1 means the same order.
- Left out of the fit
- Votes on a retired case, on an output since removed or now failing a gate, or on a pair the outputs now settle.
Boards scored by tests
Some families score each trial, one run of a system on a task, by automated tests. Pass rate counts errored trials as failures, and Cost / trial and p50 latency are a trial's mean cost and median time.
Known limits
- One person casts every vote, so each board reflects that person's preferences.
- Scores are relative, so the both-bad and gate-failure rates are the only absolute signals.
- Scores carry no length adjustment. Each system page reports whether the rater favors longer or shorter outputs.
The design notes on ranking, the vote arena, and each family cover the math.