How models perform as panelists.
Every model will be run against the same benchmark suite — cases carrying planted defects — then scored on how many of the planted defect categories it surfaced, averaged across cases. No run has been published yet; the rows below are placeholders.
| # | Model | Accuracy | Cost / corroborated finding | Latency |
|---|---|---|---|---|
| 01 | claude-sonnet-4.6 | 94.2% | n/a | 1.2s |
| 02 | gpt-4o | 92.8% | n/a | 0.9s |
| 03 | gemini-2.5-pro | 90.6% | n/a | 1.5s |
| 04 | llama-3.3-70b (local) | 88.1% | n/a | 4.1s |
| 05 | claude-haiku-4.5 | 85.4% | n/a | 0.4s |
| 06 | qwen-2.5-coder-32b (local) | 82.0% | n/a | 3.3s |
Illustrative data for the concept mockup — not a published benchmark.
When this board carries real numbers, they will be first-party measurements: the suite run by the maintainer across each model, not scores submitted by their vendors or users. That is the choice made here rather than an upstream requirement — the suite contract only guarantees a submission is suite-sourced, not that whoever sent it actually ran anything. It is why there is no submission endpoint.
No run has been published yet, so every row above is a placeholder — the suite, its case count, and the run date will appear here once one has.
Cost reads n/a on every row because nothing has been measured — the mockup carried per-1K-token prices, which are a different quantity, and relabeling them would have swapped one metric for another without saying so. The Accuracy column is the remaining open question: the suite scores category recall, and what fills "Accuracy" gets settled before any real run is published, not after.
How the suite scores a model
Seat the model
Each model in the roster reviews every case in the suite as a single panelist, emitting findings in the same standard format a real review produces.
Same ingestion path
Each case's diff goes through the same ingestion the production pipeline uses, so a model is scored on exactly the payload it would see on a real review.
Score vs. planted defects
Each panelist's findings are matched individually against that case's planted defect categories — no clustering step — giving category recall. Precision is not scored: planted defects are only a subset of what a reviewer may legitimately find.
Recall is one input, not a verdict. A cheap, fast model that catches fewer planted defects is still a strong panelist when paired with a slower, pickier one — because the reconciler scores them together. Use the table to staff a panel, not to crown a winner.
Verify your own roster against every configured endpoint before you spend a review on it.