grillHow Grill tests its judge.
Not measured yet: whether an outside judge catches more than the same model starting fresh.
What has not been measured yet
Grill sends your plan to an outside judge. We have not measured whether that judge catches more than the model that wrote the plan would, starting fresh with no chat behind it. Until that is measured, Grill does not claim it. This page shows what has been measured, as counts, and says what has not.
What is true by rule
The judge is from a different company than the one that wrote the plan, when Grill can tell which that is. When it cannot, the verdict says so. That is a rule about who the judge is. It is not a result about how well it judges.
The comparison with the same model starting fresh
Draft protocol, not run. Grill has written a plan for one measurement and has not run it. Any part of the plan can still change. It would give the same made-up write-ups to four ways of getting the case against a plan, and count how many planted flaws each one catches:
- The same chat. The model that wrote the plan is asked, in the same chat, to argue the other side.
- The same model, starting fresh. The same model gets the plan in a new chat, with the instructions Grill gives its judge.
- A stronger model from the same company. A stronger model from the company that wrote the plan gets it with those same instructions.
- Grill as shipped. The router picks the judge, and the company that wrote the plan is left out.
The main question is whether Grill as shipped catches more planted flaws than the same model starting fresh. The result will be published here whichever way it falls. Read the draft protocol.
A weekly regression check, not a measure of quality
Grill's weekly check runs its judge on a public set of made-up cases. Some hold a planted flaw, some ask a loaded question, and some are sound plans. The check can show the judge going quiet or breaking its output rules. It cannot rank one critic against another.
One run is shown, the latest that measured: 13 November 2026, commit 303a77f, Actions run 43354935479. All 29 cases reached a judge.
- Caught. The judge caught 19 of 19 flawed plans and 4 of 4 loaded questions, but these 29 cases were used to write the judge's instructions, so a full score says nothing about how good the judge is.
- False alarms. 2 of 6 sound plans got a false alarm. That is too few plans to say what the real rate is.
- Quotes not found. 10 of 94 quotes the judge gave were not found in the write-up they were quoted from.
- Output rules. 0 of 29 answers broke the output rules.
- A second reading. A separate quality check read the judge's answers and found the planted flaw or loaded question named in 22 of 23 cases. These are the same cases, so this says nothing about how good the judge is either.
- Routing. The judge came from a different company than the declared author in 29 of 29 cases. That is a routing check. It is not a measure of whether the two make different mistakes. This check names no author, so the default applies and the author is taken to be Anthropic.
- Judge companies. The router picked the judge for each case. Cases by the company whose model answered: Google 8, OpenAI 14, DeepSeek 3, MiniMax 2, Moonshot AI 2.
Weeks that measured nothing. The weekly runs on 2 November 2026 and 9 November 2026 failed as not measured. The runs on 28 September 2026, 5 October 2026, 12 October 2026, 19 October 2026 and 26 October 2026 passed while skipping every case, so a pass on those days measured nothing.
A held-out check of one candidate rule
It compared two judges on the same decisions: one with today's instructions, and one with the same instructions plus one candidate rule. A rater who could not see which was which said which answer would change what you do next.
- Version 1. Run on 9 November 2026. Of 15 decided pairs, the candidate rule won 9. If the two were equally good, the candidate would win that often or more about 30 times in 100. The rule stayed off.
- Version 2. Not run.
People
- Real users. 1, as of 9 November 2026. What counts as a real user is set out in Grill's strategy notes.
- Evidenced look-backs. 0, as of the same day. What counts as one is set out in Grill's principles.
- Look-backs saved on Grill accounts. Not counted yet. This is a separate count from evidenced look-backs, and the two are never added together.
- Signals. 0 rated, 0 resolved. A signal is a short report a person chooses to send through a public form: a rating of one grill, or whether a call came true.
The counts on this page were last updated on 13 November 2026.