Why we grade the work product, not the transcript
A model can sound right and still write the wrong discharge plan. Here is how we grade what matters: the finished work, against a rubric written before anyone sees an answer.
Written by Lena Sørensen · Research Lead, Evaluations
- Updated Nov 13, 2026
- 3 min read

Every task in the Sixth Index ends in an artefact a professional would hand over: a care plan, a contract redline, a reconciled ledger, a merged pull request. That artefact is what gets graded. Not the explanation, not the chain of reasoning, not how confident the model sounded.
This post explains why we work that way, and what it takes to make grading reliable enough that two experts agree on the result.
The problem with grading conversations
When people evaluate a model by reading its answer, they are grading two things at once: whether the answer is right, and whether it sounds right. Those come apart more often than you would think, and fluent models are very good at the second.
- A legal memo can cite the right statute and still reach the wrong conclusion.
- A financial model can explain its method clearly and still not tie out.
- A code change can be described perfectly and still break the tests.
In each case, the professional who receives the work would reject it. So that is the standard we use.
Rubric first
Before a single task is written, the experts who will grade it agree a rubric: what a correct answer must contain, what it must not contain, and how partial credit works. The rubric is fixed before anyone sees a model's output, so nobody can move the goalposts after the fact.
- Every rubric line is binary. "Adjusts the dose for kidney function": yes or no. No five-point scales.
- Every line is observable in the artefact. If you can't point to where the plan does it, it didn't.
- Every line is weighted by consequence, agreed in advance. A missed drug interaction outweighs a formatting slip.
Why binary lines matter
Scales invite disagreement. Ask two physicians to rate a plan's safety from one to five and you will often get a three and a four. Ask them whether the plan checks potassium within a week of starting the drug and they will almost always agree.
Two graders, every time
Each item is graded by two experts who did not write it, working independently. Where they disagree, a third expert reviews the item, and if the disagreement comes from an ambiguous rubric line, the line is rewritten and the batch is re-graded.
The hard part isn't finding the error. It's writing down why it's an error, so the next reviewer agrees.
That quote is from one of our attorneys, and it is the whole job in a sentence. Every score comes with a short written rationale, which is what makes the data useful for training as well as for measurement.
What you get
When we deliver an evaluation or a dataset, it comes with the numbers that let you trust it:
- Per-line and per-task agreement rates between graders.
- Error bars on every headline score.
- The rubric templates, so your team can extend the set.
- A breakdown of where models fail, not just how often.
If you want to see what this looks like on your own model, the Sixth Index shows the method applied across medicine, law, finance and software, and our whitepapers go deeper.



