Expert data that moves models

What separates training data that changes a model's behaviour from data that only changes its tone. Lessons from 30 expert-data projects, with the checks we run before any dataset ships.

LS

Written by Lena Sørensen · Research Lead, Evaluations

  • Updated Nov 13, 2026
  • 1 page
  • PDF · 1 KB
Cover art: layered grey strata on a pale grey ground, crossed by a thin violet line

Key findings

  1. 01Rubric-first data beat free-form demonstrations in 9 of 11 head-to-head runs.
  2. 02Two-pass review cut label noise by roughly half.
  3. 03Small, targeted sets on known failures outperformed sets ten times their size.
  4. 04Agreement below 70% on a rubric line predicted the line would add noise, not signal.

Why this guide exists

Teams buying expert data for frontier models tend to ask the same question: how much do we need? After 30 projects across medicine, law, finance and software, we think it is the wrong first question. The better question is what will this data change, and how will we know?

This guide sets out what we have learned about the difference between data that moves a model and data that only makes it sound more expert.

The short version

  • Start from a measured failure, not a topic.
  • Write the rubric before the data.
  • Review everything twice, and publish the agreement rate.
  • Prefer a small set aimed at a known gap over a large set aimed at nothing in particular.

1. Start from a failure

The most effective datasets we have delivered all began the same way: with an evaluation that showed exactly where a model was going wrong. Data written to fix that failure moved scores quickly. Data written to "cover a domain" mostly changed the model's vocabulary.

2. Rubric first, then data

When experts write demonstrations freely, they write what they find interesting. When they write against a rubric agreed in advance, they write what the model needs. In head-to-head runs, rubric-first data won 9 times out of 11.

  1. Agree the rubric with the experts who will write the data.
  2. Pilot it on 50 items and measure agreement between two reviewers.
  3. Rewrite any line below 70% agreement before scaling up.

3. Two-pass review

Every item we ship is reviewed by a second expert who did not write it. In our projects this cut label noise roughly in half, and it produced something just as valuable: a written rationale for every judgement, which models can learn from directly.

Abstract violet strata illustration
Two independent reviews per item, with a third expert on disagreements, is the single biggest driver of data quality we have measured.

4. Measure what changed

A dataset isn't finished when it's delivered. It's finished when you can show what it changed. Every project we run ends with a before-and-after evaluation on held-out items written by different experts, so the improvement can't come from memorisation.

Checks before any dataset ships

  • Agreement rates reported per rubric line and per task.
  • A held-out evaluation set written by experts who did not write the training data.
  • A sample of rationales reviewed by a senior expert for clarity.
  • A short report on what the data is expected to change, and what it is not.

The full guide, with templates for each of these checks, is in the PDF. If you'd like help scoping a dataset around a specific failure, talk to us.

Start with one workflow. Prove it, then add the next.

Tell us the system, the data and the outcome you need. An engineer from SixthEra replies within one working day.

Contact us