Data provenance, in RLHF and human-data pipelines generally, is the auditable link between a unit of data — a preference judgment, a demonstration, a graded evaluation — and the person who produced it: who they are, what expertise they were verified to have, and the quality trail behind the work.
That is the whole definition. The substance is in what each part rules out.
The three components
- Attribution — the unit points to a specific person, not an anonymous account ID or a batch. Accounts can be shared and resold; attribution to an account is attribution to nobody.
- Verified expertise — the person's identity, employer, role, and seniority were established against systems of record, so the attribution means something. Attribution to an unverified profile is a pointer to a claim.
- The QA trail — who reviewed the unit, against which rubric, with what outcome. Provenance without the quality record answers who, but not how well.
Why per-unit granularity is the test
Vendors sometimes claim provenance at the batch or project level: this dataset was produced by our verified workforce. The claim is unfalsifiable at exactly the moment it matters. When a contributor is later found to be misrepresented, per-unit provenance lets you trace and excise their work; batch-level provenance leaves you re-auditing everything or trusting a reassurance.
The test is simple: pick any row in the dataset and ask the vendor to walk it back to a human being.
Why RLHF makes provenance urgent
In supervised learning on curated corpora, a bad source document is one bad document. In RLHF, human judgments are the reward signal — an unqualified or fraudulent grader does not add noise, they bend the objective the model optimizes toward. And because a confident wrong label looks identical to a right one on the surface, the only durable defense is knowing, verifiably, who is doing the judging.
Rainbow attaches provenance to every unit of work we deliver: the verified expert behind it, the role and seniority they were verified under, and the QA trail. For the verification standard underneath that promise, see the essay on the anatomy of a source verification; for turning provenance into a procurement requirement, see the ten-question provenance audit.
Expert human data, verified at source.