On 28 October China's National Data Administration held a workshop with 21 chain-leader units drawn from nine fields: education and teaching, health care, scientific research, industrial manufacturing, bio-manufacturing, agriculture and rural affairs, smart energy, transport, and meteorological services. The readout, published on 2 November, sets out three principles, usually rendered as build to demand, build through use and improve through evaluation, followed by six tasks. Five of the six are organisational: map AI data demand beyond your own firm, create benefit-sharing arrangements and an industry data map, feed real usage back into updates, coordinate along the supply chain, and connect to national data infrastructure. The sixth is the one with engineering consequences: move dataset quality evaluation to a tool-based, order-based model, with shorter cycles and lower cost.
That sentence reads like procurement vocabulary and is easy to skip. It is more specific than that. An order is a unit of work with a defined deliverable; a tool is a repeatable method that produces it. Together they describe a contract in which a buyer names a task and receives a number produced by an agreed instrument. Once quality is ordered, it stops being an adjective in a slide and becomes a specification. And a specification that is not pinned down in advance gets written, by default, by whoever runs the evaluation.
A score is only worth its definition. Two datasets scored 82 under different metric definitions are not comparable: same number, different quantities. Comparability needs, at minimum, the metric definition, the sampling frame (which records, how drawn, how many), the label-schema version, the evaluation-set version and the tool version. Leave any of the five off the report and the number becomes a mood. This is not a hypothetical risk; it is the normal state of dataset quality claims in industry, where a vendor score is usually a private rubric applied once.
A registry is an inventory, not a quality claim. Task two asks chain leaders to register their datasets in the national dataset management service system. A registry entry answers what exists, who holds it, what it covers and under what terms. It does not answer whether the data is any good, and it should not pretend to. But the registry schema decides whether any later evaluation can be stratified at all. Without fields for task type, modality, collection window, label-schema version, annotator profile, provenance and licence, an evaluator cannot compare like with like, and the register degrades into a contact list.
Distributed aggregation moves the scorer, not the data. Task two also asks for an industry data map and distributed aggregation of sector data. If the premise is that data stays where it is, a central evaluation service cannot see the rows it is supposed to score. The architecture that follows is not one large upload. It is a packaged evaluator, a versioned container or script, shipped into the data holder's environment, returning metric vectors and an audit log rather than the data. That constrains the metric set to what can be computed in place: schema conformance, duplicate and near-duplicate rates by hashing, label distribution and drift, temporal coverage, provenance completeness, and item-level agreement on a small adjudicated sample. Semantic correctness still needs human eyes on sampled items; it can be estimated in place, not settled in place.
What a usable metric vector contains. Coverage of the declared intents and failure modes relative to the intended use; inter-annotator agreement with a confidence interval on a double-labelled sample; duplication; freshness measured per slice rather than as one date; contamination against the evaluation sets the buyer intends to use; slice balance; schema conformance; and consent or PII flags. A single scalar compresses all of this into something nobody can act on, and it systematically favours datasets that cover easy cases well over smaller datasets that cover hard cases honestly.
The defect list is the deliverable. A score that cannot be converted into an action is a receipt. If each metric does not map to a decision, re-collect this slice, re-label these classes, retire this source, re-split this set, then the feedback loop in task three has nothing to feed. The design question is whether the report is machine-readable enough that a collection plan can be regenerated from it. That is a different output format from a dashboard, and it is the one that makes an evaluation service useful to a data team rather than to a procurement office.
Short cycles and low cost mean sampling, and sampling has error bars. An order that promises a verdict in days at low cost will be sample-based for anything requiring judgement. A 200-item audit says something about a population within a few percentage points and almost nothing about a tail slice of thirty items. Stating the sample size, the seed and the strata is not academic formality; it is the difference between a measurement and an impression. Evaluation services that omit sample sizes will be the cheapest to buy and the hardest to compare.
Contamination and split hygiene are where scores inflate silently. If the evaluation set is drawn from the same distribution as the training corpus, or worse, if benchmark items were in the crawl, every downstream number is biased upward and nothing in the pipeline flags it. The checks are known: item-level overlap by hash and near-duplicate matching, temporal cutoffs so that evaluation data is strictly later than collection data, and entity-level separation for patient or customer records. None of these require the model. All of them require an evaluator that can see both sides, which is another argument for running the check where the data lives.
Sector failure modes do not reduce to one template. Medical datasets fail through patient-level leakage and uneven annotator qualification. Industrial logs fail through sensor calibration drift and clock skew across machines, plus rare-fault imbalance that makes headline accuracy meaningless. Meteorological data fails through physical inconsistency between variables that a per-field check will not catch. Agricultural data fails through seasonal coverage that no single collection window can fix. A national template scoring all nine fields on the same dimensions will measure the paperwork they have in common and miss the defect that decides each one. The realistic design is a small shared core, covering schema, provenance, duplication, consent and sample agreement, plus sector modules that the field's own institute defines.
The standards bodies in the room are the tell. The China Electronics Standardization Institute and CAICT attended alongside the National Development and Reform Commission and the NDA's own research institute. Standards work is where metric definitions become public and reference implementations become possible. Whether the tool-based evaluation service publishes its metric dictionary and a reference implementation, rather than only its scores, decides whether results are comparable across vendors or only within one. A procurement regime in which only the incumbent's tool reproduces the number is a lock-in with a quality label on it.
Datasets age, so the date belongs inside the score. A corpus that scores well in November can be stale by spring if the domain moves or the product changes. Freshness should be a dated measurement per slice, not an attribute refreshed once a year, and re-evaluation cadence should be part of the specification rather than an optional extra. Short cycle should mean re-scoring on a schedule, not merely fast first delivery.
Three artifacts would settle the question by spring: a published metric dictionary with version numbers; at least one reference evaluator that a third party can run against a held-out sample and obtain the same figure; and registry entries whose evaluation reports are linked, dated and reproducible. None of these require a new model, and all of them are checkable from outside the programme.
The testable commitment from this meeting is the sixth task. In six months the question will not be how many datasets were registered. It will be whether a buyer can place two datasets from different sectors side by side and read a number that means the same thing, produced by an instrument whose definition, sample and version are on the record. That artifact, rather than the communiqué, is what improve through evaluation will have to mean.
Sources. National Data Administration, readout of the 28 October workshop on building high-quality industry datasets with chain-leader units, published 2 November 2026 — https://www.nda.gov.cn/sjj/swdt/sjdt/1031/20261031151459885260005_pc.html