On 30 October Microsoft Research published a description of the Industry Evaluation Data Generation Platform, the internal system its Copilot teams use to build evaluation datasets without touching customer data. The post lists three parts: synthetic enterprise tenants, an automated query-generation pipeline, and a quality framework that scores the generated data. It also estimates savings of roughly $100,000 per evaluation dataset and $300,000 across the project. The cost figure will be quoted, but the claim with engineering consequences is smaller: a team can regenerate a dataset from the same configuration and get identical results. Everything else in the post depends on that sentence being true.
A tenant, in Microsoft's terminology, is an isolated workspace with its own users, meetings, email threads, documents and reporting structure. The detail that separates this from older synthetic-data work is that tenants evolve. Threads develop over weeks, projects acquire timelines, and the organisation changes. The stated reason is time-aware queries: a question such as what the team decided last Thursday, or what changed since the last quarterly review, produces a meaningful evaluation only if the underlying record has a real temporal structure. This is the most defensible design choice in the post. Static fixtures cannot fail a date-sensitive question correctly, so a model that ignores recency can pass them by accident.
The environments are grounded in three public sources. O*NET, the US Department of Labor occupational database, supplies roles, responsibilities and the mix and frequency of work activities. Job postings supply role titles, reporting hierarchies and departmental composition. SEC filings supply organisational scale, business unit structure and financial context. Microsoft says its legal and policy organisation reviewed and approved all three. These sources describe how organisations are shaped and what jobs contain. None of them describes how employees phrase requests, how often a summarisation request outnumbers a rewrite request, or what a wrong answer looks like to the person who filed the ticket.
Query generation runs through six stages. A large library of common workplace tasks is filtered to an industry, adapted with domain language, used to generate questions grounded in the tenant's own documents, rewritten to read more naturally, and finally validated for accuracy and uniqueness. The last stage does real work: uniqueness validation stops the same question from reappearing in near-identical form, a quiet way evaluation suites inflate their own scores as they grow. The pipeline does, however, contain an asymmetry. The world is grounded in public data; the questions are written by a model. Grounding means a question can only reference facts the tenant actually holds. It does not mean the question resembles one a real user typed.
That asymmetry determines what such a suite can certify. It can certify internal consistency: whether the system locates the right document, follows the thread across weeks, respects the reporting relationship, and produces an answer that matches the tenant's own record. This is a genuine failure class. Retrieval misses, hallucinated citations and summaries that attribute a decision to the wrong meeting are exactly what public benchmarks miss, because public benchmarks have no organisation to be wrong about. What the suite cannot certify is distributional match: that the mix of tasks resembles production, that the difficult cases are present in proportion, or that a score gain will appear for real users.
Quality is scored on four dimensions: structural fidelity, contextual relevance, behavioural realism, and content safety, including the absence of personally identifiable material and of impersonation of real people. Microsoft reports a 39 percent improvement on a composite quality metric against its initial generation approach, with some industry environments above 0.9 on a zero-to-one scale, across three verticals: pharmaceutical regulatory and trial workflows, capital markets deal and compliance work, and an IT company with sprint planning and code review. These are internal, model-scored judgements of synthetic data quality. The post does not publish the rubric, the agreement between the scoring model and human reviewers, or a calibration against real enterprise traffic. A 0.9 indicates that a tenant resembles the reference distribution the framework encodes. It is not a statement about task difficulty or coverage.
Reproducibility is the part other teams should copy, and it has preconditions the post does not enumerate. Same configuration, identical results holds only if the configuration pins more than a random seed. It has to pin the generator model and its version, the prompt library version, the tenant snapshot, the validator and any judge model, and the sampling parameters. Without that, the same configuration yields a different dataset after the next model upgrade, which in practice is the most common source of unexplained score movement in production evaluation. Teams adopting the pattern should treat the configuration as a versioned artifact and store the dataset hash alongside every result, so that any number can be traced back to the exact data that produced it.
The privacy architecture is worth separating into two claims. No customer data at any stage is a structural property, and the post says it is supported by provenance and lineage tracking, so it can be audited. The second claim is different: privacy-preserving aggregation techniques are used where real-world statistical distributions are referenced. Aggregates carry information, and the post does not give the parameters, such as minimum cell size, noise addition, or protections against differencing between releases. A team reusing the approach on regulated data should specify those parameters before claiming equivalence to a no-data-at-all baseline.
The design will be tested hardest by the direction the post names as next: multi-turn agentic evaluation, organisational dynamics over months or years, and systems that take actions rather than generate text. Each breaks an assumption in the current platform. Agentic evaluation changes the unit of correctness from an answer to a trajectory: which tools were called, in what order, what state changed, and whether the change was reversible. That needs mutable tenants and assertions on state, not text similarity. Long-horizon organisational dynamics require the synthetic history to stay consistent under branching, or the evaluation ends up measuring the generator's bookkeeping as much as the model's behaviour. Action-taking systems require the environment to enforce permissions, otherwise the test measures a model's willingness to act rather than the safety of acting.
For teams building something similar at smaller scale, five practices follow from the post. Treat the configuration, not the dataset, as the unit of record, and hash the output. Split suites into capability tests, which probe new failure modes, and regression tests, which protect fixed ones, because a single composite score hides regressions behind improvements elsewhere. Keep a small human-graded calibration set drawn from consented data, and re-score it whenever the generator changes. Report per-dimension scores rather than one composite, since structural fidelity and behavioural realism fail in different ways. And for time-aware and agentic cases, assert on state transitions rather than on prose.
None of this requires Microsoft's scale. The contribution the post actually makes is conceptual: evaluation data treated as engineered infrastructure with a version, a lineage and a regeneration path, rather than a folder of fixtures that nobody can reproduce. Whether the specific figures hold, particularly the 39 percent gain and the savings estimate, is the part that still needs an outside check, and the post does not yet supply the artifacts that would allow one.
Sources. Microsoft Research, From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence, published 30 October 2026 — https://www.microsoft.com/en-us/research/blog/from-synthetic-enterprise-data-evaluations-to-real-microsoft-copilot-ai-confidence/ ; O*NET occupational database, U.S. Department of Labor — https://www.onetcenter.org/