Microsoft Research published a blog post on 30 October describing its Industry Evaluation Data Generation Platform, an internal system for testing Microsoft Copilot on simulated organisations instead of customer data.
The platform rests on three parts: synthetic Copilot environments, automated query-generation pipelines and structured quality checks. Each simulated workspace, which Microsoft calls a tenant, is an isolated environment with its own users, meetings, email threads, documents and reporting structure, and it evolves over time so that questions such as what a team decided last Thursday have a real history behind them.
The environments are grounded in public sources: O*NET occupational data from the U.S. Department of Labor for roles and work activities, online job postings for titles and hierarchies, and SEC filings for organisational scale and business-unit structure. Microsoft says its legal and policy organisation reviewed those sources. Three industry environments were built: pharmaceutical regulatory and trial workflows, capital markets deal and compliance work, and an IT/software company with sprint planning and code review.
Queries come from a six-stage pipeline that narrows generic workplace tasks to an industry, adapts them with domain language, generates questions grounded in the synthetic tenant, rewrites them in more natural phrasing and validates each for accuracy and uniqueness. Quality is scored on structural fidelity, contextual relevance, behavioural realism and content safety, including no impersonation of real people. Microsoft says automation cut dataset creation from several days to repeatable runs and estimates savings of about $100,000 per dataset and $300,000 overall.
Our takeReproducibility is the interesting claim: the same configuration yields the same dataset, so release-over-release comparisons stop drifting. Representativeness is the open question. O*NET roles, job postings and filings describe how organisations are shaped, not how employees phrase requests, and the queries themselves are model-written. Synthetic tenants can show that a change breaks an internal expectation; they cannot show that the expectation matches real usage. A public calibration against logged, consented query distributions would make the baseline far more persuasive.