Designing synthetic datasets for the real world: Mechanism design and reasoning from first principles
Synthetic data generation should be treated as dataset-level mechanism design rather than one-sample-at-a-time prompting. Google’s Simula framework uses reasoning models to control coverage, diversity, complexity, and quality independently, enabling seedless, reproducible datasets for scarce or privacy-sensitive domains. Experiments across five domains show that thoughtfully designed data can outperform larger datasets, but the best configuration depends on the target task and model. ## Why Real-World Data Is Insufficient - Specialized AI applications often lack accessible data because domains are uncommon, expensive to label, or privacy-sensitive. - Manually creating datasets is costly, slow, and error-prone. - Real-world datasets are static, limiting rapid iteration and making it difficult to proactively generate safety edge cases. - Synthetic-first workflows can make data programmable, versioned, reproducible, and inspectable. ## Limitations of Existing Synthetic Data Methods - Many approaches depend on manual prompts, evolutionary algorithms, or large amounts of seed data. - These dependencies reduce scalability and explainability. - Generation parameters are often entangled, making it difficult to independently control diversity, difficulty, and correctness. - Most methods optimize individual samples instead of designing the dataset’s overall distribution. ## Simula’s Reasoning-First Design - Simula constructs datasets from first principles using reasoning models rather than opaque generation processes. - The framework is seedless and agentic, allowing improvements as the underlying models become better at reasoning. - It separates generation into independently controllable axes. ### Global Diversification - Reasoning models map a domain into deep, hierarchical taxonomies that serve as sampling scaffolds. - A recursive propose-and-refine process generates candidate categories, then evaluates, merges, and filters them with a critic model. - These taxonomies help datasets cover long-tail concepts instead of concentrating on common examples. - The approach was demonstrated with structures such as a Cyber Threat Intelligence taxonomy. ### Local Diversification - Taxonomy nodes are converted into “meta-prompts” representing scenarios. - Multiple distinct instantiations are generated for each scenario. - This reduces mode collapse—for example, representing SQL injection through varied contexts rather than repeating nearly identical questions. ### Complexification - A configurable portion of scenarios is made more elaborate or difficult. - Complexity can therefore be adjusted without changing the dataset’s semantic coverage. - The appropriate difficulty level depends on the capabilities of the model that will consume the data. ### Quality Checks - A dual-critic loop independently assesses whether outputs and answers are correct. - Independent verification helps reduce sycophancy and improves label reliability. - This enables quality control without requiring human review of every example. ## Reasoning-Based Evaluation - Conventional metrics such as embedding cosine distance offer only broad signals and limited practical guidance. - Simula introduces: - **Taxonomic Coverage**, which measures how thoroughly the conceptual space is represented. - **Calibrated Complexity Scoring**, which uses LLM-based batch comparisons and chess-style Elo ratings to estimate the difficulty of individual examples. - These metrics aim to evaluate diversity and difficulty in ways that better reflect downstream usefulness. ## Results Across Domains - Simula was evaluated using Gemini 2.5 Flash as a teacher and Gemma 3 4B as a student. - The experiments covered cybersecurity, legal reasoning, grade-school mathematics, and multilingual academic knowledge, with datasets reaching 512,000 examples per domain. - The full combination of global coverage, local diversity, and quality critique consistently outperformed simpler baselines. - High complexity improved math reasoning accuracy by 10% in GSM8k but harmed legal reasoning performance, where the teacher model was weaker. - Simula often achieved stronger downstream results with fewer examples, suggesting that data properties matter as much as volume. - The findings show that there is no universally optimal synthetic-data recipe; effective dataset design must be tailored to the domain and target model. Synthetic data is most effective when treated as an engineered system rather than a source of additional volume. Practitioners should separately tune coverage, variation, difficulty, and verification, then validate those choices against the downstream task.
Read original(opens in new tab)