meta·May 12, 2026·3 min readCurated summary

Migrating Data Ingestion Systems at Meta Scale

Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.

Read original(opens in new tab)

Data QualityMachine LearningMysqlData Ingestion+3

toss·Oct 30, 2025Original article

Toss People: Designing a structure (opens in new tab)

Data architecture is evolving from a reactive "cleanup" task into a proactive, end-to-end design process that ensures high data quality from the moment of creation. In fast-paced platform environments, the role of a Data Architect is to bridge the gap between rapid product development and reliable data structures, ultimately creating a foundation that both humans and AI can interpret accurately. By shifting from mere post-processing to foundational governance, organizations can maintain technical agility without sacrificing the integrity of their data assets. **From Post-Processing to End-to-End Governance** * Traditional data management often involves "fixing" or "matching puzzles" at the end of the pipeline after a service has already changed, leading to perpetual technical debt. * Effective data architecture requires a culture where data is treated as a primary design object from its inception, rather than a byproduct of application development. * The transition to an end-to-end governance model ensures that data quality is maintained throughout its entire lifecycle—from initial generation in production systems to final analysis and consumption. **Machine-Understandable Data and Ontologies** * Modern data design must move beyond human-readable metadata to structures that AI can autonomously process and understand. * The implementation of semantic-based standard dictionaries and ontologies reduces the need for "inference" or guessing by either humans or machines. * By explicitly defining the relationships and conceptual meanings of columns and tables, organizations create a high-fidelity environment where AI can provide accurate, context-aware responses without interpretive errors. **Balancing Development Speed with Data Quality** * In high-growth environments, insisting on "perfect" design can hinder competitive speed; therefore, architects must find a middle ground that allows for future extensibility. * Practical strategies include designing for current needs while leaving "logical room" for anticipated changes, ensuring that future cleanup is minimally disruptive. * Instead of enforcing rigid rules, architects should design systems where following the standard is the "path of least resistance," making high-quality data entry easier for developers than the alternative. **The Role of the Modern Data Architect** * The role has shifted from a fixed, corporate function to a dynamic problem-solver who uses structural design to solve business bottlenecks. * A successful architect must act as a mediator, convincing stakeholders that investing in a 5% quality improvement (e.g., moving from 90 to 95 points) provides significant long-term ROI in decision-making and AI reliability. * Aspiring architects should focus on incremental structural improvements, as any data professional who cares about how data functions is already operating on the path to data architecture.

Data QualityData GovernanceData ArchitectureData Modeling+4

figma·Jun 25, 2024·3 min readCurated summary

Ovetta Sampson on Inputs and Outputs | Figma Blog

Minimum viable data is the idea that AI projects should begin with representative, high-quality data—not with the most powerful model or feature. Ovetta Sampson argues that model outputs are overwhelmingly determined by their inputs, which reflect human choices and historical biases. Product builders should therefore question whether AI is necessary, who it serves, and whether the data is equitable enough to avoid harming overlooked groups. ## Data Quality Shapes AI Outcomes - The quality of an AI system depends primarily on the data used to train and operate it. - Decisions about what data to collect, exclude, label, and measure determine who the system recognizes and how it behaves. - Data is never purely objective: it is generated, engineered, and transformed by people. - Treating data as disconnected from human lives can produce “traumatized data sets,” embedding social, cultural, and economic harms into models. ## The Consequences of Omission - Historical datasets often exclude entire groups: - U.S. credit and mortgage models were developed before women could independently obtain mortgages or credit cards. - The U.S. Census did not recognize LGBTQ individuals until 2021, despite those people existing in earlier populations. - When people are absent from the data, models may fail to serve them or may expose them to harmful decisions. - The central question is not simply whether data exists, but whether it represents the people affected by the system. ## Define the Problem Before Choosing AI - Teams should first identify the problem they are trying to solve and determine whether ML or AI is appropriate. - The fact that a problem can be addressed with AI does not mean it should be. - Builders should ask: - Who is the product for? - Is the data equitable and sufficiently high quality? - What is the minimum data and technology needed? - Could the proposed solution increase human risks or reduce people to data points? - Minimum viable data means collecting what is necessary for a useful, responsible solution rather than indiscriminately gathering more data. ## Putting People Back in Control - Product builders and the public need to participate in decisions about how AI systems are designed and governed. - Important questions include who defines “good” data, who decides what enters a training set, and how much data is truly required. - Sampson recommends learning from work such as *Weapons of Math Destruction*, *Ghost Work*, and research on the lack of attention given to data work in AI development. AI development should start with the people affected by a system and the data needed to represent them fairly. Choosing the smallest appropriate dataset and validating its quality can be more responsible—and more effective—than pursuing larger models or unnecessary AI features.

Read original(opens in new tab)

Data QualityAIMachine LearningGenerative AI+2