aws·Jun 29, 2026·3 min readCurated summary

AWS Weekly Roundup: Agentic CX designer for Amazon Connect Customer, EC2 AMI Watermarks, Open Governance for MySQL, and more (June 29, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights tools aimed at making AI, infrastructure management, and cloud operations faster and more accessible. The main announcement is Amazon Connect Customer’s no-code Agentic CX designer, which lets business teams create governed AI customer experiences without relying on lengthy engineering backlogs. Other updates cover isolated serverless compute, AMI governance, guided migrations, AI-assisted security investigations, and broader community initiatives. ## Agentic Customer Experience Design - Amazon Connect Customer launched the Agentic CX designer (NLX) in preview. - The no-code canvas enables business teams to design, test, simulate, and deploy voice and digital self-service experiences. - It combines agentic and deterministic AI within a governed workflow. - AWS also introduced Live Sync in preview, allowing web or mobile interfaces to update in real time as customers speak or type. - Customers could, for example, complete forms or open product pages while continuing a voice conversation. ## New AWS Infrastructure and Operations Features - **AWS Lambda MicroVMs** - Provides VM-level isolation with near-instant startup and resume times. - Supports suspending and resuming execution for up to eight hours. - Targets multi-tenant applications running user-generated or AI-generated code. - **Amazon EC2 AMI Watermarks** - Embeds custom identifiers in private AMIs. - Watermarks persist across copies, Regions, and account shares. - Works with Allowed AMIs and Declarative Policies to enforce approved-image usage. - **AWS Outposts lifecycle management** - Adds self-service configuration, quoting, ordering, subscription management, renewal, and decommissioning. - A new quoting tool provides rapid cost estimates and identifies account or regional constraints. ## AI-Assisted Developer and Migration Tools - **Amazon MSK AI Agent Skills** gives coding assistants such as Kiro, Claude Code, and Cursor operational guidance for Amazon MSK. - It supports Kafka sizing, configuration, troubleshooting, monitoring, and migrations to MSK Express. - **Amazon OpenSearch Service Migration Assistant** now offers agent-guided migrations from Solr, Elasticsearch, and OpenSearch to managed clusters or OpenSearch Serverless. - The migration tooling adds live traffic capture and replay for Solr workloads. ## AI-Powered Security Investigations - Amazon GuardDuty’s AI-powered investigations entered preview. - It analyzes findings, account context, related activity from the previous 90 days, knowledge graphs, and threat intelligence. - Investigations produce confidence-scored assessments, MITRE ATT&CK classifications, and recommended actions to help distinguish real threats from benign activity. ## Open Governance and AWS Community Updates - Oracle announced a community governance model for MySQL, including four non-Oracle seats on a new Steering Committee and a public GitHub presence. - AWS supports the initiative and contributes fixes upstream. - AWS Certification holders can renew eligible Associate and Professional certifications for an additional year through selected Skill Builder training and hands-on labs instead of retaking an exam. - The 2026 All Builders Welcome Grant offers selected early-career builders conference admission, airfare, and lodging for AWS re:Invent. AWS’s latest releases broadly point toward more self-service cloud management: business users can design AI experiences, developers can receive operational guidance from coding assistants, and teams can apply stronger controls to infrastructure and security workflows.

Read original(opens in new tab)

MysqlAWSApache KafkaServerless+4

line·Jun 24, 2026·4 min readCurated summary

Flava DBaaS Deep Dive: From Architecture to Migration and Beyond

LY Corporation’s Flava DBaaS is designed to unify the former Verda and YNW cloud platforms on a Kubernetes-based architecture. Its operator pattern separates database business logic from IaaS management, while API servers, managers, and agents divide responsibilities within each DBMS service. The platform expands database support, improves scalability, security, and usability, and treats migration from legacy platforms as part of the DBaaS responsibility. ## Kubernetes Operator-Based Design - Flava DBaaS uses the Kubernetes operator pattern. - Users declare the desired database state through custom resources rather than issuing procedural commands. - Controllers continuously reconcile the actual state with the declared specification. - This approach: - Simplifies troubleshooting through resource status and controller logs. - Handles large database infrastructures efficiently through event-driven processing. - Reuses Kubernetes capabilities for CI/CD and access control. ## Infrastructure Operator Layer - DBaaS must manage IaaS resources such as: - Virtual machines - Storage - Domains and networking - Flava isolates this infrastructure logic in a separate infrastructure operator. - IaaS resources are exposed as Kubernetes custom resources, allowing DBaaS to create infrastructure declaratively without directly calling IaaS APIs. - The resulting layers are: - **DBaaS:** Database-specific business logic - **Infrastructure operator:** Abstraction of IaaS as Kubernetes resources - **IaaS:** Compute, network, and storage services - This separation allows multiple DBMS products to use infrastructure consistently while their developers focus on database operations. ## Custom Resources and DBaaS Components - Each database cluster is represented by a Kubernetes custom resource containing settings such as: - DBMS version - VM size - Storage type and capacity - Replication configuration - These resources are stored in Kubernetes etcd and managed through the Kubernetes API. - Each DBMS implementation consists of three components: - **API server:** Provides REST APIs for creating, modifying, and deleting database resources. Flava UI and IaC tools use these APIs. - **Manager:** Watches resource changes and reconciles the database cluster toward the declared state. - **Agent:** Runs on database VMs and executes local operating-system and database commands. - For example, creating a MySQL cluster causes the API server to create a MySQL custom resource, the manager to provision the required VMs through the infrastructure operator, and the agent to configure replication and database processes inside those VMs. ## Improvements in Flava DBaaS - Flava preserves core DBaaS capabilities such as provisioning, high availability, backup and recovery, scalability, and monitoring. - It combines the DBMS offerings of Verda and YNW, expanding the range of supported database systems. ### Flexible Storage and Scaling - Storage can be configured in 100 GiB increments. - Block-storage-based databases can use up to 5 TiB of storage. - Unlike the legacy platforms, storage is no longer tightly limited by a VM’s local disk capacity. - Custom instance types and separate block storage reduce the need to consider alternatives such as sharding for larger databases. - The 5 TiB limit was selected to cover most analyzed use cases while reducing infrastructure fragmentation. ### Consistent User Experience - All Flava DBaaS products share a common architecture and UI. - Skills learned while changing MySQL server specifications or configuring Cassandra alerts can be applied to other DBMS products. - Users do not need to learn separate operational workflows for each database system. ### Security and Convenience - TDE and TLS are provided as platform-level security features. - Additional features include: - **Custom DB Role:** Reusable database users with configurable permissions. - **Database Parameter Group:** Reusable groups of database configuration parameters. - **Restore backup:** Creation of a new cluster from a selected backup for disaster recovery or realistic performance testing. - Features not yet available for every DBaaS product are planned for broader support. - These improvements reportedly resulted in high internal user-satisfaction scores. ## Migration Responsibilities - A new DBaaS platform is expected to provide migration paths from existing platforms, not merely offer new database clusters. - For migrations between the same DBMS type, the article identifies three general approaches. ### Dump and Restore - Data is backed up from the source database and restored into the destination. - It is the simplest method. - To guarantee consistency, the application generally must be stopped during the migration. ### Replication-Based Migration - The source database is continuously replicated to the destination. - Once replication is caught up, the destination is promoted through failover. - The source database can then be removed. - Data consistency depends on the DBMS’s replication mechanism. - A short application interruption may still occur during primary-node failover. The overall recommendation is to use Flava’s layered, declarative architecture to standardize database operations while continuing to provide practical migration mechanisms from Verda and YNW.

Read original(opens in new tab)

MysqlKubernetesInfrastructure As CodeDatabase Migration+4

meta·May 12, 2026·3 min readCurated summary

Migrating Data Ingestion Systems at Meta Scale

Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.

Read original(opens in new tab)

MysqlMachine LearningData IngestionChange Data Capture+3

toss·Apr 23, 2026·4 min readCurated summary

StarRocks Operations: Isolating Multi-tenant Workloads with Resource Groups

Toss adopted StarRocks as a real-time OLAP engine to consolidate service queries, analytics, validation, and dashboard workloads on one platform. As different workloads began competing within the same clusters, the key operational challenge became deciding which queries to protect during CPU contention. The article describes a gradual strategy: classify workloads, use `cpu_weight` by default, and introduce `exclusive_cpu_cores` only when stronger isolation is required. ## Why StarRocks - Toss previously relied on separate MySQL and Hadoop-based paths for serving, validation, monitoring, and analytics. - StarRocks reduced this duplication by providing: - A MySQL-compatible SQL interface - Large-scale analytical processing - Real-time service-oriented reads - Workloads eventually included: - Advertising and loan-underwriting services - Dashboards and monitoring tools - Kafka Connect ingestion - Batch jobs and backfills - Average traffic varied by cluster: - Service cluster: approximately 69 QPS over 24 hours and 87 QPS over a week - Monitoring and batch cluster: approximately 20 QPS, plus heavier batch workloads - Peak contention between different workloads mattered more than average QPS. ## Workload Classification Toss prioritized workloads in the following order: 1. Service queries 2. Server-side batch jobs 3. Large-scale ingestion and backfills 4. Monitoring and user query tools such as Grafana, Tableau, and Redash - Service queries required strict SLA protection. - Batch jobs needed to finish reliably but did not require real-time responses. - Ingestion and backfills could overwhelm the cluster and therefore needed explicit limits. - Monitoring queries received the lowest priority. ## Using `cpu_weight` for Shared Capacity - `cpu_weight` distributes CPU proportionally when workloads compete. - Higher-weight groups receive more CPU during contention. - When the cluster is idle, all groups can use available CPU regardless of weight. - Toss used this as the default mechanism for multi-tenant workload control. - Example priorities: - `service_wg`: weight 50 - `batch_wg`: weight 10 - `dashboard_wg`: weight 5 - Resource groups could also specify `mem_limit` and `concurrency_limit`. - StarRocks uses a scheduler inspired by Linux CFS, with pipeline drivers yielding in roughly 100 ms time slices. ## Using `exclusive_cpu_cores` for Strong Isolation - `exclusive_cpu_cores` reserves physical CPU cores for a resource group. - StarRocks binds worker threads to those cores using `pthread_setaffinity_np`. - The group receives separate pools for: - `DriverExecutor` - `ScanExecutor` - `ConnectorScanExecutor` - This prevents the protected workload from competing with shared thread pools. - `exclusive_cpu_cores` and `cpu_weight` cannot be used together within the same resource group, although both types can coexist in one cluster. - The setting is limited to `(0, min_be_cpu_cores - 1]`. - Because it is more rigid and consumes dedicated capacity, Toss recommends using it only when relative priority is insufficient. ## Toss Shopping Case - A cluster handled both real-time queries from `shopping_service` and heavy workloads from `commerce_batch`. - Initially, both workloads had similar priority, allowing large batch queries to degrade service latency. - First adjustment: - Increase `shopping_service`’s `cpu_weight` - Lower `commerce_batch`’s weight - This improved prioritization but did not eliminate latency spikes when heavy batch work overlapped with roughly 1,500 service requests per minute. - Second adjustment: - Place `shopping_service` in its own resource group - Assign dedicated CPU cores with `exclusive_cpu_cores` - Afterward, service latency remained stable even during heavy batch execution. - The operational approach was therefore incremental: begin with weights and escalate to dedicated cores only when necessary. ## Classifier Design and Resource Controls - Resource Groups control how resources are allocated; Classifiers determine which queries enter each group. - Classifiers can match attributes such as: - User - Role - Query type - Source IP - Database - The article recommends using stable identifiers such as `user` or `db` for reliable production behavior. - Examples include mapping service `SELECT` queries by service account and assigning server-side batch queries according to their dedicated user. - CPU isolation alone is insufficient for memory-heavy full scans or sudden spikes involving hundreds of concurrent queries, so memory and concurrency limits are also important. Toss’s practical recommendation is to start with clear workload classification and `cpu_weight`, then add memory and concurrency limits. Use `exclusive_cpu_cores` selectively for latency-sensitive workloads whose SLAs cannot be protected through proportional CPU scheduling alone.

Read original(opens in new tab)

MysqlDatabase DesignHadoopStarrocks+4

cloudflare·Apr 16, 2026·2 min readCurated summary

Deploy Postgres and MySQL databases with PlanetScale + Workers

Cloudflare and PlanetScale are integrating more closely so developers can create and manage PlanetScale Postgres and MySQL databases from the Cloudflare dashboard and API. The integration connects these databases to Workers through Hyperdrive, providing connection pooling, query caching, and simplified configuration. Cloudflare billing for new PlanetScale databases is planned for next month, while existing setups remain billed through PlanetScale. ## Postgres and MySQL for Workers - Developers can use either PlanetScale Postgres or Vitess-based MySQL for Worker applications. - Postgres supports a broad ecosystem of tools and extensions such as `pgvector` for AI-oriented vector search. - After connecting a PlanetScale account, users can create databases from the Cloudflare dashboard. - A Hyperdrive binding in `wrangler.jsonc` connects a Worker to the database: ```json { "hyperdrive": [ { "binding": "DATABASE", "id": "<AUTO_CREATED_ID>" } ] } ``` - Workers can then use standard clients such as the Node.js `pg` package and access the connection string through `env.DATABASE`. ## PlanetScale’s Developer Experience - Cloudflare selected PlanetScale for its performance, reliability, and support for both Postgres and MySQL. - PlanetScale features include: - Query insights - Usage and cost breakdowns - Database branching for safer schema and code changes - Agent-assisted SQL performance improvements - Cloudflare users receive the standard PlanetScale experience and pricing, including all available features. - PlanetScale Postgres starts at $5 per month for a single node. ## Reducing Latency with Workers Placement - Workers normally execute close to the incoming user request, which can increase latency when accessing a centralized database. - Developers can configure explicit placement so the Worker runs near the database’s primary region: ```json { "placement": { "region": "aws:us-east-1" } } ``` - Cloudflare plans to automatically determine placement based on the PlanetScale database location, potentially reducing database access latency to single-digit milliseconds. ## Billing and Availability - PlanetScale databases can already be created or connected through the Cloudflare dashboard. - Until the billing integration launches, databases continue to be billed through PlanetScale. - Starting next month, new databases can be billed directly to a Cloudflare self-serve or enterprise account. - Cloudflare credits, startup-program benefits, and committed spend may also apply toward PlanetScale database costs. The integration is intended to give Workers developers a unified platform for globally deployed applications, with flexible SQL storage, optimized database connectivity, and eventually centralized Cloudflare billing.

Read original(opens in new tab)

MysqlDatabase DesignCloudflarePostgreSQL+3

line·Feb 13, 2026·3 min readCurated summary

Slow Query Resolution: Optimizing Bit

LINE VOOM’s post server experienced intermittent timeouts when loading profiles belonging to users with hundreds of thousands of posts. The root cause was bitwise filtering on `category_flag` and `access_flag`, which prevented MySQL from efficiently using indexes and forced scans of all posts for a user. The team resolved the issue with MySQL 8.0.13 functional indexes and by changing the query predicates to exact decimal comparisons, reducing scanned rows from 805 to 31 in testing. ## The Slow Query and Its Root Cause - Post metadata was distributed across shards and partitioned tables. - `category_flag` and `access_flag` were stored as `bit(64)` values containing multiple status flags. - The problematic query filtered by: - `user_id` - `category_flag & 0x0100` - `access_flag & 0x0001` - For heavy users, the query scanned hundreds of thousands of posts and ran for more than 30 seconds. - Bitwise expressions operated on computed results rather than raw column values, preventing normal indexes from filtering efficiently. ## Choosing Functional Indexes - The team considered hardware upgrades, caching, and additional partitioning, but none addressed the root cause adequately. - MySQL 8.0.13 functional indexes could index expression results without changing the table schema. - The proposed composite index was: ```sql ALTER TABLE post_metadata ADD INDEX idx_user_premium_searchable ( user_id, (category_flag & 0x0100), (access_flag & 0x0001) ); ``` - Functional indexes rely on the query expression matching the index definition precisely. ## Discovering the Required Query Form - Initial attempts failed to use the index: - Truthy checks such as `category_flag & 0x0100` - Comparisons using `> 0` - Equality against hexadecimal values such as `= 0x0100` - The successful form used decimal equality: ```sql WHERE user_id = '{user_id}' AND (category_flag & 0x0100) = 256 AND (access_flag & 0x0001) = 1 ``` - In testing, scanned rows dropped from 805 to 31. - Index storage increased by approximately 24%, but the DBA team determined that production capacity was sufficient. ## Rolling Out the Indexes in Production - Indexes were created before changing the application queries. - The team used online schema changes to avoid service downtime and support pausing or rollback during replication problems. - Because dozens of tables across multiple shards were affected: - One shard was handled first for validation. - Only one or two tables were processed per day. - Work was avoided during periods when emergency DBA support was unavailable. - Index creation increased replication lag, causing newly created posts to temporarily disappear from read replicas. - The team reduced the cache expiration time for the affected post lists and accepted the remaining replication delay before resuming the rollout. ## Gradual Query Deployment and a Bitwise Logic Bug - Query changes were deployed gradually through a dynamic configuration system. - Each query pattern was tested on one shard before being expanded to the remaining shards. - This allowed changes to be rolled back immediately through configuration. - During rollout, a serious visibility bug was found. - The original condition: ```sql category_flag & 0x0110 ``` matched when either `0x0100` or `0x0010` was present, effectively representing an OR condition. - Rewriting it as: ```sql (category_flag & 0x0110) = 272 ``` required both bits to be set, creating an AND condition. - Because production data stored only the premium bit, some profiles returned no content. - The incident highlighted the need to verify the semantic meaning of bit flags before converting bitwise predicates into equality comparisons. ## Practical Recommendation For slow queries involving bit flags, consider functional indexes when using MySQL 8.0.13 or later. Ensure the query expression exactly matches the index definition, validate bitwise logic carefully, and use staged schema and query rollouts with monitoring and fast rollback mechanisms.

Read original(opens in new tab)

MysqlDatabase DesignCachingSharding+4

kakao·Feb 8, 2026·4 min readCurated summary

In Search of Lost Reports: Kakao

KIMS, Kakao’s internal SMS platform, experienced rare cases where vendors sent delivery reports successfully, yet messages remained stuck in `SENT` instead of becoming `REPORTED`. The cause was a race condition: a fast vendor’s report arrived before the API server had committed the message record. The investigation showed that an unnecessarily long transaction—especially for paid messages with billing-event processing—delayed persistence and allowed valid reports to be dropped. ## KIMS Message Processing Flow - KIMS processes roughly one million SMS messages per day across multiple IDC environments and external vendors. - The normal flow is: - Route the request to a suitable vendor. - Call the vendor and record the message as `SENT`. - Deliver the message to the recipient. - Receive the vendor’s delivery report. - Update the message to `REPORTED`. - These stages run asynchronously across separate services, so their execution order is not guaranteed. ## Discovering the Missing Reports - Some messages remained in `SENT` even though Report Server logs confirmed that delivery reports had arrived. - The issue affected only about `0.02%` of messages, making it difficult to reproduce in tests or local environments. - Two patterns emerged: - Missing reports were concentrated among messages sent through one particular vendor. - Paid messages were affected more often than free messages. ## The Race Condition - The problematic vendor returned reports unusually quickly: - Other vendors typically took more than one second. - This vendor averaged around 20 ms. - Missing-report cases averaged only about 8 ms. - The API server performed additional processing before committing the message record. - For paid messages, billing-event publication was included in the same `@Transactional` scope, making the transaction longer. - Consequently, the sequence could become: 1. API Server calls the vendor. 2. API Server performs billing-related processing. 3. The vendor delivers the message and immediately sends a report. 4. Report Server receives the report before the message row exists in the database. 5. Report Server treats the report as invalid and drops it. 6. API Server finally commits the message as `SENT`. - The report was not lost at the network or vendor level; it was discarded because the system’s write path had not completed. ## Reducing Transaction Scope - The first fix was to remove nonessential work from the main transaction. - Billing-event publication was moved to asynchronous processing using `@Async` and `@TransactionalEventListener`. - The transaction was reduced to the essential state change and database commit. - This advanced the average commit point by approximately 10 ms and significantly reduced report omissions. - It also avoided a dual-write anti-pattern in which an external Kafka event was published inside a database transaction that could later roll back. ## Reconsidering the Need for a Transaction The incident prompted a broader review of whether the transaction was needed at all. - **Atomicity:** The transaction contained only one database write, with no multi-table or cross-record operation requiring all-or-nothing rollback. - **Read isolation:** Metadata such as vendor quality metrics was updated only every few minutes, and using a slightly stale value was acceptable. The independently read tables did not require a single consistent snapshot. - **Write isolation:** JPA’s dirty checking kept the status change in the persistence context until transaction completion, delaying the actual database write. This delay was precisely what allowed the report to arrive first. The article therefore presents the transaction itself—not the vendor or report receiver—as a source of unnecessary latency and an architectural anti-pattern in this workflow. ## Practical Recommendation Use transactions only when their guarantees are required. Keep critical persistence paths short, move external events and nonessential processing after commit, and critically evaluate whether delayed commit semantics could allow asynchronous consumers to observe a missing record.

Read original(opens in new tab)

MysqlApache KafkaDistributed SystemsRace Condition+4

pinterest·Feb 5, 2026·3 min readCurated summary

Next Generation DB Ingestion at Pinterest

Pinterest replaced fragmented, batch-oriented database ingestion with a unified Change Data Capture (CDC) framework. The new architecture uses Debezium/TiCDC, Kafka, Flink, Spark, and Iceberg to process only changed records, reducing latency from over 24 hours to minutes while lowering infrastructure costs. It also provides native row-level deletion, scalable operations, and improved compliance. ## Problems with the Legacy System - Batch workflows often delayed updates by more than 24 hours. - Full-table processing was inefficient because many tables changed by less than 5% each day. - Lack of row-level deletion support complicated data compliance. - Multiple independently maintained pipelines created operational complexity and inconsistent data quality. ## Unified CDC-Based Architecture - Supports MySQL, TiDB, and KVStore. - Captures database changes through a generic CDC service and publishes them to Kafka, typically in under one second. - Flink processes events in near real time and stores them in append-only CDC Iceberg tables on S3. - Spark jobs run periodically—often every 15 minutes—to merge recent changes into base Iceberg tables. - A bootstrap pipeline initializes base tables from historical database dumps. - Maintenance jobs handle compaction and snapshot expiration. - The framework is designed for at-least-once processing, petabyte-scale data, thousands of pipelines, and YAML-based configuration. ## CDC Tables and Base Tables - CDC tables act as time-series ledgers containing every change event. - CDC data typically becomes available within five minutes. - Base tables mirror the current state of the source database while retaining historical records. - Base-table latency is generally between 15 minutes and one hour. ## Upserting Changes into Base Tables - Spark first identifies the newest event for each primary key. - Events are ranked by timestamp and GTID, then deduplicated. - Iceberg’s `MERGE INTO` applies the resulting changes: - Deletes matching records when the event represents a deletion. - Updates existing records. - Inserts new records unless the event is a deletion. - The process uses a recent CDC window and a processing watermark to avoid reprocessing unnecessary data. ## Choosing Merge-on-Read - Pinterest standardized on Iceberg’s Merge-on-Read (MOR) strategy. - Copy-on-Write (COW) was rejected for most workloads because: - It requires more computation during writes. - It produces substantially larger replacement files, increasing storage costs. - MOR better balances update performance and storage efficiency for frequent incremental changes. ## Partitioning for Faster Upserts - Large base tables can be partitioned using a hash bucket of the primary key. - For example, `bucket(100, id)` distributes records across 100 partitions. - This allows Spark to process partitions in parallel and reduces the data scanned or rewritten during merges. - Iceberg tables are configured with format version 2, identifier fields, merge-on-read update and delete modes, and target file sizes. ## Small-File Challenge - Bucketing improved parallelism but caused each upsert to generate many small files within partitions. - The article indicates that Pinterest investigated this bottleneck and introduced further optimizations, though the supplied excerpt ends before describing them. Pinterest’s CDC-based design provides a substantially faster and more efficient alternative to full-table batch ingestion. Teams adopting a similar system should combine incremental CDC processing with partitioning, merge-on-read storage, bootstrapping, and ongoing file-maintenance strategies.

Read original(opens in new tab)

MysqlApache KafkaApache SparkApache Iceberg+4

naver·Jan 22, 2026Original article

Smart Store Center's Zero- (opens in new tab)

Smart Store Center successfully migrated its legacy platform from Oracle to MySQL to overcome performance instability caused by resource contention and to reduce high licensing costs. By implementing a "dual write" strategy, the team achieved a zero-downtime transition while maintaining the ability to roll back immediately without data loss. This technical journey highlights the use of proxy data sources and transaction synchronization to ensure data integrity across disparate database environments. ## Zero-Downtime Migration via Dual Writing * The migration strategy relied on "dual writing," where all Create, Update, and Delete (CUD) operations are performed on both the legacy Oracle and the new MySQL databases. * In the pre-migration phase, Oracle served as the primary source for all traffic while MySQL recorded writes in the background to build a synchronized state. * Once data was fully migrated and verified, the primary traffic was shifted to MySQL, with background writes continuing to Oracle to allow for an instantaneous rollback if performance issues occurred. * This approach decoupled the database switch from application deployment, providing a safety net against critical failures that a simple redeploy could not fix. ## Technical Implementation for JPA * To capture and replicate queries, the team utilized the `datasource-proxy` library, which allowed them to intercept Oracle queries and execute them against a separate MySQL DataSource. * To prevent MySQL write failures from impacting the primary Oracle transactions, writes to the secondary database were managed using `TransactionSynchronizationManager`. * By executing MySQL queries during the `afterCommit` phase, the team ensured that the primary service remained stable even if the secondary database encountered errors or performance bottlenecks. * The transition required modifying JPA Entity configurations, such as changing primary key generation from Oracle Sequences to MySQL’s `IDENTITY` (auto-increment) and adjusting `columnDefinition` for types like `text`, `longtext`, and `decimal`. ## Centralized MyBatis Strategy * To avoid modifying thousands of business logic points in a 10-year-old codebase, the team sought a way to implement dual writing for MyBatis at the architectural level. * The implementation focused on the MyBatis `Configuration` and `MappedStatement` objects to capture SQL execution without requiring manual updates to individual repository interfaces. * This centralized approach maintained the purity of the business logic and ensured that the dual-write logic could be easily removed once the migration was fully stabilized. For organizations managing large-scale legacy migrations, the dual-write pattern combined with asynchronous transaction synchronization is a highly recommended safety mechanism. Prioritizing the isolation of secondary database failures ensures that the user experience remains unaffected while technical validation is performed in real-time.

MysqlDatabase MigrationZero Downtime MigrationJpa+4

toss·Nov 30, 2025Original article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.

MysqlDatabase DesignApache KafkaMicroservices+4

naver·Nov 17, 2025Original article

Replacing a DB CDC Replication Tool Handling Tens (opens in new tab)

Naver Pay successfully transitioned its core database replication system from a legacy tool to "ergate," a high-performance CDC (Change Data Capture) solution built on Apache Flink and Spring. This strategic overhaul was designed to improve maintainability for backend developers while resolving rigid schema dependencies that previously caused operational bottlenecks. By leveraging a modern stream-processing architecture, the system now manages massive transaction volumes with sub-second latency and enhanced reliability. ### Limitations of the Legacy System * **Maintenance Barriers:** The previous tool, mig-data, was written in pure Java by database core specialists, making it difficult for standard backend developers to maintain or extend. * **Strict Schema Dependency:** Developers were forced to follow a rigid DDL execution order (Target DB before Source DB) to avoid replication halts, complicating database operations. * **Blocking Failures:** Because the legacy system prioritized bi-directional data integrity, a single failed record could stall the entire replication pipeline for a specific shard. * **Operational Risk:** Recovery procedures were manual and restricted to a small group of specialized personnel, increasing the time-to-recovery during outages. ### Technical Architecture and Stack * **Apache Flink (LTS 2.0.0):** Selected for its high-availability, low-latency, and native Kafka integration, allowing the team to focus on replication logic rather than infrastructure. * **Kubernetes Session Mode:** Used to manage 12 concurrent jobs (6 replication, 6 verification) through a single Job Manager endpoint for streamlined monitoring and deployment. * **Hybrid Framework Approach:** The team isolated high-speed replication logic within Flink while using Spring (Kotlin) for complex recovery modules to leverage developer familiarity. * **Data Pipeline:** The system captures MySQL binlogs via `nbase-cdc`, publishes them to Kafka, and uses Flink `jdbc-sink` jobs to apply changes to Target DBs (nBase-T and Oracle). ### Three-Tier Operational Model: Replication, Verification, and Recovery * **Real-time Replication:** Processes incoming Kafka records and appends custom metadata columns (`ergate_yn`, `rpc_time`) to track the replication source and original commit time. * **Delayed Verification:** A dedicated "verifier" Flink job consumes the same Kafka topic with a 2-minute delay to check Target DB consistency against the source record. * **Secondary Logic:** To prevent false positives from rapid updates, the verifier performs a live re-query of the Source DB if a mismatch is initially detected. * **Multi-Stage Recovery:** * **Automatic Short-term:** Retries transient failures after 5 minutes. * **Automatic Long-term:** Uses batch processes to resolve persistent discrepancies. * **Manual:** Provides an admin interface for developers to trigger targeted reconciliations via API. ### Improvements in Schema Management and Performance * **DDL Independence:** By implementing query and schema caching, ergate allows Source and Target tables to be updated in any order without halting the pipeline. * **Performance Scaling:** The new system is designed to handle 10x the current peak QPS, ensuring stability even during high-traffic events like major sales or promotions. * **Metadata Tracking:** The inclusion of specific replication identifiers allows for clear distinction between automated replication and manual force-sync actions during troubleshooting. The ergate project demonstrates that a hybrid architecture—combining the high-throughput processing of Apache Flink with the robust logic handling of Spring—is highly effective for mission-critical financial systems. Organizations managing large-scale data replication should consider decoupling complex recovery logic from the main processing stream to ensure both performance and developer productivity.

MysqlKubernetesApache KafkaKotlin+4

line·Aug 1, 2025Original article

Replacing the Payment System DB Handling (opens in new tab)

The LINE Billing Platform successfully migrated its large-scale payment database from Nbase-T to Vitess to handle high-traffic global transactions. While initially exploring gRPC for its performance reputation, the team transitioned to the MySQL protocol to ensure stability and reduce CPU overhead within their Java-based environment. This implementation demonstrates how Vitess can manage complex sharding requirements while maintaining high availability through automated recovery tools. ### Protocol Selection and Implementation - The team initially attempted to use the gRPC protocol but encountered `http2: frame too large` errors and significant CPU overhead during performance testing. - Manual mapping of query results to Java objects proved cumbersome with the Vitess gRPC client, leading to a shift toward the more mature and recommended MySQL protocol. - Using the MySQL protocol allowed the team to leverage standard database drivers while benefiting from Vitess's routing capabilities via VTGate. ### Keyspace Architecture and Data Routing - The system utilizes a dual-keyspace strategy: a "Global Keyspace" for unsharded metadata and a "Service Keyspace" for sharded transaction data. - The Global Keyspace manages sharding keys using a "sequence" table type to ensure unique, auto-incrementing identifiers across the platform. - The Service Keyspace is partitioned into $N$ shards using a hash-based Vindex, which distributes coin balances and transaction history. - VTGate automatically routes queries to the correct shard by analyzing the sharding key in the `WHERE` clause or `INSERT` statement, minimizing cross-shard overhead. ### MySQL Compatibility and Transaction Logic - Vitess maintains `REPEATABLE READ` isolation for single-shard transactions, while multi-shard transactions default to `READ COMMITTED`. - Advanced features like Two-Phase Commit (2PC) are available for handling distributed transactions across multiple shards. - Query execution plans are analyzed using `VEXPLAIN` and `VTEXPLAIN`, often managed through the VTAdmin web interface for better visibility. - Certain limitations apply, such as temporary tables only being supported in unsharded keyspaces and specific unsupported SQL cases documented in the Vitess core. ### Automated Operations and Monitoring - The team employs VTOrc (based on Orchestrator) to automatically detect and repair database failures, such as unreachable primaries or replication stops. - Monitoring is centralized via Prometheus, which scrapes metrics from VTOrc, VTGate, and VTTablet components at dedicated ports (e.g., 16000). - Real-time alerts are routed through Slack and email, using `tablet_alias` to specifically identify which MySQL node or VTTablet is experiencing issues. - A web-based recovery dashboard provides a history of automated fixes, allowing operators to track the health of the cluster over time. For organizations migrating high-traffic legacy systems to a cloud-native sharding solution, prioritizing the MySQL protocol over gRPC is recommended for better compatibility with existing application frameworks and reduced operational complexity.

MysqlDatabase DesignGrpcDatabase Migration+4

line·Jun 18, 2025Original article

Replacing a Payment System Database That Processes (opens in new tab)

The LINE Billing Platform team recently migrated its core payment database from Nbase-T to Vitess to address rising licensing costs while maintaining the high availability required for financial transactions. After a rigorous Proof of Concept (PoC) evaluating Apache ShardingSphere, TiDB, and Vitess, the team selected Vitess for its mature sharding capabilities and its ability to provide a stable, scalable environment on bare-metal infrastructure. This migration ensures the platform can handle large-scale traffic efficiently without the financial burden of proprietary license fees. ### Evaluation of Alternative Sharding Solutions Before settling on Vitess, the team analyzed other prominent distributed database technologies to determine their fit for a high-stakes payment system: * **Apache ShardingSphere:** While it offers flexible Proxy and JDBC layers, it was excluded because it requires significant manual effort for data resharding and rebalancing. The management overhead for implementing shard-key logic across various components (API, batch, admin) was deemed too high. * **TiDB:** This MySQL-compatible distributed database uses a decoupled architecture consisting of TiDB (SQL layer), PD (metadata management), and TiKV (row-based storage). Its primary advantage is automatic rebalancing and the lack of a required shard key, which significantly reduces DBA operational costs. * **Nbase-T:** The legacy system provided the highest performance efficiency per resource unit; however, the shift from a free to a paid licensing model necessitated the move to an open-source alternative. ### Vitess Architecture and Core Components Vitess was chosen for its proven track record at companies like YouTube and GitHub, offering a robust abstraction layer that makes a clustered database appear as a single instance to the application. The system relies on several specialized components: * **VTGate:** A proxy server that routes queries to the correct VTTablet, manages distributed transactions, and hides the physical topology of the database from the application. * **VTTablet:** A sidecar process running alongside each MySQL instance that manages query execution, data replication, and connection pooling. * **VTorc and Topology Server:** High availability is managed by VTorc (an automated failover tool), while metadata regarding shard locations and node status is synchronized via a topology server using ZooKeeper or etcd. ### PoC Performance and Environment Setup The team conducted performance testing by simulating real payment API scenarios (a mix of reads and writes) on standardized hardware (8vCPU, 16GB RAM). * **Comparison Metrics:** The tests focused on Transactions Per Second (TPS) and resource utilization as thread counts increased. * **Infrastructure Strategy:** Because payment systems cannot tolerate even brief failover delays, the team opted for a bare-metal deployment rather than a containerized one to ensure maximum stability and performance. * **Resource Efficiency:** While Nbase-T showed the best raw efficiency, Vitess demonstrated the necessary scalability and management features required to replace the legacy system effectively within the new cost constraints. ### Practical Recommendation For organizations managing critical core systems that require horizontal scaling without proprietary lock-in, Vitess is a highly recommended solution. While it requires a deep understanding of its various components (like VTGate and VTTablet) and careful configuration of its topology server, the trade-off is a mature, cloud-native-ready architecture that supports massive scale and automated failover on both bare-metal and cloud environments.

MysqlDatabase MigrationDatabase ShardingHigh Availability+4

datadog·May 21, 2024·3 min readCurated summary

Engineering VP spotlight: Ivo Dimitrov

Ivo Dimitrov’s career evolved from low-level systems programming into engineering leadership focused on large-scale distributed storage. His experience at Microsoft and LinkedIn shaped his approach to building scalable data platforms, while Datadog attracted him with its talented people, modern technology, and culture of experimentation. Today, he leads Datadog’s Distributed Data Systems organization, supporting the company’s metrics, events, query, alerting, and analytics infrastructure. ## From Electrical Engineering to Systems Programming - Dimitrov initially studied electrical engineering and became interested in software while working on digital control systems. - His early work included contributing to a real-time operating system kernel. - He spent roughly a decade as an individual contributor, specializing in: - High-performance systems - Low-level programming - C and C++ - System software ## Transition from Individual Contributor to Manager - At Microsoft, Dimitrov worked on an early version of Azure Blob Storage. - Following a reorganization, he accepted an opportunity to lead his team despite having no prior management experience. - Microsoft supported the transition through: - Leadership mentorship - Formal management training - Guidance on communication, conflict resolution, and interpersonal leadership - He discovered that management allowed him to expand his ownership beyond individual projects and influence broader organizational outcomes. - The role combined his technical background with responsibilities such as cross-functional coordination, team development, and engineering strategy. ## Building Internet-Scale Storage at Microsoft and LinkedIn - At Microsoft, Dimitrov worked on storage systems supporting Hotmail. - After joining LinkedIn in 2014, he adapted to a technology environment centered on open source tools such as MySQL and Java. - He led development of Espresso, LinkedIn’s proprietary key-value storage platform. - The platform matured into a core system supporting approximately 95 percent of LinkedIn’s data sets. - He also helped oversee several other large-scale storage projects: - **Venice**, an open source platform for serving derived data - **Ambry**, an open source blob storage system - **Helix**, an open source cluster manager - These systems supported critical parts of LinkedIn’s internet-scale infrastructure. ## Why Datadog Was Appealing - Dimitrov was drawn to Datadog by three main factors: - Highly capable engineers and leaders - Interesting, modern technology - The opportunity to contribute to a rapidly growing company - Compared with the legacy systems and processes that had accumulated at LinkedIn, Datadog offered less bureaucracy and more freedom to: - Take thoughtful risks - Experiment - Deliver quickly - Fail fast and learn - Iterate and innovate - He was particularly interested in Datadog’s Kubernetes-based Metrics and Events platforms and the challenge of building best-in-class infrastructure during the company’s growth. ## Distributed Data Systems at Datadog - Dimitrov leads the Distributed Data Systems organization, which owns a portfolio of storage and data technologies. - Its responsibilities include: - **Metrics**, supporting metrics and time-series data - **Events**, handling semi-structured data such as logs, profiles, and traces - **Driveline**, a main-memory database optimized for online analytics - The **Cross-Platform Queries** team provides a unified query interface across systems that historically exposed separate, domain-specific APIs. - This reduces the learning curve for engineers and customers. - It abstracts the underlying data stores behind a common API. - The organization also operates Datadog’s Alerts platform, which generates a large share of the queries sent to the Metrics and Events systems. Dimitrov’s experience demonstrates how deep systems expertise can translate into effective engineering leadership. His recommendation by example is to remain technically engaged while expanding one’s scope—from writing individual components to shaping teams, platforms, and long-term engineering direction.

Read original(opens in new tab)

MysqlDatabase DesignKubernetesPostgreSQL+4