kakao·Feb 8, 2026·4 min readCurated summary

In Search of Lost Reports: Kakao

KIMS, Kakao’s internal SMS platform, experienced rare cases where vendors sent delivery reports successfully, yet messages remained stuck in `SENT` instead of becoming `REPORTED`. The cause was a race condition: a fast vendor’s report arrived before the API server had committed the message record. The investigation showed that an unnecessarily long transaction—especially for paid messages with billing-event processing—delayed persistence and allowed valid reports to be dropped. ## KIMS Message Processing Flow - KIMS processes roughly one million SMS messages per day across multiple IDC environments and external vendors. - The normal flow is: - Route the request to a suitable vendor. - Call the vendor and record the message as `SENT`. - Deliver the message to the recipient. - Receive the vendor’s delivery report. - Update the message to `REPORTED`. - These stages run asynchronously across separate services, so their execution order is not guaranteed. ## Discovering the Missing Reports - Some messages remained in `SENT` even though Report Server logs confirmed that delivery reports had arrived. - The issue affected only about `0.02%` of messages, making it difficult to reproduce in tests or local environments. - Two patterns emerged: - Missing reports were concentrated among messages sent through one particular vendor. - Paid messages were affected more often than free messages. ## The Race Condition - The problematic vendor returned reports unusually quickly: - Other vendors typically took more than one second. - This vendor averaged around 20 ms. - Missing-report cases averaged only about 8 ms. - The API server performed additional processing before committing the message record. - For paid messages, billing-event publication was included in the same `@Transactional` scope, making the transaction longer. - Consequently, the sequence could become: 1. API Server calls the vendor. 2. API Server performs billing-related processing. 3. The vendor delivers the message and immediately sends a report. 4. Report Server receives the report before the message row exists in the database. 5. Report Server treats the report as invalid and drops it. 6. API Server finally commits the message as `SENT`. - The report was not lost at the network or vendor level; it was discarded because the system’s write path had not completed. ## Reducing Transaction Scope - The first fix was to remove nonessential work from the main transaction. - Billing-event publication was moved to asynchronous processing using `@Async` and `@TransactionalEventListener`. - The transaction was reduced to the essential state change and database commit. - This advanced the average commit point by approximately 10 ms and significantly reduced report omissions. - It also avoided a dual-write anti-pattern in which an external Kafka event was published inside a database transaction that could later roll back. ## Reconsidering the Need for a Transaction The incident prompted a broader review of whether the transaction was needed at all. - **Atomicity:** The transaction contained only one database write, with no multi-table or cross-record operation requiring all-or-nothing rollback. - **Read isolation:** Metadata such as vendor quality metrics was updated only every few minutes, and using a slightly stale value was acceptable. The independently read tables did not require a single consistent snapshot. - **Write isolation:** JPA’s dirty checking kept the status change in the persistence context until transaction completion, delaying the actual database write. This delay was precisely what allowed the report to arrive first. The article therefore presents the transaction itself—not the vendor or report receiver—as a source of unnecessary latency and an architectural anti-pattern in this workflow. ## Practical Recommendation Use transactions only when their guarantees are required. Keep critical persistence paths short, move external events and nonessential processing after commit, and critically evaluate whether delayed commit semantics could allow asynchronous consumers to observe a missing record.

Read original(opens in new tab)

JpaApache KafkaDistributed SystemsMysql+4

naver·Jan 22, 2026Original article

Smart Store Center's Zero- (opens in new tab)

Smart Store Center successfully migrated its legacy platform from Oracle to MySQL to overcome performance instability caused by resource contention and to reduce high licensing costs. By implementing a "dual write" strategy, the team achieved a zero-downtime transition while maintaining the ability to roll back immediately without data loss. This technical journey highlights the use of proxy data sources and transaction synchronization to ensure data integrity across disparate database environments. ## Zero-Downtime Migration via Dual Writing * The migration strategy relied on "dual writing," where all Create, Update, and Delete (CUD) operations are performed on both the legacy Oracle and the new MySQL databases. * In the pre-migration phase, Oracle served as the primary source for all traffic while MySQL recorded writes in the background to build a synchronized state. * Once data was fully migrated and verified, the primary traffic was shifted to MySQL, with background writes continuing to Oracle to allow for an instantaneous rollback if performance issues occurred. * This approach decoupled the database switch from application deployment, providing a safety net against critical failures that a simple redeploy could not fix. ## Technical Implementation for JPA * To capture and replicate queries, the team utilized the `datasource-proxy` library, which allowed them to intercept Oracle queries and execute them against a separate MySQL DataSource. * To prevent MySQL write failures from impacting the primary Oracle transactions, writes to the secondary database were managed using `TransactionSynchronizationManager`. * By executing MySQL queries during the `afterCommit` phase, the team ensured that the primary service remained stable even if the secondary database encountered errors or performance bottlenecks. * The transition required modifying JPA Entity configurations, such as changing primary key generation from Oracle Sequences to MySQL’s `IDENTITY` (auto-increment) and adjusting `columnDefinition` for types like `text`, `longtext`, and `decimal`. ## Centralized MyBatis Strategy * To avoid modifying thousands of business logic points in a 10-year-old codebase, the team sought a way to implement dual writing for MyBatis at the architectural level. * The implementation focused on the MyBatis `Configuration` and `MappedStatement` objects to capture SQL execution without requiring manual updates to individual repository interfaces. * This centralized approach maintained the purity of the business logic and ensured that the dual-write logic could be easily removed once the migration was fully stabilized. For organizations managing large-scale legacy migrations, the dual-write pattern combined with asynchronous transaction synchronization is a highly recommended safety mechanism. Prioritizing the isolation of secondary database failures ensures that the user experience remains unaffected while technical validation is performed in real-time.

JpaMysqlDatabase MigrationZero Downtime Migration+4