kakao·Aug 6, 2026·4 min readCurated summary

Experience Building and Operating a Personalized Airflow Testing Environment

Kakao’s data engineering team built AirZone to make Airflow DAG testing faster, easier, and safer across an ecosystem containing thousands of DAGs and multiple Hadoop clusters. Existing approaches required local setup, repeated Git synchronization, file copying, VPN access, or risky testing on production Airflow. AirZone instead creates an isolated, production-like Airflow environment for each pull request, managed through GitHub comments and Kubernetes automation. ## Limitations of Existing Testing Methods - **Local Airflow** - Requires configuring Airflow, Hadoop authentication, connections, and Docker locally. - Has a high initial setup cost and may differ from production. - **Development Airflow** - Requires committing and pushing every code change. - Git submodule updates and DAG parsing introduce long feedback delays. - **Test Airflow with SSH** - Allows files to be copied directly into a container. - Still requires copying files after every edit. - Access to production Hadoop requires connecting to a production VPN. - **Testing on production Airflow** - Heavy test DAGs consume shared scheduler, worker, and node resources. - A resource-intensive test can delay or interrupt unrelated projects. - Per-user isolation is therefore essential. ## AirZone Requirements - Provide an Airflow environment without requiring users to understand Kubernetes or Helm. - Allow code editing through a browser using Jupyter Notebook. - Execute DAGs against Hadoop and authentication mechanisms similar to production. - Create an independent environment for each pull request. - Prevent one user’s tests from affecting other workflows. ## PR-Based, Isolated Architecture - GitHub pull request comments serve as the user interface. - Users can create or delete an environment directly from a PR. - The resulting environment link is posted back to the PR. - Each PR receives a dedicated Kubernetes namespace based on the repository and PR number. - Airflow web server, scheduler, PostgreSQL, Jupyter, DAG volumes, and logs are isolated. - Multiple PRs can be tested simultaneously. - Cleanup is straightforward because the namespace defines the environment boundary. - A dedicated AirZone Helm chart packages the complete test environment. - Production-only components such as PGBouncer and external database connections are omitted where unnecessary. - Airflow, PostgreSQL, DAG storage, Jupyter, authentication, TLS, and logging are deployed together. ## Separating Requests from Deployment - `airzone-api` only validates requests: - Confirms that the PR exists and is open. - Checks branch information. - Prevents duplicate namespaces. - Kubernetes Jobs perform the long-running work: - Install the Helm release. - Run health checks. - Handle creation and deletion independently from the API process. - Job names include the operation and namespace, such as: - `create-airzone-{namespace}` - `delete-airzone-{namespace}` - Failed Jobs can be removed and recreated for retries. - Independent Job logs and status make deployment failures easier to diagnose. - A daily CronJob removes environments that remain after their PRs are closed. ## Building the Airflow Environment Each Helm deployment includes the components needed for a realistic test environment: - **Git integration:** Synchronizes the PR’s head repository and branch. - **DAG PVC:** Lets the scheduler and Jupyter use the same working directory. - **Airflow configuration:** Uses KubernetesExecutor and test-specific DAG scanning, logging, and Hadoop settings. - **Authentication:** Injects user and shared principals, keytabs, Jupyter tokens, and TLS certificates. - **Infrastructure placement:** Selects suitable node groups and a storage class in the same region. - **Centralized logging:** Connects Airflow logs to Elasticsearch and Kibana. - **Hadoop execution:** Existing infrastructure runs Hadoop tasks in dedicated pods using custom Hadoop images, Kerberos initialization, Spark, and Hive. ## Notifications and Security - KakaoWork sends: - An initial notification when a request is received. - A completion notification after deployment. - Operational error alerts. - Sensitive information, including Jupyter and Kubernetes namespace tokens, is not posted in public PR comments. - Tokens are delivered through KakaoWork instead, keeping authentication data separate from the broader PR audience. AirZone’s main recommendation is to make testing a disposable, reproducible environment tied to the pull request itself. By combining per-PR Kubernetes namespaces, Helm-based deployment, asynchronous Jobs, production-like Hadoop access, and automatic cleanup, teams can test DAGs quickly without burdening shared Airflow or production resources.

Read original(opens in new tab)

AirflowKubernetesPostgreSQLApache Spark+3

toss·Apr 16, 2026·5 min readCurated summary

Extending Real-time Ad Frequency Capping Aggregation to One Week with Apache Flink + RocksDB Tuning

The post describes Toss’s expansion of real-time advertising frequency-capping from short Flink windows to periods of up to seven days. The new system provides accurate sliding counts from one minute to seven days through a single Redis lookup, while treating Flink state as the authoritative source and Redis as its projection. The migration addressed architectural complexity, backfill consistency, and distinct RocksDB bottlenecks across three specialized Flink applications. ## Frequency Capping and Its Business Impact - Frequency capping controls how many times an individual user sees an advertisement. - Incorrect counts can: - Waste an advertiser’s budget through excessive exposure. - Prevent valid impressions when the system believes a limit has already been reached. - Different products require different windows, such as: - Three impressions per day. - One impression over the previous seven days. - The target system therefore needed accurate, real-time sliding counts from one minute through seven days. ## Limitations of the Previous Batch-Oriented System The original architecture combined three Airflow-managed layers: - **Head** - Stored current-day and previous-day events in Redis through a Spring Kafka consumer. - Updated counts immediately per event. - **Mid** - Used daily Spark jobs to pre-aggregate data from D-2 through D-7. - **Tail** - Added hourly correction data around the boundary between Head and Mid. - Airflow workflows ran approximately 75 times per day. At serving time, the API could perform up to four Redis lookups and combine the results. - This structure was difficult to maintain because of the dependencies and boundary conditions between Head, Mid, and Tail. - Time-based truncation made precise event-level sliding windows difficult. - The architecture remains useful for longer windows such as 30 days and fixed daily aggregates, especially when data exceeds Kafka retention and must be recovered from batch storage. - Extending the existing short-window Flink system was chosen to simplify serving and reduce DAG complexity. ## Three Flink Applications Rather than place all windows in one Flink job, the team split processing into three applications with shared code but independent RocksDB configurations: - **Minutes** - Handles one- to 30-minute windows. - Frequent event expiration creates heavy write traffic. - Its main concern is RocksDB Write Buffer Manager pressure and resulting Write Stalls. - **Hours** - Handles windows up to 12 hours. - Maintains many more advertisement IDs in state. - Filter Block Cache misses can saturate CPU. - Redis synchronization requires an O(N) scan over advertisement IDs in each window. - Filter Block tuning and additional managed memory are important. - **Days** - Handles the largest state volume. - A seven-day window can produce approximately 68 GB of live SST files and 220–230 GB savepoints. - Checkpoint I/O becomes the primary bottleneck, motivating a Flink Changelog design. Separating the applications allowed each workload’s RocksDB and runtime bottlenecks to be optimized independently without affecting the others. ## Backfill and Catch-up Architecture The most difficult migration problem was maintaining correctness at the transition point between historical data and live processing. - **Backfill** - Loads seven days of historical events. - Only increments counts. - Does not register expiration timers. - Synchronizes the initialized values to Redis once and then finishes. - **Catch-up** - Re-reads historical events from Kafka. - Rebuilds both counts and expiration timers. - Begins writing to Redis after reaching the historical scan end. - Enables each window only after sufficient lookback data has been reconstructed. The two phases cannot safely share one pipeline: - Backfill must only add historical counts. - Live or catch-up processing must both add new events and subtract events that leave the sliding window. - If expiration timers ran while backfill was incomplete, decrements could occur before all historical increments had been applied, producing incorrect results. - Flink batch mode was rejected because state is discarded when the job finishes. - A Spark and Hive-based approach was also rejected because it would introduce additional systems and complicate the single-source-of-truth model. Separate Kafka consumer groups were required so that backfill offsets would not cause catch-up events to be skipped. ## State as the Single Source of Truth - Flink state stores the authoritative aggregate. - Redis is treated only as a serving projection. - If Redis becomes inconsistent, it can be reconstructed from Flink state. - This design preserves correctness during failures, restarts, and Redis resynchronization. ## Maintaining Transition Consistency Three mechanisms were combined to make the backfill-to-catch-up boundary reliable: - **Redis write condition** - Writes are based on each event’s `eventTime` being after the backfill completion point. - Using the global watermark directly could block all writes because one slow or idle partition can hold back the watermark. - **`withIdleness` set to 60 seconds** - Excludes inactive Kafka partitions from watermark progression. - A longer timeout avoids falsely marking a partition idle just before a bounded source emits `MAX_WATERMARK`. - **Timer state TTL** - Must exceed the sliding-window expiration period. - If the timer fires after its associated state has expired, `timerState.get()` returns null and the decrement is skipped. - This would leave counts artificially high after delays or recovery. - The state is manually cleaned up after timer processing. ## RocksDB and Flink Runtime Tuning Once the system was serving real-time results, operational metrics exposed different bottlenecks in each application. - The minutes application initially experienced RocksDB Write Stalls caused by pressure on the shared Write Buffer Manager. - RocksDB first stores writes in MemTables and flushes them into SST files organized across levels L0–L6. - Flink maps managed state types such as `MapState` and `ValueState` to separate RocksDB Column Families. - Because multiple Column Families share the Write Buffer Manager’s memory budget, write-heavy workloads require careful tuning of RocksDB memory and write paths. - The hours and days applications require different optimizations focused on cache misses, CPU usage, checkpoint I/O, and level management. ## Practical Conclusion For real-time frequency capping, a unified Flink-based design can simplify serving and improve sliding-window accuracy, but long windows should not automatically be combined with short ones in a single job. Separate applications, state-as-SSOT, distinct backfill and catch-up pipelines, and workload-specific RocksDB tuning are essential for maintaining correctness and operability at scale.

Read original(opens in new tab)

AirflowDatabase DesignRedisApache Flink+3

toss·Dec 8, 2025Original article

Improving Business Data Literacy: (opens in new tab)

Toss’s Business Data Team addressed the lack of centralized insights into their business customer (BC) base by building a standardized Single Source of Truth (SSOT) data mart and an iterative Monthly BC Report. This initiative successfully unified fragmented data across business units like Shopping, Ads, and Pay, enabling consistent data-driven decision-making and significantly raising the organization's overall data literacy. ## Establishing a Single Source of Truth (SSOT) - Addressed the inefficiency of fragmented data across various departments by integrating disparate datasets into a unified, enterprise-wide data mart. - Standardized the definition of an "active" Business Customer through cross-functional communication and a deep understanding of how revenue and costs are generated in each service domain. - Eliminated communication overhead by ensuring all stakeholders used a single, verified dataset rather than conflicting numbers from different business silos. ## Designing the Monthly BC Report for Actionable Insights - Visualized monthly revenue trends by segmenting customers into specific tiers and categories, such as New, Churn, and Retained, to identify where growth or attrition was occurring. - Implemented Cohort Retention metrics by business unit to measure platform stickiness and help teams understand which services were most effective at retaining business users. - Provided granular Raw Data lists for high-revenue customers showing significant growth or churn, allowing operational teams to identify immediate action points. - Refined reporting metrics through in-depth interviews with Product Owners (POs), Sales Leaders, and Domain Heads to ensure the data addressed real-world business questions. ## Technical Architecture and Validation - Built the core SSOT data mart using Airflow for scalable data orchestration and workflow management. - Leveraged Jenkins to handle the batch processing and deployment of the specific data layers required for the reporting environment. - Integrated Tableau with SQL-based fact aggregations to automate the monthly refresh of charts and dashboards, ensuring the report remains a "living" document. - Conducted "collective intelligence" verification meetings to check metric definitions, units, and visual clarity, ensuring the final report was intuitive for all users. ## Driving Organizational Change and Data Literacy - Sparked a surge in data demand, leading to follow-up projects such as daily real-time tracking, Cross-Domain Activation analysis, and deeper funnel analysis for BC registrations. - Transitioned the organizational culture from passive data consumption to active utilization, with diverse roles—including Strategy Managers and Business Marketers—now using BC data to prove their business impact. - Maintained an iterative approach where the report format evolves every month based on stakeholder feedback, ensuring the data remains relevant to the shifting needs of the business. Establishing a centralized data culture requires more than just technical infrastructure; it requires a commitment to iterative feedback and clear communication. By moving from fragmented silos to a unified reporting standard, data analysts can transform from simple "number providers" into strategic partners who drive company-wide literacy and growth.

AirflowSqlData AnalysisData Pipeline+4