spotify·Apr 6, 2026·3 min readCurated summary

Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4) | Spotify Engineering

Spotify used its Honk background coding agent with Backstage and Fleet Management to automate migrations from two deprecated datasets to new versions. The effort targeted roughly 1,800 downstream pipelines and produced 240 automated pull requests, potentially saving about 10 engineering weeks. The experience showed that agents perform best when repositories follow standardized patterns, prompts contain precise technical context, and automated testing is available. ## The Challenge of Large-Scale Dataset Migrations - Two heavily used datasets needed replacement to support new dimensions and features. - The datasets had approximately 1,800 direct downstream pipelines and affected thousands more indirectly. - Migrations spanned three frameworks: - BigQuery Runner - dbt - Scala-based Scio - Manual migration was estimated to require around 10 engineering weeks within a six-month deadline. ## Using Backstage to Identify Consumers - Backstage’s endpoint lineage pages revealed downstream dataset consumers. - Its Codesearch plugin located relevant repositories across Spotify’s GitHub Enterprise environment. - The Fleetshift plugin used those results to organize and orchestrate repository migrations. - Backstage also provided a centralized view for tracking progress and opening generated pull requests. ## Context Engineering for Honk - Honk needed detailed, self-contained prompts because it could not access external documentation, dataset schemas, MCPs, or custom Claude skills during execution. - Scio was excluded because its flexible, inconsistent implementations made it difficult to describe all migration cases in one reliable prompt. - BigQuery Runner and dbt were more standardized, making them better candidates for automation. - An initial prompt based on a human migration guide was insufficient and caused incorrect assumptions about field mappings. - Explicit mapping tables in the context file significantly improved results. - Prompts also specified cases where fields should not be migrated automatically. - Honk left those fields unchanged. - It added comments linking to human migration guidance for later review. ## Testing and Automated Pull Requests - BigQuery Runner and dbt repositories generally lacked build-time unit tests. - As a result, Honk could not automatically verify and correct its changes, one of its key capabilities. - Downstream teams had to manually test the generated pull requests before merging. - Despite this limitation, the team successfully created 240 automated migration PRs. - Fleetshift’s Backstage interface simplified monitoring, troubleshooting, repository navigation, and communication with owning teams. ## Lessons for Future Agent-Driven Maintenance - Large-scale automation depends on standardizing frameworks and data practices across repositories. - Consistent testing and validation requirements are essential so agents can verify their own changes. - Future Honk functionality will allow agents to gather context from sources such as JIRA tickets and documentation before editing code. - Better context gathering should reduce the need for exhaustive prompt files and improve migration quality. Spotify’s experience suggests that background coding agents can substantially reduce migration toil, but their effectiveness depends on disciplined standardization, explicit migration rules, and strong automated testing.

Read original(opens in new tab)

ScalaDatabase DesignData PipelineBigquery+4

datadog·Sep 30, 2021Original article

How we optimized our Akka application using Datadog’s Continuous Profiler | Datadog (opens in new tab)

Datadog engineers discovered a significant 20–30% CPU overhead in their Akka-based Java applications caused by inefficient thread management within the `ForkJoinPool`. Through continuous profiling, the team found that irregular task flows were forcing the runtime to waste cycles constantly parking and unparking threads. By migrating bursty actors to a dispatcher with a more stable workload, they achieved a major performance gain, illustrating how high-level framework abstractions can mask low-level resource bottlenecks. ### Identifying the Performance Bottleneck * While running A/B tests on a new log-parsing algorithm, the team noticed that expected CPU reductions did not materialize; in some cases, performance actually degraded. * Flame graphs revealed that the application was spending a disproportionate amount of CPU time inside the `ForkJoinPool.scan()` and `Unsafe.park()` methods. * A summary table of CPU usage by thread showed that the "work" pool was only using 1% of the CPU, while the default Akka dispatcher was the primary consumer of resources. * The investigation narrowed the cause down to the `LatencyReportActor`, which handled latency metrics for log events. ### Analyzing the Root Cause of Thread Fluctuations * The `ForkJoinPool` manages worker threads dynamically, calling `Unsafe.park()` to suspend idle threads and `Unsafe.unpark()` to resume them when tasks increase. * The `LatencyReportActor` exhibited an irregular task flow, processing several hundred events in milliseconds and then remaining idle until the next second. * Because the default dispatcher was configured to use a thread pool equal to the number of processor cores (32), the system was waking up 32 threads every second for a tiny burst of work. * This constant cycle of waking and suspending threads created massive CPU overhead through expensive native calls to the operating system's thread scheduler. ### Implementing a Configuration-Based Fix * The solution involved moving the `LatencyReportActor` from the default Akka dispatcher to the main "work" dispatcher. * Because the "work" dispatcher already maintained a consistent flow of log processing tasks, the threads remained active and did not trigger the frequent park/unpark logic. * A single-line configuration change was used to route the actor to the stable dispatcher. * Following the change, the default dispatcher’s thread pool shrank from 32 to 2 threads, and overall service CPU usage dropped by an average of 30%. To maintain optimal performance in applications using `ForkJoinPool` or Akka, developers should monitor the `ForkJoinPool.scan()` method; if it accounts for more than 10–15% of CPU usage, the thread pool is likely unstable. Recommendations for remediation include limiting the number of actor instances, capping the maximum threads in a pool, and utilizing task queues to buffer short spikes. The ultimate goal is to ensure a stable count of active threads and avoid the performance tax of frequent thread state transitions.

ScalaDatadogJavaAkka+4

datadog·Sep 30, 2021·2 min readCurated summary

How we optimized our Akka application using Datadog’s Continuous Profiler

Datadog discovered that an unexpected 20–30% CPU overhead came from Akka’s use of `ForkJoinPool`, not from the log-processing code they initially suspected. Profiling showed that an actor handling intermittent latency metrics repeatedly caused worker threads to park and unpark. Moving that actor to a busier, more stable dispatcher reduced CPU usage by about 30%. ## How Profiling Revealed the Problem - Datadog used Akka to parallelize log-event processing through actors and dispatchers. - An optimization to log parsing produced little improvement, despite reducing parsing CPU time. - Continuous Profiler flame graphs showed increased CPU time in: - `ForkJoinPool.scan()` - `Unsafe.park()` - Thread-level analysis revealed that the default Akka dispatcher—not the expected dedicated work pool—was responsible. - Many of the affected threads were executing a latency-reporting actor. ## Why `ForkJoinPool` Was Consuming CPU - `ForkJoinPool` dynamically manages worker threads: - It creates threads when work increases. - It suspends idle threads with `Unsafe.park()`. - It resumes them with `Unsafe.unpark()`. - It terminates idle workers after a default period. - The latency actor received a few hundred events per second, processed them within milliseconds, and then remained idle until the next batch. - Because the pool allowed up to 32 threads—matching the number of processor cores—it repeatedly activated and suspended many workers. - These frequent parking and unparking operations created short CPU spikes and excessive time in `ForkJoinPool.scan()`. ## The Dispatcher Change - The team moved the latency actor from Akka’s default dispatcher to the main `work-dispatcher`. - The work dispatcher already handled a steadier stream of log-processing tasks, keeping its worker threads active. - This required only a configuration change assigning the actor to `work-dispatcher`. - CPU usage fell by roughly 30% across services. - The default dispatcher also shrank from 32 threads to 2, confirming that unnecessary thread activation was the cause. ## Recommendations - Monitor CPU time spent in `ForkJoinPool.scan()`, especially when it exceeds roughly 10–15%. - Limit the number of Akka actor instances. - Set a suitable maximum thread count for each pool. - Reduce the number of separate thread pools where practical. - Use task queues to absorb frequent, short-lived workload spikes. - Aim to keep the number of active `ForkJoinPool` workers relatively stable and avoid repeated parking and unparking.

Read original(opens in new tab)

ScalaDatabase DesignJavaProfiling+4