Unbiased Java CPU profiling with JFR in JDK 25 (opens in new tab)
Java Flight Recorder (JFR) provides low-overhead, production-safe diagnostics, but its `ExecutionSample` event can produce biased CPU profiles because it samples JVM-observed runnable threads rather than strictly measuring CPU time. For CPU-bound workloads—particularly reactive applications—this may obscure the real hotspots. Modern profilers therefore combine JFR with JVMTI, `SIGPROF`, `AsyncGetCallTrace`, and JVM-internal techniques, while the Java ecosystem works toward a supported CPU-sampling mechanism. ## How Sampling Profilers Work - Continuous profilers repeatedly capture stack traces and aggregate them to reveal recurring behavior. - CPU profilers commonly sample at fixed intervals, such as every 20 milliseconds. - **CPU-time sampling** highlights code actively consuming processor cycles. - **Wall-clock sampling** reveals latency sources, including I/O waits, lock contention, and blocked threads. - Other profilers trigger on events such as allocations, garbage collection, thread parking, or lock contention. - Regardless of the trigger, profilers capture a stack, associate it with an event, and aggregate the results. ## Limitations of JFR’s `ExecutionSample` - JFR is integrated into the JVM and designed for low-overhead, always-on production use. - Its `ExecutionSample` event captures stacks from a rotating subset of runnable threads. - CPU-heavy threads tend to appear more often, but samples are not strictly proportional to actual CPU consumption. - This can lead to incomplete or biased results on CPU-saturated systems. - Reactive applications are a notable example: their scheduling behavior can cause thread CPU usage and hotspots to be underrepresented. ## CPU Sampling with `AsyncGetCallTrace` - JVMTI agents can use operating-system signals such as `SIGPROF` to sample threads according to CPU time. - The signal handler invokes HotSpot’s `AsyncGetCallTrace` to walk Java stacks asynchronously. - This approach avoids safepoint bias and can capture stacks during arbitrary execution states. - Tools such as `async-profiler` use this technique to produce profiles that more closely match actual CPU usage. - The drawback is that `AsyncGetCallTrace` is an unsupported internal JVM API. - Under heavy load, it can occasionally fault, requiring profilers to add extensive safeguards. - Datadog also uses **vmstructs walking**, which reads internal JVM metadata to recover stack and runtime information unavailable through standard APIs. ## The Safety–Accuracy Tradeoff - JFR offers stability, structured runtime telemetry, and low overhead. - `AsyncGetCallTrace` and vmstructs walking offer more accurate CPU sampling. - Relying on JVM internals creates maintenance and reliability risks because those interfaces are not officially stable. - Consequently, modern profilers combine JFR with unsupported sampling mechanisms rather than choosing only one approach. ## Toward a Supported CPU Profiling Event - Datadog, SAP, Amazon, and OpenJDK contributors recognized that this limitation affected the broader profiling ecosystem. - JFR was already the natural foundation for safe, continuous profiling. - The missing capability was a first-class CPU sampling event that could provide accurate CPU-based results without depending on unsupported JVM internals. - Datadog participated in OpenJDK discussions to explain why existing sampling was insufficient and to help improve the platform’s profiling foundation. Ultimately, accurate production CPU profiling requires both JFR’s safety and CPU-time-based sampling. A supported JFR CPU profiling event would remove the ecosystem’s dependence on fragile JVM internals while preserving the low-overhead behavior needed for continuous use.