Profiling improvements in Go 1.18

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped SIGPROF signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog.

More Accurate Linux CPU Profiling

• Earlier Go versions used setitimer(2) to request a SIGPROF signal every 10 ms of CPU time.
• On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue.
• As a result, many signals were dropped:
  • A service using 20 CPU cores might generate roughly 2,000 profiling signals per second.
  • Its Go profile could contain only about 240 samples per second.
• Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage.
• setitimer(2) also distributed process-directed signals unevenly across threads, creating additional profiling bias.

Combining timer_create and setitimer

• timer_create(2) provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems.
• Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code.
• The Go 1.18 fix combined both timer mechanisms:
  • The signal handler identifies the signal source.
  • Signals from inferior sources are discarded.
  • The implementation accounts for short-lived threads and cgo edge cases.
• The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers.

Profiler Label Correctness

• Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines.
• Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata.
• Testing at Datadog revealed that some stack samples were missing labels they should have carried.
• The cause was a CPU profiler lookup using the wrong goroutine reference.
• The fix changed the profiler to use gp.m.curg, the thread’s actual current goroutine, rather than relying on gp, which can differ in certain runtime situations.

Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.