Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug

Postgres was crashing with segmentation faults when executing certain expensive queries on an Arm64 Kubernetes cluster. Investigators reduced the failure to a simple table scan and discovered that disabling JIT compilation prevented the crash. Assembly-level debugging ultimately traced the problem to a bug in LLVM’s Arm64 JIT support.

Isolating the Crash

• The failures occurred across multiple EC2 nodes, ruling out faulty hardware.
• Query logs showed that the crashes consistently followed a small number of query patterns.
• The simplest reproducer was:

  SELECT repo_id FROM repository;
  

• Core dumps had badly corrupted stacks, but surviving frames pointed to ExecRunCompiledExpr, suggesting a failure during JIT execution.
• The unusually short backtraces reinforced the suspicion that the stack itself had been corrupted.

How PostgreSQL JIT Works

• PostgreSQL normally evaluates SQL expressions through a general-purpose interpreter.
• JIT compilation converts expressions such as 1+1 into native machine code, reducing interpreter overhead for large workloads.
• JIT can also optimize tuple deforming by converting disk tuples into in-memory values more efficiently.
• PostgreSQL uses LLVM to generate the compiled code.
• Because compilation adds overhead and compiled functions are not reused between queries, PostgreSQL enables JIT primarily for expensive queries based on cost thresholds.

The Query of Death

• The affected query scanned a partitioned repository table with:
  • 64 partitions
  • More than 1.6 million rows
  • 128 JIT-generated functions
• Its query plan enabled expression compilation and tuple deforming:

  JIT:
    Functions: 128
    Expressions: true
    Deforming: true
    Inlining: false
    Optimization: false
  

• Running the query with:

  SET jit = off;
  

  completed successfully.
• Disabling JIT cluster-wide immediately stopped the crashes without noticeable query-latency effects.

Root Cause Direction

• The release build of PostgreSQL offered limited debugging flexibility, so the team planned to reproduce the failure in a dedicated test environment.
• Further investigation eventually isolated the defect to JIT compilation on Arm64 systems.
• The underlying issue was identified as an LLVM bug rather than a PostgreSQL query or hardware problem.
• The investigation continued down to generated assembly and resulted in an upstream fix.

The immediate mitigation was to disable PostgreSQL JIT, while the durable solution was to adopt the LLVM fix addressing the Arm64 code-generation bug.