Optimizing Recommendation Systems with JDK’s Vector API (opens in new tab)

Netflix’s Ranker service used significant CPU for video serendipity scoring, which compares candidate-title embeddings with a member’s viewing history. The team reduced CPU usage by progressively replacing scalar dot products with batched computation, improving memory layout, reusing buffers, and investigating optimized matrix-multiplication libraries. The main lesson was that mathematical optimization alone is insufficient; allocation behavior, cache locality, SIMD support, and runtime overhead all matter.

The Serendipity Scoring Hotspot

• Each candidate title and history item is represented by a vector embedding.
• The service computes cosine similarity between every candidate and every history item.
• It selects the maximum similarity and converts it into a novelty score:
  • serendipity = 1.0 - maxSimilarity
• The original implementation performed M × N individual dot products, creating:
  • Sequential computational work
  • Repeated embedding lookups
  • Scattered memory access
  • Poor cache locality
• This logic consumed roughly 7.5% of CPU per Ranker node.
• Although 98% of requests contained one video, large batch requests represented about half of the total videos processed.

Batching Similarity Computations

• The team reorganized the calculation as matrix multiplication:
  • Candidate embeddings form an M × D matrix.
  • History embeddings form an N × D matrix.
  • Rows are normalized to unit length.
  • Similarities are computed as C = A × Bᵀ.
• This replaces many separate dot products with one larger operation better suited to CPU-optimized kernels.
• The implementation added batchEncode() while preserving the existing encode() path for single-video requests.

Why the First Batched Version Regressed

• Initial canary tests showed a 5% performance regression.
• The batched implementation created double[][] arrays for candidates, history, and results on every request.
• These allocations:
  • Increased garbage-collection pressure
  • Used non-contiguous memory
  • Added pointer chasing and reduced cache efficiency
• The matrix multiplication itself was scalar Java code and did not exploit SIMD hardware.
• Batching therefore introduced overhead without delivering corresponding compute gains.

Flat Buffers and Thread-Local Reuse

• The team replaced multidimensional arrays with flat double[] buffers in row-major order.
• Contiguous storage improved predictability and cache locality.
• A ThreadLocal<BufferHolder> was used to retain reusable candidate, history, and scratch buffers per thread.
• Buffers grow when necessary but do not shrink, avoiding repeated allocations while preventing cross-thread contention.
• This reduced GC pressure and made batch performance more stable.

Evaluating BLAS

• BLAS appeared promising in isolated microbenchmarks but did not provide the expected production improvement.
• The default netlib-java configuration used F2J, a Java implementation rather than truly native BLAS.
• Native BLAS introduced setup costs and JNI transition overhead.
• Java’s row-major data layout also created an impedance mismatch with common BLAS expectations.