Optimizing Recommendation Systems with JDK’s Vector API (opens in new tab)
Netflix’s Ranker service used significant CPU for video serendipity scoring, which compares candidate-title embeddings with a member’s viewing history. The team reduced CPU usage by progressively replacing scalar dot products with batched computation, improving memory layout, reusing buffers, and investigating optimized matrix-multiplication libraries. The main lesson was that mathematical optimization alone is insufficient; allocation behavior, cache locality, SIMD support, and runtime overhead all matter. ## The Serendipity Scoring Hotspot - Each candidate title and history item is represented by a vector embedding. - The service computes cosine similarity between every candidate and every history item. - It selects the maximum similarity and converts it into a novelty score: - `serendipity = 1.0 - maxSimilarity` - The original implementation performed `M × N` individual dot products, creating: - Sequential computational work - Repeated embedding lookups - Scattered memory access - Poor cache locality - This logic consumed roughly 7.5% of CPU per Ranker node. - Although 98% of requests contained one video, large batch requests represented about half of the total videos processed. ## Batching Similarity Computations - The team reorganized the calculation as matrix multiplication: - Candidate embeddings form an `M × D` matrix. - History embeddings form an `N × D` matrix. - Rows are normalized to unit length. - Similarities are computed as `C = A × Bᵀ`. - This replaces many separate dot products with one larger operation better suited to CPU-optimized kernels. - The implementation added `batchEncode()` while preserving the existing `encode()` path for single-video requests. ## Why the First Batched Version Regressed - Initial canary tests showed a 5% performance regression. - The batched implementation created `double[][]` arrays for candidates, history, and results on every request. - These allocations: - Increased garbage-collection pressure - Used non-contiguous memory - Added pointer chasing and reduced cache efficiency - The matrix multiplication itself was scalar Java code and did not exploit SIMD hardware. - Batching therefore introduced overhead without delivering corresponding compute gains. ## Flat Buffers and Thread-Local Reuse - The team replaced multidimensional arrays with flat `double[]` buffers in row-major order. - Contiguous storage improved predictability and cache locality. - A `ThreadLocal<BufferHolder>` was used to retain reusable candidate, history, and scratch buffers per thread. - Buffers grow when necessary but do not shrink, avoiding repeated allocations while preventing cross-thread contention. - This reduced GC pressure and made batch performance more stable. ## Evaluating BLAS - BLAS appeared promising in isolated microbenchmarks but did not provide the expected production improvement. - The default `netlib-java` configuration used F2J, a Java implementation rather than truly native BLAS. - Native BLAS introduced setup costs and JNI transition overhead. - Java’s row-major data layout also created an impedance mismatch with common BLAS expectations.