TurboQuant: Redefining AI efficiency with extreme compression

TurboQuant is a quantization framework designed to dramatically reduce memory use in large language models and vector search without sacrificing accuracy. It combines PolarQuant’s efficient vector compression with QJL’s one-bit residual correction to eliminate the overhead found in traditional quantization. Experiments show that it can compress KV caches to 3 bits, reduce memory by at least 6×, and accelerate attention-logit computation by up to 8×.

The Memory Challenge in AI

• High-dimensional vectors power language understanding, image features, vector search, and model attention.
• These vectors consume substantial memory, particularly in the key-value (KV) cache used to store frequently accessed attention information.
• Traditional vector quantization reduces vector size but often requires full-precision scaling or normalization constants for each block.
• This metadata can add one or two bits per value, undermining the benefits of compression.

TurboQuant’s Two-Stage Approach

• TurboQuant first applies a random rotation to simplify the geometry of the data.
• PolarQuant then compresses the transformed vectors using a standard quantizer, dedicating most bits to the vector’s primary information.
• A remaining single bit is used by QJL to encode residual error.
• QJL removes bias from the initial compression, improving the accuracy of attention-score calculations.
• The approach requires no model training or fine-tuning.

QJL: One-Bit Error Correction

• QJL builds on the Johnson-Lindenstrauss Transform, which preserves important distances and relationships in high-dimensional data.
• It represents each transformed value using only its sign: +1 or −1.
• A specialized estimator combines low-precision stored data with a high-precision query.
• This preserves accurate attention scores while introducing effectively zero memory overhead.

PolarQuant: Compression Without Metadata Overhead

• PolarQuant converts vectors from Cartesian coordinates into polar coordinates.
• Instead of separately storing coordinate values, it represents vectors through:
  • A radius, capturing magnitude or signal strength
  • Angles, capturing direction and semantic structure
• Because angular values follow a predictable, concentrated distribution, PolarQuant avoids expensive per-block normalization constants.
• It recursively groups coordinate pairs and transforms their radii until the vector becomes one final radius plus a collection of angles.
• This produces a compact representation with fixed, predictable boundaries.

Experimental Results

• The methods were tested on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval using Gemma and Mistral models.
• TurboQuant achieved strong dot-product distortion and recall results while minimizing KV-cache memory.
• On needle-in-a-haystack tasks, TurboQuant maintained perfect downstream performance while reducing KV memory by at least 6×.
• PolarQuant was also nearly lossless on these tasks.
• TurboQuant compressed KV caches to 3 bits without accuracy degradation.
• Quantized models ran faster than the original uncompressed models.
• On H100 GPUs, 4-bit TurboQuant delivered up to an 8× speedup for attention-logit computation compared with 32-bit keys.
• The method has negligible runtime overhead and is relatively simple to implement.

TurboQuant is presented as a practical way to make long-context LLMs and large-scale vector search more memory-efficient. Its combination of metadata-free PolarQuant compression and one-bit QJL correction is especially promising for deployments constrained by KV-cache capacity, latency, or GPU memory.