Inference Optimization

dropboxOct 23, 20253 min readCurated summary

With Mobius Labs' Aana models, we're bringing deeper multimodal understanding to Dropbox Dash

Dropbox is integrating Mobius Labs’ Aana multimodal models into Dropbox Dash to improve how it understands text, images, audio, and video together. Aana is designed for efficient, large-scale processing, enabling Dash to interpret scenes, actions, speech, and relationships across media rather than relying on isolated metadata. The goal is to make rich media searchable and support future AI agents that can analyze content and act on it. ## Dropbox Dash’s Multimodal Goal - Dash combines AI assistance and search to understand a team’s work, context, and tools. - Its multimodal capabilities are intended to cover formats such as: - Text - Photos - Videos - Audio - This is especially valuable for creative, technical, media, and marketing teams whose work is distributed across many applications and content repositories. ## Why Multimedia Understanding Is Difficult - A video may require analysis of: - Scene changes - Speaker shifts - On-screen text - Objects and actions - Audio cues and dialogue - Images require systems to understand people, locations, events, and visual context. - Each modality has different timelines and semantic boundaries; transcripts, shots, visuals, and audio do not automatically align. - Effective understanding requires connecting modalities—for example, linking what someone says to the diagram they are showing. - Processing this information across exabytes of content would be prohibitively expensive without highly efficient infrastructure. ## How Aana Combines Models and Context - Aana analyzes images, audio, video, and related text as a connected system instead of separate data streams. - It combines fine-tuned, open-source foundation models for: - Speech, including optimized Whisper-based models such as `faster-whisper-large-v3-turbo` - Vision - Language - Transformer and mixture-of-experts architectures support fast inference on standard GPUs. - Aana tracks how scenes evolve over time, including: - Object movement - Actions - Changing layouts - Cross-modal events - Insights are stored in a shared vector space, enabling searches such as “the part where the presenter explains the API flow” rather than requiring users to scrub through a video manually. ## Efficiency at Dropbox Scale - Aana uses low-bit 8-bit and 4-bit inference through its HQQ system to reduce memory and compute requirements. - Gemlite provides custom GPU kernels to accelerate matrix multiplication and attention operations. - The Aana SDK coordinates models, batching, GPU utilization, and production pipelines. - Teams can configure and deploy different combinations of models while experimenting with new workflows. - These optimizations are intended to make analysis of exabytes of media possible with a much smaller compute footprint than conventional approaches. ## Future Applications in Dash - Aana could help users find specific visual elements across large creative archives. - It could summarize years of client meetings into searchable highlights. - Its capabilities may support agentic workflows that: - Analyze multimedia - Surface relevant insights automatically - Generate ideas - Take actions for teams Dropbox’s integration of Aana is intended to make Dash a more context-aware search and AI assistant for rich media. The practical outcome is less time spent organizing and manually reviewing content, and more time spent creating and acting on information.

Read original

Inference Optimization Multimodal AI Transformer Agentic Workflows+4

dropboxOct 22, 20253 min readCurated summary

Half-Quadratic Quantization of large machine learning models

Half-Quadratic Quantization (HQQ) is a calibration-free method for compressing large machine learning models while retaining quality comparable to calibration-based techniques such as GPTQ and AWQ. It minimizes weight reconstruction error rather than activation error and uses a sparsity-promoting \(l_p\) loss to better handle outliers. Because HQQ relies on closed-form alternating updates instead of gradient-based optimization, it can quantize models dramatically faster—reportedly processing Llama-2-70B in under five minutes. ## Why Quantization Matters - Large language models require substantial memory for training and inference. - Methods such as bitsandbytes, GPTQ, and AWQ make models like Llama-2 usable on consumer GPUs. - Weight-only quantization approaches fall into two groups: - **Calibration-free methods**, such as bitsandbytes, use only model weights. - **Calibration-based methods**, such as GPTQ and AWQ, use external datasets. - Calibration-based approaches can provide better quality but: - Their results may depend on calibration-data bias. - Calibration can be computationally expensive for very large models. ## HQQ’s Quantization Objective - Standard quantization can significantly distort weights, particularly outliers with unusually large values. - GPTQ and AWQ reduce the effect of these distortions by minimizing layer-output or activation error using calibration data. - HQQ instead minimizes reconstruction error directly in the weights. - It uses a sparsity-promoting \(l_p\) loss, especially with \(p<1\), to model heavy-tailed outlier errors more effectively than squared error. - Quantization is defined using: - A scale \(s\) - A zero-point \(z\) - A quantization operator \(Q_{z,s}(W)=\text{round}(W/s+z)\) - A dequantization operator \(Q^{-1}_{z,s}(W_q)=s(W_q-z)\) - HQQ fixes the scale and optimizes the zero-point, simplifying the optimization problem. ## Half-Quadratic Optimization - Since the \(l_p\) objective with \(p<1\) is non-convex, HQQ introduces an auxiliary error variable \(W_e\). - The resulting problem is solved through alternating optimization: - Update \(W_e\) while holding \(z\) fixed. - Update \(z\) while holding \(W_e\) fixed. - Increase a positive penalty parameter \(\beta\) by a factor \(\kappa\) each iteration. - This decomposition turns the original difficult problem into simpler sub-problems with closed-form solutions. ## Solving the Sub-Problems - The \(W_e\) update is a proximal operation. - For \(l_1\) regularization, it corresponds to soft thresholding. - HQQ uses a generalized soft-thresholding operator for \(0\leq p\leq1\): \[ \text{shrink}_{l_p}(x,\beta) =\text{sign}(x)\,\text{relu}\left(|x|-\frac{|x|^{p-1}}{\beta}\right) \] - The zero-point update: - Recomputes quantized weights using the current zero-point. - Calculates the difference between quantized weights and corrected original weights. - Sets the new zero-point to the average over the quantization grouping axis. - The implementation optimizes the inverse scale \(1/s\), which is more numerically stable in half-precision arithmetic. ## Speed and Practical Advantages - HQQ uses closed-form updates rather than gradients or automatic differentiation. - Quantization can run in inference mode with half-precision arithmetic. - The solver typically converges in only a few iterations. - In contrast, AdamW with PyTorch autograd may require thousands of iterations and fails when using \(p<1\). - The article reports HQQ as: - More than 100 times faster than autograd for quantizing Llama-2-7B. - More than 50 times faster than GPTQ for Llama-2-70B. - Capable of quantizing the largest models in only a few minutes. - A 2-bit HQQ version of Llama-2-70B reportedly outperforms full-precision Llama-2-13B at a comparable memory footprint. HQQ is therefore presented as a practical alternative to calibration-based quantization: it combines calibration-free operation and very high speed with competitive compression quality, making rapid experimentation and deployment of large models more feasible.

Read original

Inference Optimization Large Language Models Machine Learning Model Quantization+4

googleSep 10, 2025Original article

Speculative cascades — A hybrid approach for smarter, faster LLM inference (opens in new tab)

Speculative cascades represent a hybrid inference method that integrates the cost-efficiency of model cascades with the latency-reducing benefits of speculative decoding. By utilizing a smaller drafter model to generate token sequences that are verified in parallel by a larger expert model, this approach allows for high-speed generation while maintaining flexible quality standards. The result is a system that achieves superior cost-quality trade-offs and higher speed-ups than either traditional cascading or standard speculative decoding alone. ### Limitations of Cascades and Speculative Decoding * **Sequential Bottlenecks in Cascades:** Traditional cascades use a deferral rule to decide if a small model can handle a prompt. If the small model is not confident, the system waits for it to finish before starting the large model from scratch, wasting significant time. * **Strict Matching in Speculative Decoding:** This method requires the large model to verify the small model’s tokens. Even if the small model produces a factually correct and high-quality response, the large model will reject the entire draft if the tokens do not match its own preferred output exactly. * **Trade-off Divergence:** Cascades prioritize reducing computational costs but suffer from latency when deferring, while speculative decoding prioritizes speed but often performs redundant work because it mandates identical output to the larger model. ### The Speculative Cascades Mechanism * **Parallel Verification with Deferral:** Speculative cascades use the parallel processing of speculative decoding but introduce a flexible decision rule. The system can choose to accept the smaller model’s draft even if it differs from the larger model’s prediction, provided it meets a confidence threshold. * **Flexible Token Matching:** Unlike standard speculative decoding, which often relies on strict token-by-token matching, speculative cascades allow for "probabilistic matches" or quality-based acceptance to prevent unnecessary rejections. * **Resource Optimization:** By strategically deferring to the smaller model for certain segments of the generation, the system reduces the total work required from the expensive expert model without losing the speed of parallel execution. ### Empirical Results and Performance * **Model Testing:** The approach was validated using Gemma and T5 models across diverse language tasks, including reasoning, coding, translation, and question answering. * **Superior Trade-offs:** Testing showed that speculative cascades consistently outperformed baselines in cost-quality metrics, providing faster inference without the strict "all-or-nothing" quality constraints of speculative decoding. * **Task Versatility:** The hybrid method proved effective across both creative tasks (like summarization) and factual tasks (like math or coding), where different levels of "correctness" are acceptable. Speculative cascades offer a practical path for scaling LLM deployments by balancing the high cost of large models with the need for low-latency user experiences. Developers looking to optimize inference should consider this hybrid approach to capture the efficiency of small models while retaining the oversight of larger, more capable ones.

Inference Optimization AI Large Language Models Gemma+5