dropbox·Oct 23, 2025·3 min readCurated summary

With Mobius Labs' Aana models, we're bringing deeper multimodal understanding to Dropbox Dash

Dropbox is integrating Mobius Labs’ Aana multimodal models into Dropbox Dash to improve how it understands text, images, audio, and video together. Aana is designed for efficient, large-scale processing, enabling Dash to interpret scenes, actions, speech, and relationships across media rather than relying on isolated metadata. The goal is to make rich media searchable and support future AI agents that can analyze content and act on it. ## Dropbox Dash’s Multimodal Goal - Dash combines AI assistance and search to understand a team’s work, context, and tools. - Its multimodal capabilities are intended to cover formats such as: - Text - Photos - Videos - Audio - This is especially valuable for creative, technical, media, and marketing teams whose work is distributed across many applications and content repositories. ## Why Multimedia Understanding Is Difficult - A video may require analysis of: - Scene changes - Speaker shifts - On-screen text - Objects and actions - Audio cues and dialogue - Images require systems to understand people, locations, events, and visual context. - Each modality has different timelines and semantic boundaries; transcripts, shots, visuals, and audio do not automatically align. - Effective understanding requires connecting modalities—for example, linking what someone says to the diagram they are showing. - Processing this information across exabytes of content would be prohibitively expensive without highly efficient infrastructure. ## How Aana Combines Models and Context - Aana analyzes images, audio, video, and related text as a connected system instead of separate data streams. - It combines fine-tuned, open-source foundation models for: - Speech, including optimized Whisper-based models such as `faster-whisper-large-v3-turbo` - Vision - Language - Transformer and mixture-of-experts architectures support fast inference on standard GPUs. - Aana tracks how scenes evolve over time, including: - Object movement - Actions - Changing layouts - Cross-modal events - Insights are stored in a shared vector space, enabling searches such as “the part where the presenter explains the API flow” rather than requiring users to scrub through a video manually. ## Efficiency at Dropbox Scale - Aana uses low-bit 8-bit and 4-bit inference through its HQQ system to reduce memory and compute requirements. - Gemlite provides custom GPU kernels to accelerate matrix multiplication and attention operations. - The Aana SDK coordinates models, batching, GPU utilization, and production pipelines. - Teams can configure and deploy different combinations of models while experimenting with new workflows. - These optimizations are intended to make analysis of exabytes of media possible with a much smaller compute footprint than conventional approaches. ## Future Applications in Dash - Aana could help users find specific visual elements across large creative archives. - It could summarize years of client meetings into searchable highlights. - Its capabilities may support agentic workflows that: - Analyze multimedia - Surface relevant insights automatically - Generate ideas - Take actions for teams Dropbox’s integration of Aana is intended to make Dash a more context-aware search and AI assistant for rich media. The practical outcome is less time spent organizing and manually reviewing content, and more time spent creating and acting on information.

Read original(opens in new tab)

Multimodal AITransformerAgentic WorkflowsVector Db+4

dropbox·Oct 22, 2025·3 min readCurated summary

Half-Quadratic Quantization of large machine learning models

Half-Quadratic Quantization (HQQ) is a calibration-free method for compressing large machine learning models while retaining quality comparable to calibration-based techniques such as GPTQ and AWQ. It minimizes weight reconstruction error rather than activation error and uses a sparsity-promoting \(l_p\) loss to better handle outliers. Because HQQ relies on closed-form alternating updates instead of gradient-based optimization, it can quantize models dramatically faster—reportedly processing Llama-2-70B in under five minutes. ## Why Quantization Matters - Large language models require substantial memory for training and inference. - Methods such as bitsandbytes, GPTQ, and AWQ make models like Llama-2 usable on consumer GPUs. - Weight-only quantization approaches fall into two groups: - **Calibration-free methods**, such as bitsandbytes, use only model weights. - **Calibration-based methods**, such as GPTQ and AWQ, use external datasets. - Calibration-based approaches can provide better quality but: - Their results may depend on calibration-data bias. - Calibration can be computationally expensive for very large models. ## HQQ’s Quantization Objective - Standard quantization can significantly distort weights, particularly outliers with unusually large values. - GPTQ and AWQ reduce the effect of these distortions by minimizing layer-output or activation error using calibration data. - HQQ instead minimizes reconstruction error directly in the weights. - It uses a sparsity-promoting \(l_p\) loss, especially with \(p<1\), to model heavy-tailed outlier errors more effectively than squared error. - Quantization is defined using: - A scale \(s\) - A zero-point \(z\) - A quantization operator \(Q_{z,s}(W)=\text{round}(W/s+z)\) - A dequantization operator \(Q^{-1}_{z,s}(W_q)=s(W_q-z)\) - HQQ fixes the scale and optimizes the zero-point, simplifying the optimization problem. ## Half-Quadratic Optimization - Since the \(l_p\) objective with \(p<1\) is non-convex, HQQ introduces an auxiliary error variable \(W_e\). - The resulting problem is solved through alternating optimization: - Update \(W_e\) while holding \(z\) fixed. - Update \(z\) while holding \(W_e\) fixed. - Increase a positive penalty parameter \(\beta\) by a factor \(\kappa\) each iteration. - This decomposition turns the original difficult problem into simpler sub-problems with closed-form solutions. ## Solving the Sub-Problems - The \(W_e\) update is a proximal operation. - For \(l_1\) regularization, it corresponds to soft thresholding. - HQQ uses a generalized soft-thresholding operator for \(0\leq p\leq1\): \[ \text{shrink}_{l_p}(x,\beta) =\text{sign}(x)\,\text{relu}\left(|x|-\frac{|x|^{p-1}}{\beta}\right) \] - The zero-point update: - Recomputes quantized weights using the current zero-point. - Calculates the difference between quantized weights and corrected original weights. - Sets the new zero-point to the average over the quantization grouping axis. - The implementation optimizes the inverse scale \(1/s\), which is more numerically stable in half-precision arithmetic. ## Speed and Practical Advantages - HQQ uses closed-form updates rather than gradients or automatic differentiation. - Quantization can run in inference mode with half-precision arithmetic. - The solver typically converges in only a few iterations. - In contrast, AdamW with PyTorch autograd may require thousands of iterations and fails when using \(p<1\). - The article reports HQQ as: - More than 100 times faster than autograd for quantizing Llama-2-7B. - More than 50 times faster than GPTQ for Llama-2-70B. - Capable of quantizing the largest models in only a few minutes. - A 2-bit HQQ version of Llama-2-70B reportedly outperforms full-precision Llama-2-13B at a comparable memory footprint. HQQ is therefore presented as a practical alternative to calibration-based quantization: it combines calibration-free operation and very high speed with competitive compression quality, making rapid experimentation and deployment of large models more feasible.

Read original(opens in new tab)

Large Language ModelsMachine LearningInference OptimizationModel Quantization+4

dropbox·Oct 2, 2025·3 min readCurated summary

A practical blueprint for evaluating conversational AI at scale

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.

Read original(opens in new tab)

Large Language ModelsNatural Language ProcessingPrompt EngineeringRAG+4

dropbox·Aug 27, 2025·2 min readCurated summary

Hack Week 2025: How these engineers liquid-cooled a GPU server

Dropbox engineers used Hack Week 2025 to build a liquid-cooling system for GPU servers, anticipating the rising heat and power demands of AI workloads. Their prototype, assembled from radiators, fans, a pump, reservoir, tubing, manifolds, and sensors, reduced temperatures by 20–30°C during stress tests compared with air cooling. The project also enabled quieter operation and may help Dropbox use less data-center space and energy as GPU requirements increase. ## Building a Custom Liquid-Cooling System - The team built the system from scratch after failing to source a complete commercial setup in time. - It replicated key data-center cooling components: - Radiators and fans for heat dissipation - A pump and reservoir to circulate coolant - Tubing and manifolds - Sensors to verify flow and monitor performance - Engineers tested the cooling loop before connecting expensive GPUs, then integrated it with a GPU server. ## Thermal and Noise Improvements - Liquid cooling lowered CPU and GPU temperatures by approximately 20–30°C during demanding torture tests. - Because the liquid loop handled the primary heat-generating components, the team could: - Remove some fans - Run others at lower speeds - Reduce noise and potentially power consumption - Remaining airflow was directed toward lower-heat components such as DIMMs and network cards. - The team considered using an airflow baffle to cool those components more precisely. ## Preparing for AI-Era Infrastructure - High-end GPUs increasingly consume more power and generate more heat, potentially making air cooling insufficient. - Liquid cooling could allow Dropbox to: - Fit more powerful servers into existing data-center footprints - Avoid spreading servers across additional space - Reduce cooling energy and operating costs - Although vendors do not yet universally require liquid cooling for top-tier GPUs, the engineers expect that requirement to become more common. - Dropbox’s growing focus on AI workloads provided additional motivation to investigate the technology early. ## Hack Week as an Experimentation Platform - The project received funding and organizational support from Dropbox’s infrastructure teams. - Hack Week gave engineers time to explore a long-term infrastructure problem outside their normal work. - The team’s Learn Fast award recognized the project’s emphasis on experimentation and rapid learning. - Working in person helped the engineers exchange ideas, troubleshoot quickly, and collaborate with colleagues across the company. Dropbox plans to expand testing with additional liquid-cooling labs in multiple data centers. The prototype is an early step toward infrastructure capable of supporting increasingly powerful, GPU-intensive AI systems.

Read original(opens in new tab)

Ai InfrastructureGpu ComputingLiquid CoolingThermal Management+3