dropboxJun 25, 2026

How we used DSPy to turn AI evaluations into better responses in Dash chat (opens in new tab)

Dropbox uses DSPy to turn AI evaluations into improvements for its Dash chat agent. The process first calibrates LLM judges against human-labeled conversations, then uses those judges to optimize the agent’s system prompt. This feedback loop reduced incomplete answers and token usage while maintaining answer quality. ## The Complexity of Evaluating AI Agents - Agent quality depends on more than the final response: - Understanding user intent - Selecting relevant context - Choosing and using tools - Synthesizing information across documents, messages, and meetings - Handling ambiguity and follow-up turns - Producing grounded, complete answers - Evaluations therefore inspect the full interaction trajectory, not just the output. - Separate evaluations for intent understanding, tool use, context selection, grounding, adaptation, and task completion help identify the source of failures. - Reliable judges were necessary before evaluation results could safely guide agent improvements. ## Calibrating LLM Judges with Human Labels - Dropbox sampled internal chats containing final answers and agent trace logs. - Human reviewers scored five dimensions: - User-intent following - Semantic relevance - Tool calling - Instruction following - Context selection - Reviewers followed a structured process: - Determine whether the agent understood the request. - Check whether it selected appropriate context. - Inspect searches, retrievals, and other tool actions. - Verify that final claims were supported by evidence. - Score relevance, grounding, completeness, and instruction adherence. - Many metrics used a 1–5 scale. - Reviewers also added: - Reasoning notes explaining their scores - Failure codes for issues such as stale evidence, missing context, unsupported claims, incomplete coverage, and poor personalization - These richer annotations helped improve judge prompts while also supporting debugging, error analysis, roadmap planning, and prioritization. ## Using DSPy to Improve Evaluation - DSPy was used to make LLM judges align more closely with human evaluations. - Judges were required to follow a retrospective workflow: - Infer the user’s intent - Inspect the conversation and agent trace - Review supporting evidence - Assess context selection and tool use - Produce scores, failure codes, and reasoning notes - GEPA and MIPROv2, optimization algorithms within DSPy, automatically proposed and tested prompt changes against human-labeled examples. - Optimization supported several scenarios: - Rewriting judge instructions entirely - Adapting a judge to another underlying model - Targeting specific failure modes while preserving the existing evaluation behavior The overall approach creates a scalable improvement loop: human labels calibrate the judges, calibrated judges provide consistent evaluation signals, and those signals guide improvements to the chat agent itself.

llmagentic-aidspyprompt-optimization+4

dropboxJun 12, 2026

How Dropbox uses MCP and Dash to close the design-to-code security gap (opens in new tab)

Dropbox found a significant gap between security design reviews and implementation. Only 12% of implementing pull requests linked back to their original threat models, and the median delay between review and code submission was about five weeks. To close this gap, Dropbox built a system using Dash, Model Context Protocol (MCP), and foundational models to automatically retrieve relevant security requirements and compare them with code during review. ## The Design-to-Code Gap - Threat models document risks, attack scenarios, and agreed-upon mitigations during security review. - These documents often remain in wikis or documentation systems, while implementation happens later through pull requests. - At Dropbox: - Only 12% of implementing PRs linked to their original design review. - Among 79 verified pairs, 54% of PRs were opened more than a month after the review. - The median delay was approximately five weeks, with some delays exceeding 11 months. - Only 29% of PRs were opened within two weeks of the security review. - About 15% of design reviews were filed retroactively, suggesting that some security-sensitive work was not identified early enough. ## Why Existing Tools Fall Short - Static analysis can detect whether certain security patterns or controls exist. - It generally cannot determine whether those controls satisfy the specific requirements agreed upon during design review. - Manual linking between PRs and threat models depends on engineers remembering additional workflow steps. - Reminder bots may improve compliance temporarily, but adherence tends to decline. - The core problem is not a lack of security documentation; it is the difficulty of making that existing context available during implementation. ## Dash and MCP as a Context Bridge - Dash already indexes Dropbox content and connected applications, including threat models and engineering documentation. - Dropbox used Dash’s MCP server to let an AI security agent search and read this content. - MCP provides a common interface for bringing multiple context sources into one agent session. - When a PR is opened, the agent retrieves relevant threat models and supporting documents. - A foundational model then compares the documented requirements with the proposed code. - For example, it can identify whether code implementing an endpoint enforces authentication required by the corresponding threat model. - Unlike traditional static analysis, this approach evaluates code against documented security intent, not just known code patterns. ## Integrating Security into Code Review - Dropbox surfaced the system directly within the existing code-review workflow rather than creating a separate security process. - This allows developers and reviewers to receive relevant security context where implementation decisions are already evaluated. - The same mechanism can potentially identify missing reviews when code appears security-sensitive but has no corresponding threat model. Dropbox’s approach shows how retrieval and AI reasoning can reconnect design decisions with implementation. Organizations can apply the same pattern beyond security—for example, to compliance requirements, privacy reviews, accessibility standards, or other design-to-code checks.

llmmodel-context-protocolcode-reviewstatic-analysis+2

dropboxFeb 26, 2026

Using LLMs to amplify human labeling and improve Dash search relevance (opens in new tab)

Dropbox Dash improves AI answers through retrieval-augmented generation (RAG): enterprise search retrieves relevant company documents, and an LLM uses a small subset of them to generate grounded responses. Because ranking determines which documents reach the LLM, search relevance depends heavily on high-quality query–document labels. Dash combines a small set of human judgments with large-scale LLM-generated labels to produce training data efficiently while retaining human oversight. ## How Dash search ranking works - Dash uses a trained ranking model, such as XGBoost, rather than manually configured rules. - The model learns from query–document pairs labeled on a 1–5 relevance scale: - **5:** Closely matches the user’s intent. - **1:** Not useful enough to display. - Relevance depends on the query, user context, and timing; it is not an intrinsic property of a document. - Ranking quality is especially important because enterprises may have millions or billions of indexed documents, while only a small selection can be sent to the answer-generating LLM. ## Sources of relevance labels - Labels can come from: - User behavior, such as clicks or skipped results. - Human evaluators assigning relevance scores. - LLMs directly judging query–document relevance. - Behavioral signals are useful but often sparse, biased by existing rankings, and unevenly distributed, so they work best as a supplement. - Human evaluators can provide comprehensive judgments across result sets, but labeling is expensive, difficult to scale, and vulnerable to inconsistency. - Humans also cannot directly review sensitive or proprietary customer data in this process, and different content types—such as Slack messages, Jira tickets, and Salesforce records—require different contextual expertise. ## LLM-assisted relevance evaluation - LLMs can evaluate far larger candidate sets at lower cost and with greater consistency than human annotators. - They can operate across languages and analyze customer content within established compliance boundaries. - Their judgments still depend on the model’s quality and the clarity of the evaluation prompt. - LLM-generated labels therefore require calibration and validation before being used for model training. ## Combining human review with LLM scale - Dropbox first creates a relatively small, high-quality dataset using human evaluators and limited, non-sensitive internal data. - These human labels are used to tune LLM prompts and model parameters. - Once the LLM meets quality thresholds, it generates hundreds of thousands or millions of relevance labels. - This approach multiplies human labeling effort by roughly 100 times, enabling broader and more representative training data. - LLMs are used offline rather than directly at query time because production-time use would introduce excessive latency and context-window limitations. - The LLM acts as a teacher for smaller, faster ranking models that can serve searches at scale. ## Evaluation as the foundation - Dash follows an iterative process: measure performance, change the model or instructions, and measure again. - The article compares this to chess engines, where the quality of the evaluation function determines which possible moves are preserved or discarded. - The same principle applies to ranking: poor relevance judgments can cause useful search-result patterns to be eliminated, while accurate judgments guide the model toward better rankings. Dash’s approach uses humans for quality control and contextual grounding, then uses LLMs to expand that expertise into large-scale training data. This hybrid strategy offers a practical way to improve enterprise search relevance without exposing customer data to human reviewers or imposing LLM latency on every search.

llmmachine-learningragxgboost+3

dropboxFeb 12, 2026

How low-bit inference enables efficient AI (opens in new tab)

Low-bit inference reduces the memory, compute, and energy required to serve modern AI models by representing values with fewer bits. Quantization can substantially increase GPU throughput, but its benefits depend on model accuracy, hardware support, and whether workloads prioritize latency or throughput. The article presents low-bit inference as a production trade-off rather than a universally optimal technique. ## The Rising Cost of Modern Models - Models are growing rapidly, increasing demand for: - Memory capacity - Compute power - Energy - Low-latency serving infrastructure - Dropbox uses attention-based models for Dash and other capabilities involving: - Text, image, video, and audio understanding - Search and summarization - Reasoning over large collections of content - Production deployment requires balancing model capability with hardware utilization, cost, and responsiveness. ## Where Inference Compute Is Spent - Most computation comes from repeated matrix multiplications in two areas: - **Linear layers**, including attention projections, MLP layers, and final output layers. - **Attention mechanisms**, which calculate relationships between input tokens and become increasingly expensive with longer contexts. - GPUs accelerate these operations using specialized hardware: - NVIDIA Tensor Cores - AMD Matrix Cores - These cores execute matrix multiply-accumulate operations much faster than general-purpose CUDA cores. ## How Lower Precision Improves Efficiency - Quantization reduces the number of bits used to represent model values. - Converting values from 16-bit to 8-bit or 4-bit formats: - Reduces memory usage - Lowers memory-transfer costs - Can increase matrix-operation throughput - Reduces energy consumption - GPU throughput generally improves as precision decreases; halving precision can approximately double the number of operations performed per second in suitable workloads. - Eight-bit quantization maps values into 256 discrete levels. Formats below 8 bits typically require **bitpacking**, combining multiple values into types such as `uint8` or `int32` because 4-bit values are not normally stored as native hardware types. - Newer hardware, such as Blackwell GPUs with FP4 support, can provide major energy savings compared with higher-precision systems like the H100. ## Limits of Extremely Low-Bit Formats - Binary and ternary quantization restricts weights to two or three possible values, offering greater theoretical savings. - These formats are not well matched to today’s GPUs because they cannot fully use Tensor or Matrix Cores. - Specialized accelerators could make them more practical, but adoption remains limited by: - Weak ecosystem support - Hardware availability - Concerns about model quality - Practical gains therefore depend not only on bit width, but also on how well the format is supported by existing hardware and software. ## Quantization Formats and Deployment Trade-offs - Quantization is a family of techniques with different choices for: - Numerical representation - Scaling - Execution strategy - These choices affect: - Model accuracy - Inference speed - Memory consumption - Hardware utilization - Different workloads have different priorities: - Latency-sensitive applications need fast individual requests. - Throughput-oriented workloads prioritize processing large volumes efficiently. - Depending on the workload, inference may be limited by software overhead, memory bandwidth, or specialized GPU compute units. ## Pre-MXFP and MXFP Approaches - The article divides modern low-bit formats into two broad groups following the introduction of **MXFP microscaling**: - **Pre-MXFP formats** rely on software-managed scaling and explicit dequantization. - **MXFP formats** move scaling and related operations into Tensor Core hardware. - MXFP aims to standardize low-bit data types while making them more directly usable by modern GPUs. - The choice between these approaches depends on the hardware generation and the specific performance requirements of each production workload. Low-bit inference is most effective when quantization formats, model quality, and hardware capabilities are considered together. Teams should select formats based on the actual bottleneck—memory, bandwidth, latency, or compute—rather than assuming that the fewest possible bits will always deliver the best result.

llmgpu-accelerationmodel-optimizationquantization+3

dropboxOct 22, 2025

Half-Quadratic Quantization of large machine learning models (opens in new tab)

Half-Quadratic Quantization (HQQ) is a calibration-free method for compressing large machine learning models while retaining quality comparable to calibration-based techniques such as GPTQ and AWQ. It minimizes weight reconstruction error rather than activation error and uses a sparsity-promoting \(l_p\) loss to better handle outliers. Because HQQ relies on closed-form alternating updates instead of gradient-based optimization, it can quantize models dramatically faster—reportedly processing Llama-2-70B in under five minutes. ## Why Quantization Matters - Large language models require substantial memory for training and inference. - Methods such as bitsandbytes, GPTQ, and AWQ make models like Llama-2 usable on consumer GPUs. - Weight-only quantization approaches fall into two groups: - **Calibration-free methods**, such as bitsandbytes, use only model weights. - **Calibration-based methods**, such as GPTQ and AWQ, use external datasets. - Calibration-based approaches can provide better quality but: - Their results may depend on calibration-data bias. - Calibration can be computationally expensive for very large models. ## HQQ’s Quantization Objective - Standard quantization can significantly distort weights, particularly outliers with unusually large values. - GPTQ and AWQ reduce the effect of these distortions by minimizing layer-output or activation error using calibration data. - HQQ instead minimizes reconstruction error directly in the weights. - It uses a sparsity-promoting \(l_p\) loss, especially with \(p<1\), to model heavy-tailed outlier errors more effectively than squared error. - Quantization is defined using: - A scale \(s\) - A zero-point \(z\) - A quantization operator \(Q_{z,s}(W)=\text{round}(W/s+z)\) - A dequantization operator \(Q^{-1}_{z,s}(W_q)=s(W_q-z)\) - HQQ fixes the scale and optimizes the zero-point, simplifying the optimization problem. ## Half-Quadratic Optimization - Since the \(l_p\) objective with \(p<1\) is non-convex, HQQ introduces an auxiliary error variable \(W_e\). - The resulting problem is solved through alternating optimization: - Update \(W_e\) while holding \(z\) fixed. - Update \(z\) while holding \(W_e\) fixed. - Increase a positive penalty parameter \(\beta\) by a factor \(\kappa\) each iteration. - This decomposition turns the original difficult problem into simpler sub-problems with closed-form solutions. ## Solving the Sub-Problems - The \(W_e\) update is a proximal operation. - For \(l_1\) regularization, it corresponds to soft thresholding. - HQQ uses a generalized soft-thresholding operator for \(0\leq p\leq1\): \[ \text{shrink}_{l_p}(x,\beta) =\text{sign}(x)\,\text{relu}\left(|x|-\frac{|x|^{p-1}}{\beta}\right) \] - The zero-point update: - Recomputes quantized weights using the current zero-point. - Calculates the difference between quantized weights and corrected original weights. - Sets the new zero-point to the average over the quantization grouping axis. - The implementation optimizes the inverse scale \(1/s\), which is more numerically stable in half-precision arithmetic. ## Speed and Practical Advantages - HQQ uses closed-form updates rather than gradients or automatic differentiation. - Quantization can run in inference mode with half-precision arithmetic. - The solver typically converges in only a few iterations. - In contrast, AdamW with PyTorch autograd may require thousands of iterations and fails when using \(p<1\). - The article reports HQQ as: - More than 100 times faster than autograd for quantizing Llama-2-7B. - More than 50 times faster than GPTQ for Llama-2-70B. - Capable of quantizing the largest models in only a few minutes. - A 2-bit HQQ version of Llama-2-70B reportedly outperforms full-precision Llama-2-13B at a comparable memory footprint. HQQ is therefore presented as a practical alternative to calibration-based quantization: it combines calibration-free operation and very high speed with competitive compression quality, making rapid experimentation and deployment of large models more feasible.

llmmachine-learninginference-optimizationmodel-quantization+4

dropboxOct 2, 2025

A practical blueprint for evaluating conversational AI at scale (opens in new tab)

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.

llmnlpprompt-engineeringrag+4