dropboxJun 25, 2026

How we used DSPy to turn AI evaluations into better responses in Dash chat (opens in new tab)

Dropbox uses DSPy to turn AI evaluations into improvements for its Dash chat agent. The process first calibrates LLM judges against human-labeled conversations, then uses those judges to optimize the agent’s system prompt. This feedback loop reduced incomplete answers and token usage while maintaining answer quality. ## The Complexity of Evaluating AI Agents - Agent quality depends on more than the final response: - Understanding user intent - Selecting relevant context - Choosing and using tools - Synthesizing information across documents, messages, and meetings - Handling ambiguity and follow-up turns - Producing grounded, complete answers - Evaluations therefore inspect the full interaction trajectory, not just the output. - Separate evaluations for intent understanding, tool use, context selection, grounding, adaptation, and task completion help identify the source of failures. - Reliable judges were necessary before evaluation results could safely guide agent improvements. ## Calibrating LLM Judges with Human Labels - Dropbox sampled internal chats containing final answers and agent trace logs. - Human reviewers scored five dimensions: - User-intent following - Semantic relevance - Tool calling - Instruction following - Context selection - Reviewers followed a structured process: - Determine whether the agent understood the request. - Check whether it selected appropriate context. - Inspect searches, retrievals, and other tool actions. - Verify that final claims were supported by evidence. - Score relevance, grounding, completeness, and instruction adherence. - Many metrics used a 1–5 scale. - Reviewers also added: - Reasoning notes explaining their scores - Failure codes for issues such as stale evidence, missing context, unsupported claims, incomplete coverage, and poor personalization - These richer annotations helped improve judge prompts while also supporting debugging, error analysis, roadmap planning, and prioritization. ## Using DSPy to Improve Evaluation - DSPy was used to make LLM judges align more closely with human evaluations. - Judges were required to follow a retrospective workflow: - Infer the user’s intent - Inspect the conversation and agent trace - Review supporting evidence - Assess context selection and tool use - Produce scores, failure codes, and reasoning notes - GEPA and MIPROv2, optimization algorithms within DSPy, automatically proposed and tested prompt changes against human-labeled examples. - Optimization supported several scenarios: - Rewriting judge instructions entirely - Adapting a judge to another underlying model - Targeting specific failure modes while preserving the existing evaluation behavior The overall approach creates a scalable improvement loop: human labels calibrate the judges, calibrated judges provide consistent evaluation signals, and those signals guide improvements to the chat agent itself.

gepallmagentic-aidspy+4

lineJun 23, 2026

From Manual to AI Prompt Tuning: Genetic Algorithm–Based Automated Optimization and Acceleration (opens in new tab)

LY Corporation automated LLM prompt tuning with the GEPA genetic algorithm, reducing a process that previously took days or weeks to roughly one hour. GEPA evolves prompt candidates using evaluation scores and natural-language feedback, allowing it to improve prompts without manually inspecting every output. The approach was applied to Yahoo! JAPAN Search’s AI responses for health and medical queries, balancing policy compliance with improved readability. ## Challenges of Manual Prompt Tuning - Each prompt change requires repeated output generation and human review. - Practical tuning knowledge often remains with individual engineers and is difficult to document or explain. - The cycle of editing, generating, and evaluating responses can take days or weeks. - Model changes and version updates can alter output quality, requiring repeated retuning. - Manual effort leaves less time for defining evaluation criteria, judging quality, and verifying policy compliance. ## Automated Prompt Optimization Approaches - **Reinforcement learning:** Learns prompt-generation policies from scalar rewards, such as with GRPO. - **Bayesian optimization:** Efficiently searches candidate instructions and few-shot examples, as in MIPROv2. - **Genetic algorithms:** Iteratively evolve a population of prompt candidates, as in GEPA. - Genetic methods are well suited to discrete, natural-language prompts because they can use natural-language reflection to identify problems and propose improvements rather than relying only on numerical rewards. ## How GEPA Works - Generates and evaluates multiple prompt candidates. - Uses **Reflective Prompt Mutation** to analyze execution results and create improved instructions. - Applies Pareto-frontier selection to preserve candidates that perform well across multiple evaluation dimensions. - Repeats the process over several to dozens of generations until prompts converge toward the evaluation objectives. - The article notes that GEPA has reportedly outperformed previous optimization methods, including results presented at ICLR 2026. ## Implementation with DSPy - DSPy allows prompt optimization to be controlled programmatically. - A task is defined as a DSPy module with a signature containing input and output fields. - The signature’s docstring becomes an instruction for the LLM. - GEPA rewrites this instruction during optimization. - Separate models can be assigned for: - Task inference - Output evaluation - Reflection and prompt improvement ## Designing the Evaluation Function - GEPA requires an overall scalar score, even when quality is judged across multiple criteria. - Individual scores can be assigned to dimensions such as accuracy, completeness, and style, then normalized and averaged. - The evaluator can also return natural-language feedback through `dspy.Prediction(score=..., feedback=...)`. - Feedback explains why a candidate was penalized, giving GEPA a clearer direction for improvement than a score alone. - Evaluation can use: - LLM-as-a-Judge - Gold answers or labels - Rule-based correctness checks - In the example, an evaluator scores three criteria from 0 to 10, averages them into a single score, and passes the explanation to GEPA for reflection. ## Yahoo! JAPAN Search Health and Medical Queries - Health-related answers must follow stricter policies than general search responses. - Requirements include: - Avoiding definitive medical diagnoses or severity judgments - Matching wording to the strength of available evidence - Recommending medical consultation appropriately - Limiting responses to general explanations where necessary - The project pursued two goals simultaneously: - Satisfy medical and health-policy requirements. - Apply readable Markdown formatting, including headings, lists, and emphasis. - Improving one goal manually could easily damage the other, making automated optimization attractive. ## Applying GEPA to the Production Task - The system takes a search query as input and generates an AI answer. - The initial prompt combined an existing general-purpose prompt with additional health and medical policy instructions. - GEPA rewrote and optimized the instruction section rather than requiring engineers to manually redesign the entire prompt. - The optimization aimed to preserve policy compliance while improving structure and readability. Overall, GEPA with DSPy provides a practical way to shorten prompt-tuning cycles and make the improvement process more reproducible. Its effectiveness depends heavily on carefully designed evaluation criteria and meaningful natural-language feedback, especially for high-risk domains such as medical information.

gepallmllm-as-a-judgeprompt-optimization+3