How we optimized Dash's relevance judge with DSPy

Dropbox Dash needed a relevance judge that could score query–document pairs accurately, cheaply, and reliably at scale. Its original judge used OpenAI’s o3, but the cost made it impractical for large-scale labeling, while its prompt performed poorly when moved to the cheaper gpt-oss-120b model. Dropbox used DSPy’s GEPA optimizer to turn prompt tuning into a measurable feedback loop, improving alignment with human judgments while preserving production-ready output formatting.

Measuring Agreement with Human Reviewers

• The judge rates each query–document pair on a 1–5 relevance scale:
  • 5 means a perfect match.
  • 1 means no meaningful connection to the query or user intent.
• Human annotators provide both:
  • A relevance score.
  • A short explanation for their judgment.
• Dropbox evaluates the model with normalized mean squared error (NMSE):
  • It measures the squared difference between model and human ratings.
  • Scores are normalized to a 0–100 scale.
  • 0 represents perfect agreement; higher values indicate worse performance.
• Invalid JSON or incorrectly structured responses are treated as fully incorrect because they cannot be consumed reliably by downstream systems.
• The optimization objective is therefore twofold:
  • Minimize disagreement with human ratings.
  • Ensure consistently parseable, production-ready outputs.

Moving from o3 to a Lower-Cost Model

• The original judge used OpenAI’s o3 because it delivered strong agreement with human ratings.
• Running o3 across orders of magnitude more query–document pairs was too expensive.
• Dropbox selected gpt-oss-120b, an open-weight model offering a better cost-performance balance.
• The carefully tuned o3 prompt did not transfer directly:
  • Relevance quality declined under the NMSE metric.
  • Manual prompt rewriting would have required extensive iteration and regression testing.

DSPy and GEPA-Based Prompt Optimization

• Dropbox defined the optimization problem using:
  • A fixed relevance-rating task.
  • Human-annotated examples.
  • NMSE as the evaluation metric.
• DSPy’s GEPA optimizer iteratively improves prompts for a specific target model.
• Instead of relying only on an aggregate score, GEPA analyzes individual disagreements and generates structured feedback.
• Feedback combines:
  • The difference and direction between predicted and human ratings.
  • The human annotator’s explanation.
  • The model’s reasoning.
• DSPy then uses a reflection loop:
  • Evaluate the current prompt.
  • Identify recurring failure modes.
  • Revise the prompt with generalizable rules.
  • Repeat the process against the human-alignment metric.
• This approach can address systematic errors such as:
  • Overvaluing keyword overlap.
  • Undervaluing document recency.
  • Misinterpreting user intent.
• The feedback explicitly discourages overfitting to individual examples and preserves core task constraints, including the 1–5 rating range.

Dropbox’s experience suggests that relevance judges should be optimized systematically rather than tuned manually. Defining a clear human-alignment metric, including structural validity, allows DSPy to adapt prompts across models while reducing cost and limiting regressions.