Llm As A Judge | Techlist.io

lineJun 23, 20264 min readCurated summary

From Manual to AI Prompt Tuning: Genetic Algorithm–Based Automated Optimization and Acceleration

LY Corporation automated LLM prompt tuning with the GEPA genetic algorithm, reducing a process that previously took days or weeks to roughly one hour. GEPA evolves prompt candidates using evaluation scores and natural-language feedback, allowing it to improve prompts without manually inspecting every output. The approach was applied to Yahoo! JAPAN Search’s AI responses for health and medical queries, balancing policy compliance with improved readability. ## Challenges of Manual Prompt Tuning - Each prompt change requires repeated output generation and human review. - Practical tuning knowledge often remains with individual engineers and is difficult to document or explain. - The cycle of editing, generating, and evaluating responses can take days or weeks. - Model changes and version updates can alter output quality, requiring repeated retuning. - Manual effort leaves less time for defining evaluation criteria, judging quality, and verifying policy compliance. ## Automated Prompt Optimization Approaches - **Reinforcement learning:** Learns prompt-generation policies from scalar rewards, such as with GRPO. - **Bayesian optimization:** Efficiently searches candidate instructions and few-shot examples, as in MIPROv2. - **Genetic algorithms:** Iteratively evolve a population of prompt candidates, as in GEPA. - Genetic methods are well suited to discrete, natural-language prompts because they can use natural-language reflection to identify problems and propose improvements rather than relying only on numerical rewards. ## How GEPA Works - Generates and evaluates multiple prompt candidates. - Uses **Reflective Prompt Mutation** to analyze execution results and create improved instructions. - Applies Pareto-frontier selection to preserve candidates that perform well across multiple evaluation dimensions. - Repeats the process over several to dozens of generations until prompts converge toward the evaluation objectives. - The article notes that GEPA has reportedly outperformed previous optimization methods, including results presented at ICLR 2026. ## Implementation with DSPy - DSPy allows prompt optimization to be controlled programmatically. - A task is defined as a DSPy module with a signature containing input and output fields. - The signature’s docstring becomes an instruction for the LLM. - GEPA rewrites this instruction during optimization. - Separate models can be assigned for: - Task inference - Output evaluation - Reflection and prompt improvement ## Designing the Evaluation Function - GEPA requires an overall scalar score, even when quality is judged across multiple criteria. - Individual scores can be assigned to dimensions such as accuracy, completeness, and style, then normalized and averaged. - The evaluator can also return natural-language feedback through `dspy.Prediction(score=..., feedback=...)`. - Feedback explains why a candidate was penalized, giving GEPA a clearer direction for improvement than a score alone. - Evaluation can use: - LLM-as-a-Judge - Gold answers or labels - Rule-based correctness checks - In the example, an evaluator scores three criteria from 0 to 10, averages them into a single score, and passes the explanation to GEPA for reflection. ## Yahoo! JAPAN Search Health and Medical Queries - Health-related answers must follow stricter policies than general search responses. - Requirements include: - Avoiding definitive medical diagnoses or severity judgments - Matching wording to the strength of available evidence - Recommending medical consultation appropriately - Limiting responses to general explanations where necessary - The project pursued two goals simultaneously: - Satisfy medical and health-policy requirements. - Apply readable Markdown formatting, including headings, lists, and emphasis. - Improving one goal manually could easily damage the other, making automated optimization attractive. ## Applying GEPA to the Production Task - The system takes a search query as input and generates an AI answer. - The initial prompt combined an existing general-purpose prompt with additional health and medical policy instructions. - GEPA rewrote and optimized the instruction section rather than requiring engineers to manually redesign the entire prompt. - The optimization aimed to preserve policy compliance while improving structure and readability. Overall, GEPA with DSPy provides a practical way to shorten prompt-tuning cycles and make the improvement process more reproducible. Its effectiveness depends heavily on carefully designed evaluation criteria and meaningful natural-language feedback, especially for high-risk domains such as medical information.

Read original

Llm As A Judge Large Language Models Prompt Optimization Dspy+3

netflixApr 10, 20263 min readCurated summary

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix developed an LLM-as-a-Judge system to evaluate show synopses at the scale of its extensive catalog. The system assesses creative quality against expert-defined standards while also examining whether scores predict member behavior. With calibrated prompts, extended reasoning, and consensus scoring, the approach achieves more than 85% agreement with creative writers and can identify potentially impactful synopsis problems before a title launches. ## Defining a Good Synopsis - Synopsis quality is measured in two ways: - **Creative quality:** how well a synopsis follows Netflix’s editorial standards. - **Member feedback:** how the synopsis affects viewing decisions and early engagement. - Strong synopses help members quickly understand and choose titles. - Weak or misleading synopses can cause frustration, abandonment, and reduced viewing. ## Building Expert-Labeled Evaluation Data - Creative experts initially labeled roughly 1,000 diverse synopses. - Three writers scored each synopsis and explained their decisions. - Because the task was subjective, Netflix used eight calibration rounds to improve consistency. - Techniques that increased agreement included: - Replacing 1–4 ratings with binary scores. - Allowing writers to consult previous examples. - Maintaining a searchable taxonomy of recurring errors. - A model-in-the-loop process helped resolve disagreements: - Multiple writers supplied scores. - An LLM aggregated the judgments. - Writers reviewed cases with significant disagreement. - The resulting “golden set” contains about 600 synopses with criterion-level labels and explanations. ## Measuring Member Impact - Netflix uses two behavioral metrics: - **Take fraction:** how often members who see a synopsis start watching the title. - **Abandonment rate:** how often viewers stop shortly after beginning. - These metrics act as short-term proxies for long-term retention and have been validated through A/B testing. - Netflix evaluates whether LLM-generated quality scores can predict these engagement outcomes. ## Criterion-Specific LLM Judges - Initial prompts provide: - Relevant show metadata. - A summary of the applicable quality guidelines. - A request for an explanation followed by a binary score. - A single prompt covering every criterion performed poorly because it overloaded the model. - Separate judges for individual criteria performed better. - Binary outputs make evaluation straightforward using accuracy against the expert-labeled golden set. ## Improving Prompts and Reasoning - Netflix applies Automatic Prompt Optimization to a development set of about 300 examples. - Prompts are then manually refined with LLM assistance. - Performance varies significantly by criterion: prompts work well for areas such as precision but less well for subjective criteria such as clarity. - Inference-time scaling improves difficult judgments through: - **Longer rationales**, which give the model more room to reason. - **Consensus scoring**, which samples multiple judgments and combines their results. ## Tiered Rationales - Longer explanations generally improve accuracy, but they become harder for creative experts to read and audit. - Netflix therefore uses tiered rationales: - The model may reason at length internally. - It produces a concise explanation before the final score. - This approach preserves the benefits of extended reasoning while improving interpretability. - For example, the tone evaluator’s accuracy increased from 86.55% to 87.85% with tiered rationales. Netflix’s approach combines expert standards, calibrated evaluation data, specialized prompts, and inference-time reasoning to scale synopsis-quality review. The practical recommendation is to use LLM judges as carefully aligned evaluators—not generic critics—while validating their scores against both human judgment and real member behavior.

Read original

Llm As A Judge Database Design Automatic Prompt Optimization Creative Quality+4

daangnFeb 27, 20265 min readCurated summary

Things I learned using 2

Karrot’s Taxonomy team built an LLM-powered system to classify marketplace posts, group activities, and local businesses into a shared category and attribute structure. After finding that manually managed taxonomies and event-only pipelines were difficult to scale, they created a configurable Taxonomy Management System using Dataflow/Beam, BigQuery, Kafka, and multiple LLM strategies. The system emphasizes scalable inference, rapid evaluation, multilingual support, and continuous taxonomy expansion. ## What a Taxonomy Is and Why It Matters - A taxonomy is a hierarchical category system, such as `Outerwear > Padding/Down > Long Padding`. - It can also include attributes that describe an item’s characteristics: - Category: long padding - Attributes: brand=Nike, color=black, material=polyester - A consistent taxonomy acts as a shared language across: - Search, including parent and child-category expansion - Recommendations and diversity controls - Advertising and targeting segments - Analytics and machine-learning features ## Karrot’s Taxonomy Challenges - Karrot manages roughly 1,400 marketplace categories across up to three levels. - Users are not required to manually select highly detailed categories because that would increase posting friction and produce unreliable labels. - Earlier systems used a Golang Kafka consumer to receive posting events and extract categories with an LLM. - This approach had several limitations: - Taxonomy definitions were managed separately by different teams. - Categories alone could not express useful properties such as season or material. - Batch processing and backfilling were difficult. - Expanding to data sources outside Kafka was inconvenient. - Quality monitoring and failure handling were insufficient. - Changes to prompts or models required slow offline and online experiments. ## The Taxonomy Management System - The new system centrally manages taxonomies, performs LLM-based classification, delivers category and attribute results, and monitors quality. - Dataflow with Apache Beam was selected because it supports: - Parallel, high-throughput LLM inference - Both streaming and large-scale batch processing - Existing team expertise compared with alternatives such as Spark or Flink - BigQuery serves as the source of truth for inference results. - Analysts and data scientists can query results directly. - Online consumers can receive results through Kafka sinks into the internal feature platform. ## Configuration-Driven and Extensible Design - Taxonomy definitions are stored in YAML, allowing different services and category trees to use the same framework. - Pipeline settings, worker sizing, Kafka topics, and BigQuery destinations are also configured through YAML. - LLM models and inference strategies can be selected through configuration, including: - Primary and evaluation models - Single-shot or two-stage categorization - Attribute extraction modes - Evaluation sampling ratios - The system is designed for multilingual taxonomies. - Large translation jobs are divided into chunks. - One LLM generates translations and another validates consistency and naturalness. - A depth-first traversal carries parent-category translations into child-category prompts to maintain terminology consistency. ## Creating and Expanding Taxonomies with LLMs - New taxonomies are developed by researching established taxonomies and generating candidate trees from real data. - Existing taxonomies are expanded by: - Classifying sampled data against the current taxonomy - Asking the LLM to suggest categories for unsuitable examples - Merging similar suggestions using LLM similarity judgments - Promoting sufficiently strong candidates for review - Candidates undergo two evaluations: - Whether the originating examples are correctly assigned to the new category - Regression testing comparing classifications under the old and new taxonomies - This process enabled the team to move beyond the existing 1,400 three-level categories and create taxonomies with more than 10,000 categories and six or more levels. ## LLM Categorization Strategies The team supports multiple strategies because the best approach depends on the model and taxonomy size: - **Single shot:** Provide all categories and ask the model to choose one. - **Hierarchical classification:** Select the best category at each depth, then continue through the chosen branch. - **Two-stage tournament:** Split categories into chunks, select candidates from each chunk, and run a second selection among those candidates. - Categorization and attribute assignment are separate modular Beam `DoFn` stages: - `Article → Category inference → Attribute inference` - New approaches can be added as interchangeable strategies without redesigning the whole pipeline. ## Evaluation with LLM-as-a-Judge - A sample of production data is processed by multiple different models. - Their labels are combined into a ground-truth label, generally through majority voting. - Each model’s output is compared against that ground truth. - Accuracy changes are tracked whenever the team modifies: - The LLM model - Prompts - Pipeline structure - Categorization or attribute strategies - The ground-truth method varies depending on whether the task involves: - A single category - Multiple categories - Multi-label attributes - Category quality is measured as a precision-at-one-style accuracy: the primary model’s category must match the ground-truth category. - Attributes are evaluated with precision and recall because a post can legitimately contain multiple attribute-value pairs. The main recommendation is to treat LLM classification as a production data pipeline rather than a one-off prompt: centralize taxonomy management, support both batch and streaming execution, make inference strategies configurable, and build automated evaluation and monitoring into the system from the beginning.

Read original

Llm As A Judge Large Language Models Apache Kafka Prompt Engineering+4

spotifyFeb 19, 20263 min readCurated summary

Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3) | Spotify Engineering

Spotify argues that unsupervised coding agents become reliable only when surrounded by strong, automated feedback loops. Its “Honk” system uses component-specific verifiers, mandatory pre-PR checks, and an LLM judge to catch build failures, test failures, scope creep, and functionally incorrect changes. The conclusion is that constrained, sandboxed agents with rich verification are more predictable than flexible agents operating independently. ## Failure Modes at Scale - Agents may fail to produce a pull request, which is inconvenient but usually manageable. - They may produce PRs that fail CI, leaving engineers to repair incomplete work. - Most seriously, they may produce PRs that pass CI but are functionally wrong and potentially reach production. - These failures are more likely when components lack tests, agents modify code beyond the prompt, or agents cannot correctly run builds and tests. - Reviewing invalid or nonsensical PRs can become a significant engineering time sink. ## Verification Loops - Honk uses independent verifiers that provide incremental feedback while the agent works. - Verifiers activate automatically based on the repository contents; for example, a Maven verifier runs when a root-level `pom.xml` is present. - The agent sees an abstract MCP tool rather than the implementation details of Maven, test runners, or build systems. - Verifiers handle formatting, compilation, testing, and output parsing, returning concise error messages instead of consuming the agent’s context with raw logs. - All applicable verifiers run before a PR is opened. In Claude Code, this is enforced with a stop hook. - If verification fails, the PR is blocked and the user receives an error. ## An LLM as a Judge - Deterministic checks cannot detect every problem, especially when an agent makes unnecessary refactors or disables flaky tests. - Honk therefore sends the original prompt and proposed diff to a separate LLM judge. - The judge runs after the regular verifiers and can veto changes that exceed the requested scope. - Across thousands of sessions, the judge rejects roughly one quarter of proposed changes. - Agents successfully correct about half of the vetoed changes. - Spotify has not yet built formal evaluations for the judge, but observed that scope violations are its most common reason for rejection. ## Constrained Agents and Sandboxing - The agent has limited responsibilities: inspect the relevant code, edit files, and invoke verification tools. - Surrounding infrastructure handles prompt creation, pushing code, and user communication through systems such as Slack. - Restricting the agent’s capabilities improves predictability and provides security benefits. - Agents run in heavily sandboxed containers with limited permissions, few installed binaries, and almost no access to surrounding systems. - Spotify reports that agents solve increasingly complex tasks reliably when these feedback loops are present, but often produce unusable code without them. ## Future Expansion - Spotify plans to support more hardware and operating systems. - Current verifiers run only on Linux x86, limiting support for systems that require macOS, such as iOS applications, or ARM64 environments. - The company also intends to integrate Honk more deeply with existing CI/CD pipelines. The practical recommendation is to treat autonomous coding as an infrastructure and verification problem, not merely a prompting problem: keep agents narrowly scoped, isolate them securely, and require layered automated checks before accepting their changes.

Read original

Llm As A Judge Large Language Models AI Agents CI/CD+4

naverDec 4, 2025Original article

Naver TV (opens in new tab)

Processing complex PDF documents remains a significant bottleneck for Large Language Models (LLMs) due to the intricate layouts, nested tables, and visual charts that standard text extractors often fail to capture. To address this, NAVER developed PaLADIN, an LLM-friendly PDF parser designed to transform visual document elements into structured data that models can accurately interpret. By combining specialized vision models with advanced OCR, the system enables high-fidelity document understanding for demanding tasks like analyzing financial reports. ### Challenges in Document Intelligence * Standard PDF parsing often loses the semantic structure of the document, such as the relationship between headers and body text. * Tables and charts pose the greatest difficulty, as numerical values and trends must be extracted without losing the spatial context that defines their meaning. * A "one-size-fits-all" approach to text extraction results in "hallucinations" when LLMs attempt to reconstruct data from fragmented strings. ### The PaLADIN Architecture and Model Integration * **Element Detection:** The system utilizes `Doclayout-Yolo` to identify and categorize document components like text blocks, titles, tables, and figures. * **Table Extraction:** Visual table structures are processed through `nemoretriever-table-structure-v1`, ensuring that cell boundaries and headers are preserved. * **Chart Interpretation:** To convert visual charts into descriptive text or data, the parser employs `google/gemma3-27b-it`, allowing the LLM to "read" visual trends. * **Text Recognition:** For high-accuracy character recognition, particularly in multi-lingual contexts, the pipeline integrates NAVER’s `Papago OCR`. * **Infrastructure:** The architecture leverages `nv-ingest` for optimized throughput and speed, making it suitable for large-scale document processing. ### Evaluation and Real-world Application * **Performance Metrics:** NAVER established a dedicated parsing evaluation set to measure accuracy across diverse document types, focusing on speed and structural integrity. * **AIB Securities Reports:** The parser is currently applied to summarize complex stock market reports, where precision in numerical data is critical. * **LLM-as-a-Judge:** To ensure summary quality, the system uses an automated evaluation framework where a high-performing LLM judges the accuracy of the generated summaries against the parsed source data. For organizations building RAG (Retrieval-Augmented Generation) systems, the transition from basic text extraction to a layout-aware parsing pipeline like PaLADIN is crucial. Future improvements focusing on table cell coordinate precision and more granular chart analysis will further reduce the error rates in automated document processing.

Llm As A Judge AI Large Language Models Ocr+4

dropboxOct 2, 20253 min readCurated summary

A practical blueprint for evaluating conversational AI at scale

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.

Read original

Llm As A Judge Large Language Models Natural Language Processing Prompt Engineering+4