Llm Evaluation | Techlist.io

stripeMar 2, 20263 min readCurated summary

Can AI agents build real Stripe integrations? We built a benchmark to find out

State-of-the-art LLM agents can complete many scoped coding tasks, but fully autonomous software engineering remains difficult because real projects require long-term planning, persistent state, debugging, and end-to-end validation. Stripe evaluated this gap through a benchmark of realistic backend, frontend, database, and browser-based integration tasks. The results were stronger than expected: agents demonstrated substantial full-stack capability, but still struggled with ambiguity and the judgment required to distinguish genuine failures from bad test inputs. ## Building the Stripe Integration Benchmark - Stripe created 11 environments based on real integration challenges, including Checkout migrations and Billing API modeling. - Each environment included: - A complete codebase, database, scripts, and test Stripe credentials. - Deterministic graders using API calls, automated browser tests, or inspection of Stripe objects. - A consistent agent harness with terminal, browser, and Stripe-specific search tools through MCP. - Challenges were divided into: - **Backend-only tasks:** SDK upgrades, API changes, and database migrations. - **Full-stack tasks:** Coordinated server and client changes requiring browser verification. - **Gym problem sets:** Focused exercises testing deep knowledge of features such as Checkout and subscriptions. ## Stronger-than-Expected Agent Performance - The benchmark intentionally used fewer, harder tasks designed to expose weaknesses. - Agents successfully: - Navigated browser interfaces. - Debugged live issues. - Worked with underdocumented API behavior. - Continued productively across long interactions, with top runs averaging 63 turns. - Claude Opus 4.5 achieved a 92% average score across four full-stack tasks. - GPT-5.2 achieved a 73% average score across two gym problem sets. - In a migration from Card Element to Checkout, an agent completed and verified a test purchase using Link, despite no payment method being specified. ## Reverse-Engineering Checkout Configurations - A Checkout gym task required agents to infer API parameters from 20 prebuilt Checkout UIs. - Agents had to: - Inspect products and quantities shown in each session. - Locate matching product IDs through the Products API. - Identify shipping costs, custom fields, tax settings, and other customizations. - Translate those details into valid Checkout Session parameters. - Agents provided more than 80% of the correct parameters. - The best-performing agent recognized that one UI’s color options were hidden behind an interactive dropdown, explored the control, and included the missing values. ## Remaining Challenges with Ambiguity - Agents struggled when evaluation situations required judgment rather than straightforward implementation. - In SDK upgrade tasks, some agents supplied nonexistent Stripe data, received expected 400 errors, and treated those responses as evidence that their implementation was broken. - This illustrates a broader limitation: successful autonomous engineering requires not only writing code, but also designing meaningful tests, interpreting failures correctly, and validating behavior against realistic system state. The benchmark suggests that agents are increasingly capable of substantial Stripe integration work, including full-stack implementation and browser-based verification. However, reliable autonomy will require better handling of ambiguity, realistic test data, persistent project state, and rigorous end-to-end validation.

Read original

Llm Evaluation Api Integration State Management Database Migrations+4

googleJan 28, 20263 min readCurated summary

Towards a science of scaling agent systems: When and why agent systems work

AI agent systems do not improve simply by adding more agents. Google Research’s evaluation of 180 configurations found that coordination helps substantially on parallelizable tasks but can severely hurt sequential workflows and tool-heavy tasks. The study proposes measurable design principles and a predictive model that selected the best architecture for 87% of unseen tasks. ## Defining Agentic Tasks The study distinguishes agentic tasks from static benchmarks by requiring: - Sustained, multi-step interaction with an external environment. - Iterative information gathering under partial observability. - Adaptive strategy changes based on environmental feedback. Researchers tested five architectures across Finance-Agent, BrowseComp-Plus, PlanCraft, and Workbench: - **Single-agent:** One agent handles reasoning and actions sequentially. - **Independent:** Agents work in parallel without communication and combine results at the end. - **Centralized:** An orchestrator delegates work and synthesizes outputs. - **Decentralized:** Agents communicate directly in a peer-to-peer network. - **Hybrid:** Hierarchical oversight is combined with peer coordination. ## Coordination Must Match the Task - Multi-agent systems produced very different results across GPT, Gemini, and Claude models. - On parallelizable financial reasoning tasks, centralized coordination improved performance by **80.9%** over a single agent. - Parallel agents can independently analyze areas such as revenue, costs, and market comparisons before combining their findings. - On sequential planning tasks, every multi-agent architecture performed worse, with declines of **39–70%**. - Communication and synchronization overhead can fragment reasoning and consume the available cognitive budget. ## The Tool-Coordination Trade-off - As tasks require more tools, coordinating multiple agents becomes increasingly expensive. - Tool-heavy systems, such as coding agents with access to 16 or more tools, face a disproportionate coordination “tax.” - Adding agents is therefore especially risky when actions must be tightly ordered or frequently synchronized. ## Architecture and Reliability - Architecture affects not only performance but also how errors spread. - Independent agents amplified errors by up to **17.2×**, because no mechanism checked their intermediate results. - Centralized systems limited error amplification to **4.4×**. - An orchestrator acts as a validation bottleneck, detecting and containing mistakes before they propagate. ## Predicting the Best Architecture - The researchers built a predictive model using properties such as task decomposability and tool count. - The model achieved an **R² of 0.513**. - It correctly predicted the optimal coordination strategy for **87% of unseen task configurations**. - These results point toward systematic, task-driven agent design rather than relying on the assumption that more agents are always better. For practical deployments, choose architecture based on the task: use coordinated parallel agents for decomposable work, simpler sequential systems for tightly ordered reasoning, and centralized oversight when reliability and error containment are priorities.

Read original

Llm Evaluation Large Language Models AI Agents Multi Agent Systems+3

naverDec 3, 2025Original article

Naver TV (opens in new tab)

Naver’s VLOps framework introduces an event-driven approach to MLOps, designed to overcome the rigidity of traditional pipeline-based systems like Kubeflow. By shifting from a monolithic pipeline structure to a system governed by autonomous sensors and typed messages, Naver has achieved a highly decoupled and scalable environment for multimodal AI development. This architecture allows for seamless functional expansion and cross-cloud compatibility, ultimately simplifying the transition from model training to large-scale evaluation and deployment. ### Event-Driven MLOps Architecture * Operations such as training, evaluation, and deployment are defined as "Typed Messages," which serve as the primary units of communication within the system. * An "Event Sensor" acts as the core logic hub, autonomously detecting these messages and triggering the corresponding tasks without requiring a predefined, end-to-end pipeline. * The system eliminates the need for complex version management of entire pipelines, as new features can be integrated simply by adding new message types. * This approach ensures loose coupling between evaluation and deployment systems, facilitating easier maintenance and infrastructure flexibility. ### Omni-Evaluator and Unified Benchmarking * The Omni-Evaluator serves as a centralized platform that integrates various evaluation engines and benchmarks into a single workflow. * It supports real-time monitoring of model performance, allowing researchers to track progress during the training and validation phases. * The system is designed specifically to handle the complexities of Multimodal LLMs, providing a standardized environment for diverse testing scenarios. * User-driven triggers are supported, enabling developers to initiate specific evaluation cycles manually when necessary. ### VLOps Dashboard and User Experience * The VLOps Dashboard acts as a central hub where users can manage the entire ML lifecycle without needing deep knowledge of the underlying orchestration logic. * Users can trigger complex pipelines simply by issuing a message, abstracting the technical difficulties of cloud infrastructure. * The dashboard provides a visual interface for monitoring events, message flows, and evaluation results, improving overall transparency for data scientists and researchers. For organizations managing large-scale multimodal models, moving toward an event-driven architecture is highly recommended. This model reduces the overhead of maintaining rigid pipelines and allows engineering teams to focus on model quality rather than infrastructure orchestration.

Llm Evaluation Multimodal AI Orchestration Mlops+4

dropboxOct 2, 20253 min readCurated summary

A practical blueprint for evaluating conversational AI at scale

Conversational AI systems depend on many probabilistic stages, so even small changes can cause unexpected regressions. Dropbox Dash’s experience shows that evaluation should be treated like production engineering: systematic, repeatable, and required before changes are shipped. The approach combines curated datasets, actionable metrics, LLM-based judging, and human review. ## Evaluation as a Development Discipline - AI pipelines include intent classification, retrieval, ranking, prompt construction, inference, and safety filtering. - Changes to any stage can affect final answer quality in unpredictable ways. - Dropbox initially used ad-hoc testing, but shifted to a standardized process in which every model, prompt, or retrieval change had to pass evaluation before merging. - The evaluation framework covers datasets, metrics, tools, and workflows. - Future-proof evaluation must extend beyond text to images, video, and audio. ## Curating Public and Internal Datasets - Public datasets established baseline retrieval and question-answering performance: - **Natural Questions** tested retrieval from very large documents. - **MS MARCO** emphasized queries requiring multiple document hits. - **MuSiQue** tested multi-hop reasoning. - Internal datasets captured real-world usage from anonymized Dropbox employee queries and content. - Representative query sets reflected actual user behavior, using proxy labels or internal annotators. - Representative content sets focused on shared files, documentation, and connected data sources. - LLM-generated synthetic questions and answers covered tables, images, tutorials, and factual lookups. - These datasets became the foundation for automated pass/fail checks and experiment monitoring. ## Why Traditional Metrics Fall Short - Metrics such as BLEU, ROUGE, METEOR, BERTScore, and embedding similarity are fast and useful for detecting major regressions. - They often fail to measure production requirements, including: - Correct source citations - Factual accuracy - Valid file names and references - Reliable table parsing - Clear formatting - High ROUGE or BERTScore can coexist with hallucinations, missing citations, or buried factual errors. - Production evaluation therefore requires metrics tied directly to task requirements. ## Using LLMs as Evaluation Judges - LLM judges can assess dimensions traditional metrics miss, including: - Whether an answer addresses the query - Whether claims are supported by context - Citation correctness - Clarity, formatting, and tone - Judges receive the query, model answer, source context, and sometimes a hidden reference answer. - They return structured scores—scalar or categorical—alongside explanations. - Rubrics and judge models must themselves be tested, calibrated, versioned, and improved. - Specialized judges may be needed for particular languages or technical domains. ## Calibration and Human Review - Periodic manual labeling of sampled outputs created calibration sets for comparing human and judge-model decisions. - The team tracked agreement and judge drift over time. - Engineers manually reviewed 5–10% of each release’s regression suite. - Disagreements were investigated as either judge-prompt defects or model errors. - Recurring problems led to prompt revisions or more detailed scoring criteria. Dropbox’s evaluation-first approach treats AI changes like code changes: test them against realistic datasets, score them with task-specific rubrics, validate the evaluators, and retain human oversight. This makes conversational AI development more reliable as systems and modalities grow more complex.

Read original

Llm Evaluation Large Language Models Natural Language Processing Prompt Engineering+4

googleAug 25, 2025Original article

A scalable framework for evaluating health language models (opens in new tab)

Researchers at Google have developed a scalable framework for evaluating health-focused language models by replacing subjective, high-complexity rubrics with granular, binary criteria. This "Adaptive Precise Boolean" approach addresses the high costs and low inter-rater reliability typically associated with expert-led evaluation in specialized medical domains. By dynamically filtering rubric questions based on context, the framework significantly improves both the speed and precision of model assessments. ## Limitations of Traditional Evaluation * Current evaluation practices for health LLMs rely heavily on human experts, making them cost-prohibitive and difficult to scale. * Standard tools, such as Likert scales (e.g., 1-5 ratings) or open-ended text, often lead to subjective interpretations and low inter-rater consistency. * Evaluating complex, personalized health data requires a level of detail that traditional broad-scale rubrics fail to capture accurately. ## Precise Boolean Rubrics * The framework "granularizes" complex evaluation targets into a larger set of focused, binary (Yes/No) questions. * This format reduces ambiguity by forcing raters to make definitive judgments on specific aspects of a model's response. * By removing the middle ground found in multi-point scales, the framework produces a more robust and actionable signal for programmatic model refinement. ## The Adaptive Filtering Mechanism * To prevent the high volume of binary questions from overwhelming human raters, the researchers introduced an "Adaptive" layer. * The framework uses the Gemini model as a zero-shot classifier to analyze the user query and LLM response, identifying only the most relevant rubric questions. * This data-driven adaptation ensures that human experts only spend time on pertinent criteria, resulting in "Human-Adaptive Precise Boolean" rubrics. ## Performance and Reliability Gains * The methodology was validated in the domain of metabolic health, covering topics like diabetes, obesity, and cardiovascular disease. * The Adaptive Precise Boolean approach reduced human evaluation time by over 50% compared to traditional Likert-scale methods. * Inter-rater reliability, measured through intra-class correlation coefficients (ICC), was significantly higher than the baseline, proving that simpler scoring can provide a higher quality signal. This framework demonstrates that breaking down complex medical evaluations into simple, machine-filtered binary questions is a more efficient path toward safe and accurate health AI. Organizations developing domain-specific models should consider adopting adaptive binary rubrics to balance the need for expert oversight with the requirements of large-scale model iteration.

Llm Evaluation AI Large Language Models Gemini+4