Science One Framework: A verifiable autonomous research framework via Chain-of-Evidence
The Science One Framework addresses a central weakness in autonomous AI research: polished papers can contain fabricated citations, unreproducible results, or methods that do not match the underlying code. Its Chain-of-Evidence (CoE) framework requires every claim to be linked to concrete evidence and introduces CoE Audit to test those links automatically. In evaluations, Science One produced fully verifiable papers while matching or exceeding baseline research agents and human performance on several benchmarks. ## Chain-of-Evidence for Verifiable Research - CoE defines trustworthy research artifacts through two requirements: - **Completeness:** Every claim has a recorded evidence chain. - **Correctness:** The evidence genuinely supports the claim. - Claims may include: - Bibliographic references - Reported scores - Method descriptions - Conclusions - Evidence can include peer-reviewed papers, experiment logs, executed code, or result tables. - Hallucinated citations, unreproducible scores, and discrepancies between described and implemented methods are treated as broken evidence chains. ## The Science One Framework The framework builds evidence into the research process instead of attempting to verify a paper after it has been written. - **Problem Investigator** - Uses the Semantic Scholar API to construct a citation graph. - Reads up to 100 full-text PDFs per topic. - Produces a structured research brief. - Restricts final-paper references to sources retrieved through the grounded API, avoiding citations generated from model memory. - **Discovery Engine** - Explores ideas through parallel explore-exploit branches. - Each isolated cycle includes a Solver agent and a task-specific evaluator. - High-performing solutions are iteratively refined. - Raw evaluator outputs are preserved in strict, read-only records. - **Paper Writer and Claim Verifier** - Creates a structured inventory of factual claims. - Attaches inline evidence tags linking claims to workspace artifacts. - Checks each claim against its declared source. - Rewrites unsupported claims conservatively rather than allowing them to exceed the evidence. ## CoE Audit Integrity Checks CoE Audit is an automated, post-hoc forensic review of a paper, solution, code, and references. - **Score verification:** Re-runs the submitted code independently and compares the result with the paper’s reported score. - **Specification violation:** Checks whether the code solves the intended task without exploiting the evaluator or accessing ground-truth answers. - **Reference verification:** Validates every bibliography entry against academic APIs. - **Method-code alignment:** Compares the paper’s method description with the actual implementation using LLM-based judges. ## Evaluation Results - The audit evaluated 75 papers across five systems-optimization tasks: Prism, Cloudcast, EPLB, LLM-SQL, and transaction scheduling. - Science One led the evaluated systems on all four integrity checks. - It had: - Zero phantom references - Perfect score verification - The strongest method-code alignment - Baseline systems hallucinated up to 21% of references and sometimes described advanced algorithms that were implemented as simple deterministic heuristics. - Strong verification did not reduce performance: - Science One matched or exceeded human experts on all five ADRS tasks. - It achieved the best overall result on Cloudcast and EPLB. - On additional MLE-Bench and Parameter Golf evaluations, the framework also demonstrated competitive performance, including two Gold Medals across five difficult Kaggle competitions. The main recommendation is to design autonomous research systems around evidence generation from the beginning. Grounded retrieval, immutable experiment records, claim-level verification, and independent auditing can substantially improve reliability without necessarily sacrificing research performance.
Read original(opens in new tab)