Testing LLMs on superconductivity research questions
LLMs may help physicists navigate complex research, but their reliability depends heavily on the quality and curation of their sources. In a high-temperature superconductivity study, systems grounded in expert-selected literature—especially NotebookLM and a custom retrieval-augmented generation system—outperformed general web-access models. The results suggest that trustworthy scientific AI requires balanced reasoning, strong evidence, and carefully controlled reference collections. ## Evaluating LLMs on Superconductivity - Researchers from Google Research and Cornell University tested whether LLMs could answer expert-level questions in condensed matter physics. - The study focused on cuprate high-temperature superconductors, whose underlying mechanism remains unresolved despite decades of research. - Understanding superconductivity in these materials could help scientists discover compounds that work at higher temperatures. - The field contains thousands of experimental and theoretical papers and competing explanations, making it difficult for researchers—especially newcomers—to establish a balanced view. ## Study Design and Sources - Six systems were evaluated: - GPT-4o - Perplexity - Claude 3.5 - Gemini Advanced Pro 1.5 - Google NotebookLM - A custom retrieval-augmented generation (RAG) system - Four models had broad web access, including 765 open-access experimental papers and 1,553 theoretical papers. - NotebookLM and the custom RAG system used a curated database: - Twelve superconductivity experts selected 15 review articles. - Those reviews contained approximately 3,300 references. - A final collection of 1,726 experimental papers and reviews was assembled. - Experts created 67 difficult questions, including questions about doping levels and evidence for quantum criticality in cuprates. ## Evaluation Criteria Experts used masked reviews and scored responses from 0 to 2 on: - Balance between competing scientific perspectives - Comprehensiveness and factual depth - Conciseness and clarity - Evidence and links to sources - Relevance of supplied images - Qualitative comments ## Results - NotebookLM achieved the strongest overall performance. - The custom RAG system ranked second overall, showing the value of retrieval from the same expert-curated sources. - NotebookLM, Gemini, and the custom RAG system performed best at presenting balanced and comprehensive answers. - NotebookLM provided the strongest evidence and citations but was less concise than the other systems. - Image quality was generally weaker; the custom RAG system performed best among the models that regularly supplied images. - All systems showed areas needing improvement, particularly when addressing nuanced, unresolved research questions. ## Practical Implication For scientific research, LLMs should be paired with expert-curated, quality-controlled literature rather than relying solely on unrestricted web searches. Such systems can serve as research tutors or thought partners, but their answers still require expert verification, especially in fields with competing theories and rapidly evolving evidence.
Read original(opens in new tab)