Transformer Architecture

2 posts

google3 min readCurated summary

Catalyzing scientific impact through global partnerships and open resources

Google Research argues that scientific breakthroughs have the greatest impact when their software, datasets, and methods are openly shared and responsibly maintained through global partnerships. Its open-science efforts span genomics, neuroscience, climate, biodiversity, and healthcare, reaching more than 250,000 researchers and developers. The post concludes that collaboration and open resources can turn individual discoveries into tools for broader scientific progress and real-world benefits. ## Partnerships Across the Scientific Ecosystem - Google Research works with organizations including UCSC’s Genomics Institute, Janelia Research Campus, ISTA, CSIRO, AIIMS, and the Centre for Population Genomics. - It supports major international initiatives such as: - The Human Pangenome Research Consortium - The Earth BioGenome Project - The NIH BRAIN Initiative - Google is also developing communities of practice for scientific developers, beginning in India, Korea, Japan, and Australia. ## Open-Source Tools and Datasets - **Genomics** - DeepVariant, DeepConsensus, and DeepPolisher support DNA analysis from sequencing through genome assembly. - These tools have helped process exomes and whole genomes from 2.5 million people. - **Neuroscience** - Flood-filling networks, Neuroglancer, and TensorStore enable analysis and visualization of petascale brain reconstructions. - The public H01 dataset contains 1.4 petabytes of human brain tissue data and has been accessed more than 200,000 times. - MICrONS provides a large wiring and functional map of the mouse visual cortex. - **Earth and Atmospheric Science** - Open Buildings contains 1.8 billion building detections across 58 million square kilometers. - Caravan supports large-scale hydrology and flood forecasting in 150 countries, covering roughly 2 billion people. - Groundsource includes 2.6 million historical urban flood events from more than 150 countries. - NeuralGCM is a differentiable hybrid atmospheric model, while FireBench supports wildfire research with high-resolution synthetic data. - **Biodiversity** - SpeciesNet classifies 2,498 animal categories in wildlife-camera images. - **Healthcare** - HAI-DEF provides open-weight medical foundation models, including MedGemma, with more than 4.8 million downloads. - Open Health Stack offers secure, offline-capable tools based on modern healthcare standards. - OHS-powered applications have reached more than 65 million people across over 10 countries. ## Scientific and Humanitarian Impact - **Genomics** - Work with UCSC improved pangenome references and reduced genetic-variant identification errors by 50%. - The research contributes to more representative genomic resources through the Human Pangenome Research Consortium. - **Weather and Agriculture** - The University of Chicago’s Human-Centered Weather Forecasts Initiative used NeuralGCM and ECMWF systems to predict India’s monsoon onset up to a month ahead. - Forecasts, including an unusual dry spell, were delivered by SMS to 38 million Indian farmers to support planting decisions. - **Disaster Response** - UNHCR and other organizations use Open Buildings to improve survey sampling for displaced populations. - The dataset also supports research into building vulnerability to sea-level rise in the Global South. - Sunbird AI uses the data to assess energy needs in urban and rural communities. - **Neuroscience and Medicine** - Johns Hopkins researchers used the H01 brain dataset to identify a possible new form of neuronal communication, suggesting that current models of brain organization may be incomplete. - The finding could have implications for understanding conditions such as Alzheimer’s disease. - Google also partnered with Stanford Medicine and UCSC to accelerate genome analysis in urgent cases of suspected genetic disease. ## Practical Conclusion The post presents open-source scientific infrastructure, accessible datasets, and cross-border partnerships as essential to accelerating discovery. Researchers and institutions can maximize impact by sharing reproducible tools, maintaining resources collaboratively, and applying them to urgent global challenges.

Read original(opens in new tab)
googleOriginal article

Making LLMs more accurate by using all of their layers (opens in new tab)

Self Logits Evolution Decoding (SLED) is a novel decoding strategy designed to reduce hallucinations and improve the factual accuracy of large language models without requiring external data or fine-tuning. By leveraging the internal representations of all model layers rather than just the final output, SLED aligns generation with the model’s intrinsic knowledge more effectively. Research shows that this approach consistently enhances performance across diverse tasks, including complex reasoning, multiple-choice questions, and open-ended generation. ## Limitations of Standard Decoding * Standard LLMs typically generate text by relying solely on the "logits" (prediction scores) of the final layer to determine the next token. * This process often leads to hallucinations because the final layer may prioritize "popular" or common patterns from training data over factual accuracy. * While techniques like Retrieval Augmented Generation (RAG) provide external context, they increase system complexity and do not address the model's internal tendency to ignore subtle contextual cues during the final projection. ## The Technical Mechanism of SLED * SLED utilizes "early exit" logits from every intermediate layer of the Transformer architecture, rather than just the final one. * The strategy reuses the model's final projection matrix on these intermediate layers to create multiple probability distributions across the same set of potential tokens. * By calculating a weighted average of the distributions from all layers, SLED refines the prediction to better reflect the model's latent knowledge. * This multi-layer approach allows the model to catch nuances—such as specific math constraints or geographic facts—that might be "smoothed over" by the final layer’s preference for high-probability sequences. ## Practical Performance and Reasoning * In chain-of-thought tasks, SLED helps the model maintain logic; for example, it can correctly identify when a discount should be applied in a math problem by favoring intermediate layers that recognize the "if/then" logic over a simple arithmetic pattern. * The method is model-agnostic and has shown consistent accuracy gains across various LLM scales and configurations. * SLED is highly flexible and can be integrated with existing factuality decoding methods or speculative decoding to further reduce hallucinations without the need for additional training data. For developers and researchers seeking to boost the reliability of LLMs, SLED offers a computationally efficient alternative to fine-tuning. By simply adjusting the decoding strategy to incorporate the rich information available in intermediate layers, models can achieve higher factuality and more robust reasoning capabilities in real-world applications.