AI | Techlist.io

googleOct 29, 2025Original article

Toward provably private insights into AI use (opens in new tab)

Google Research has introduced Provably Private Insights (PPI), a framework designed to analyze generative AI usage patterns while providing mathematical guarantees of user privacy. By integrating Large Language Models (LLMs) with differential privacy and trusted execution environments (TEEs), the system enables developers to derive aggregate trends from unstructured data without exposing individual user content. This approach ensures that server-side processing remains limited to privacy-preserving computations that are fully auditable by external parties. ### The Role of LLMs in Structured Summarization The system employs "data expert" LLMs to transform unstructured generative AI data into actionable, structured insights. * The framework utilizes open-source Gemma 3 models to perform specific analysis tasks, such as classifying transcripts into topics or identifying user frustration levels. * This "structured summarization" occurs entirely within a TEE, ensuring that the model processes raw data in an environment inaccessible to human operators or external processes. * Developers can update LLM prompts frequently to answer new research questions without compromising the underlying privacy architecture. ### Confidential Federated Analytics (CFA) Infrastructure The PPI system is built upon Confidential Federated Analytics, a technique that isolates data through hardware-based security and cryptographic verification. * User devices encrypt data and define specific authorized processing steps before uploading it to the server. * A TEE-hosted key management service only releases decryption keys to processing steps that match public, open-source code signatures. * System integrity is verified using Rekor, a public, tamper-resistant transparency log that allows external parties to confirm that the code running in the TEE is exactly what was published. ### Anonymization via Differential Privacy Once the LLM extracts features from the data, the system applies differential privacy (DP) to ensure that the final output does not reveal information about any specific individual. * The extracted categories are aggregated into histograms, with DP noise added to the final counts to prevent the identification of single users. * Because the privacy guarantee is applied at the aggregation stage, the system remains secure even if a developer uses a prompt specifically designed to isolate a single user's data. * All aggregation algorithms are open-source and reproducibly buildable, allowing for end-to-end verifiability of the privacy claims. By open-sourcing the PPI stack through the Google Parfait project and deploying it in applications like Pixel Recorder, this framework establishes a new standard for transparent data analysis. Developers should look to integrate similar TEE-based federated analytics to balance the need for product insights with the necessity of provable, hardware-backed user privacy.

AI Large Language Models Generative AI Differential Privacy+5

googleOct 28, 2025Original article

StreetReaderAI: Towards making street view accessible via context-aware multimodal AI (opens in new tab)

StreetReaderAI is a research prototype designed to make immersive street-level imagery accessible to the blind and low-vision community through multimodal AI. By integrating real-time scene analysis with context-aware geographic data, the system transforms visual mapping data into an interactive, audio-first experience. This framework allows users to virtually explore environments and plan routes with a level of detail and independence previously unavailable through traditional screen readers. ### Navigation and Spatial Awareness The system offers an immersive, first-person exploration interface that mimics the mechanics of accessible gaming. * Users navigate using keyboard shortcuts or voice commands, taking "virtual steps" forward or backward and panning their view in 360 degrees. * Real-time audio feedback provides cardinal and intercardinal directions, such as "Now facing North," to maintain spatial orientation. * Distance tracking informs the user how far they have traveled between panoramic images, while "teleport" features allow for quick jumps to specific addresses or landmarks. ### Context-Aware AI Describer At the core of the tool is a subsystem backed by Gemini that synthesizes visual and geographic data to generate descriptions. * The AI Describer combines the current field-of-view image with dynamic metadata about nearby roads, intersections, and points of interest. * Two distinct modes cater to different user needs: a "Default" mode focusing on pedestrian safety and navigation, and a "Tour Guide" mode that provides historical and architectural details. * The system utilizes Gemini to proactively predict and suggest follow-up questions relevant to the specific scene, such as details about crosswalks or building entrances. ### Interactive Dialogue and Session Memory StreetReaderAI utilizes the Multimodal Live API to facilitate real-time, natural language conversations about the environment. * The AI Chat agent maintains a large context window of approximately 1,048,576 tokens, allowing it to retain a "memory" of up to 4,000 previous images and interactions. * This memory allows users to ask retrospective spatial questions, such as "Where was that bus stop I just passed?", with the agent providing relative directions based on the user's current location. * By tracking every pan and movement, the agent can provide specific details about the environment that were captured in previous steps of the virtual walk. ### User Evaluation and Practical Application Testing with blind screen reader users confirmed the system's utility in practical, real-world scenarios. * Participants successfully used the prototype to evaluate potential walking routes, identifying critical environmental features like the presence of benches or shelters at bus stops. * The study highlighted the importance of multimodal inputs—combining image recognition with structured map data—to provide a more accurate and reliable description than image analysis alone could offer. While StreetReaderAI remains a proof-of-concept, it demonstrates that the integration of multimodal LLMs and spatial data can bridge significant accessibility gaps in digital mapping. Future implementation of these technologies could transform how visually impaired individuals interact with the world, turning static street imagery into a functional tool for independent mobility and exploration.

AI Accessibility Gemini Multimodal AI+5

figmaOct 27, 20253 min readCurated summary

Schema 2025: Design Systems For A New Era | Figma Blog

Figma’s Schema 2025 announcements present design systems as living infrastructure for an AI-driven product development process. They are evolving beyond static rules into a shared language connecting design, code, people, and AI. The updates focus on scaling across brands, enabling flexibility without sacrificing consistency, and improving the connection between design intent and implementation. ## Design systems for the AI era - As product, design, and engineering roles increasingly overlap, more people and AI tools contribute to product development. - Design systems can provide the common language needed to maintain consistency across these contributors. - Figma aims to help teams preserve quality and design intent while allowing ideas to evolve across products and platforms. - The announced features emphasize: - Power balanced with flexibility - Better connections between design and code - Broader participation in the design process ## Extended collections for multi-brand systems - Traditional variables work well for straightforward theming but can be limiting for organizations with multiple products and distinct brands. - Extended collections allow teams to create a white-labeled version of a core design system. - Individual teams can add, publish, and reuse their own themes while remaining connected to the parent system. - Extensions automatically inherit updates such as new variables or color changes. - Explicitly overridden values remain customized, allowing local flexibility without losing synchronization. - Extended collections are expected to become available in November. ## Slots for flexible components - Conventional Figma components restrict how designers can modify instances. - For example, dropdown components traditionally required hidden list items or detaching the component to add new content. - Slots will allow designers to insert their own layers inside component instances without breaking the connection to the design system. - Component authors can specify which types of instances a slot accepts. - This improves usability while preserving design-system compliance. - Slots are available through an early-access program. ## Check designs for better token usage - Developers often need clarification about which exact design token corresponds to a raw value in a design. - The Check designs linter identifies elements that should align with the design system, including variables. - Figma’s custom model suggests the appropriate variable for each context. - Designers can review suggestions before applying them and then hand off more reliable designs to development. - Early access is available to organizations and Enterprise full-seat plans. Figma’s direction is to make design systems more adaptable and intelligent: centralized enough to preserve consistency, but flexible enough for multiple brands, contributors, and use cases. Teams should look toward systems that can actively guide design and implementation rather than merely document standards.

Read original

AI Figma Design Systems Design Tokens+1

netflixOct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

AI Machine Learning Transformer Recommendation Systems+4

googleOct 22, 2025Original article

Google Earth AI: Unlocking geospatial insights with foundation models and cross-modal reasoning (opens in new tab)

Google Earth AI introduces a framework of geospatial foundation models and reasoning agents designed to solve complex, planetary-scale challenges through cross-modal reasoning. By integrating Gemini-powered orchestrators with specialized imagery, population, and environmental models, the system deconstructs multifaceted queries into actionable multi-step plans. This approach enables a holistic understanding of real-world events, such as disaster response and disease forecasting, by grounding AI insights in diverse, grounded geospatial data. ## Geospatial Reasoning Agents * Utilizes Gemini models as intelligent orchestrators to manage complex queries that require data from multiple domains. * The agent deconstructs a high-level question—such as predicting hurricane landfalls and community vulnerability—into a sequence of smaller, executable tasks. * It executes these plans by autonomously calling specialized foundation models, querying vast datastores, and utilizing geospatial tools to fuse disparate data points into a single, cohesive answer. ## Remote Sensing and Imagery Foundations * Employs vision-language models and open-vocabulary object detection trained on a large corpus of high-resolution overhead imagery paired with text descriptions. * Enables "zero-shot" capabilities, allowing users to find specific objects like "flooded roads" or "building damage" using natural language without needing to retrain the model for specific classes. * Technical evaluations show a 16% average improvement on text-based image search tasks and more than double the baseline accuracy for detecting novel objects in a zero-shot setting. ## Population Dynamics and Mobility * Focuses on the interplay between people and places using globally-consistent embeddings across 17 countries. * Includes monthly updated embeddings that capture shifting human activity patterns, which are essential for time-sensitive forecasting. * Research conducted with the University of Oxford showed that incorporating these population embeddings into a Dengue fever forecasting model in Brazil improved the R² metric from 0.456 to 0.656 for long-range 12-month predictions. ## Environmental and Disaster Forecasting * Integrates established Google research into weather nowcasting, flood forecasting, and wildfire boundary mapping. * Provides the reasoning agent with the data necessary to evaluate environmental risks alongside population density and infrastructure imagery. * Aims to provide Search and Maps users with real-time, accurate alerts regarding natural disasters grounded in planetary-scale environmental data. Developers and enterprises looking to solve high-level geospatial problems can now express interest in accessing these capabilities through Google Earth and Google Cloud. By leveraging these foundation models, organizations can automate the analysis of satellite imagery and human mobility data to better prepare for environmental and social challenges.

AI Gemini Foundation Models Embeddings+5

figmaOct 20, 20252 min readCurated summary

Design Systems: From the Basics to Big Things Ahead | Figma Blog

Design systems create consistency at scale while connecting design work to production. The post argues that as AI accelerates product development, organizations need a well-documented, shared foundation more than ever. It offers a progression from design-system fundamentals to adoption and measurement, alongside examples and resources for teams at different maturity levels. ## Design Systems 101 - Design systems evolved from: - Graphic-design and print-era style guides - Typographic standards and brand guidelines - Digital systems created by companies such as IBM, Microsoft, Apple, and Google - These systems translated visual and interaction standards from paper into digital interfaces. - The National Park Service’s adaptation of Massimo Vignelli’s 1977 design system illustrates how established systems can be carried into modern digital products. - Figma’s design-system series is intended for both teams starting their first system and organizations scaling existing ones. ## Documentation Drives Adoption - Documentation turns abstract principles into practical guidance for designers and developers. - It provides a shared reference point and helps users understand how to work with system tools and components. - Alaska Airlines prioritized documentation for its Auro design system, including guidance for Figma features such as auto layout and branch merging. - Documentation must serve different roles: - Some users need detailed specifications. - Others benefit from high-level explanations or visual examples. - As more roles participate in product design, documentation needs to remain clear, accessible, and relevant. ## Measuring Design-System ROI - Organizations can evaluate business impact through: - Component usage - Adoption rates - Consistency scores - Metrics can reveal not only whether a system is being used, but also where it needs improvement. - At athenahealth, increased detachments from a container component prompted investigation. - Detaching may indicate: - A component bug - Missing functionality - Unexpected combinations of existing elements - These signals help design-system teams prioritize fixes and enhancements. The practical recommendation is to treat a design system as an evolving product: establish strong foundations, document them for varied audiences, and use adoption and usage data to continually improve the system.

Read original

AI Figma Design Systems Design Tokens+3

googleOct 19, 2025Original article

A picture's worth a thousand (private) words: Hierarchical generation of coherent synthetic photo albums (opens in new tab)

Researchers at Google have developed a hierarchical method for generating differentially private (DP) synthetic photo albums, providing a way to share representative datasets while protecting sensitive individual information. By utilizing an intermediate text representation and a two-stage generation process, the approach maintains thematic coherence across multiple images in an album—a significant challenge for traditional synthetic data methods. This framework allows organizations to apply standard, non-private analytical techniques to safe synthetic substitutes rather than modifying every individual analysis method for differential privacy. ## The Hierarchical Generation Process * The workflow begins by converting original photo albums into structured text; an AI model generates detailed captions for each image and a summary for the entire album. * Two large language models (LLMs) are privately fine-tuned using DP-SGD: the first is trained to produce album summaries, and the second generates individual photo captions based on those summaries. * Synthetic data is then produced hierarchically, where the model first generates a global album summary to serve as context, followed by a series of individual photo captions that remain consistent with that context. * The final step uses a text-to-image AI model to transform the private, synthetic text captions back into a set of coherent images. ## Benefits of Intermediate Text Representations * Text summarization is inherently privacy-enhancing because it is a "lossy" operation, meaning the text description is unlikely to capture the exact unique details of an original photo. * Using text as a midpoint allows for more efficient resource management, as generated albums can be filtered and curated at the text level before undergoing the computationally expensive process of image generation. * The hierarchical approach ensures that photos within a synthetic album share the same characters and themes, as every caption in a set is derived from the same contextual summary. * Training two separate models with shorter context windows is significantly more efficient than training one large model, because the computational cost of self-attention scales quadratically with the length of the context. This hierarchical, text-mediated approach demonstrates that high-level semantic information and thematic coherence can be preserved in synthetic datasets without sacrificing individual privacy. Organizations should consider this workflow—translating complex multi-modal data into structured text before synthesis—to scale differentially private data generation for advanced modeling and analysis.

AI Large Language Models Generative AI Differential Privacy+4

googleOct 19, 2025Original article

Teaching Gemini to spot exploding stars with just a few examples (opens in new tab)

Researchers have demonstrated that Google’s Gemini model can classify cosmic events with 93% accuracy, rivaling specialized machine learning models while providing human-readable explanations. By utilizing few-shot learning with only 15 examples per survey, the model addresses the "black box" limitation of traditional convolutional neural networks used in astronomy. This approach enables scientists to efficiently process the millions of alerts generated by modern telescopes while maintaining a transparent and interactive reasoning process. ## Bottlenecks in Modern Transient Astronomy * Telescopes like the Vera C. Rubin Observatory are expected to generate up to 10 million alerts per night, making manual verification impossible. * The vast majority of these alerts are "bogus" signals caused by satellite trails, cosmic rays, or instrumental artifacts rather than real supernovae. * Existing specialized models often provide binary "real" or "bogus" labels without context, forcing astronomers to either blindly trust the output or spend hours on manual verification. ## Multimodal Few-Shot Learning for Classification * The research utilized few-shot learning, providing Gemini with only 15 annotated examples for three major surveys: Pan-STARRS, MeerLICHT, and ATLAS. * Input data consisted of image triplets—a "new" alert image, a "reference" image of the same sky patch, and a "difference" image—each 100x100 pixels in size. * The model successfully generalized across different telescopes with varying pixel scales, ranging from 0.25" per pixel for Pan-STARRS to 1.8" per pixel for ATLAS. * Beyond simple labels, Gemini generates a textual description of observed features and an interest score to help astronomers prioritize follow-up observations. ## Expert Validation and Self-Assessment * A panel of 12 professional astronomers evaluated the model using a 0–5 coherence rubric, confirming that Gemini’s logic aligned with expert reasoning. * The study found that Gemini can effectively assess its own uncertainty; low self-assigned "coherence scores" were strong indicators of likely classification errors. * This ability to flag its own potential mistakes allows the model to act as a reliable partner, alerting scientists when a specific case requires human intervention. The transition from "black box" classifiers to interpretable AI assistants allows the astronomical community to scale with the data flood of next-generation telescopes. By combining high-accuracy classification with transparent reasoning, researchers can maintain scientific rigor while processing millions of cosmic events in real time.

AI Generative AI Gemini Multimodal AI+4

googleOct 16, 2025Original article

Solving virtual machine puzzles: How AI is optimizing cloud computing (opens in new tab)

Google researchers have developed LAVA, a scheduling framework designed to optimize virtual machine (VM) allocation in large-scale data centers by accurately predicting and adapting to VM lifespans. By moving beyond static, one-time predictions toward a "continuous re-prediction" model based on survival analysis, the system significantly improves resource efficiency and reduces fragmentation. This approach allows cloud providers to solve the complex "bin packing" problem more effectively, leading to better capacity utilization and easier system maintenance. ### The Challenge of Long-Tailed VM Distributions * Cloud workloads exhibit a extreme long-tailed distribution: while 88% of VMs live for less than an hour, these short-lived jobs consume only 2% of total resources. * The rare VMs that run for 30 days or longer account for a massive fraction of compute resources, meaning their placement has a disproportionate impact on host availability. * Poor allocation leads to "resource stranding," where a server's remaining capacity is too small or unbalanced to host new VMs, effectively wasting expensive hardware. * Traditional machine learning models that provide only a single prediction at VM creation are often fragile, as a single misprediction can block a physical host from being cleared for maintenance or new tasks. ### Continuous Re-prediction via Survival Analysis * Instead of predicting a single average lifetime, LAVA uses an ML model to generate a probability distribution of a VM's expected duration. * The system employs "continuous re-prediction," asking how much longer a VM is expected to run given how long it has already survived (e.g., a VM that has run for five days is assigned a different remaining lifespan than a brand-new one). * This adaptive approach allows the scheduling logic to automatically correct for initial mispredictions as more data about the VM's actual behavior becomes available over time. ### Novel Scheduling and Rescheduling Algorithms * **Non-Invasive Lifetime Aware Scheduling (NILAS):** Currently deployed on Google’s Borg cluster manager, this algorithm ranks potential hosts by grouping VMs with similar expected exit times to increase the frequency of "empty hosts" available for maintenance. * **Lifetime-Aware VM Allocation (LAVA):** This algorithm fills resource gaps on hosts containing long-lived VMs with jobs that are at least an order of magnitude shorter. This ensures the short-lived VMs exit quickly without extending the host's overall occupation time. * **Lifetime-Aware Rescheduling (LARS):** To minimize disruptions during defragmentation, LARS identifies and migrates the longest-lived VMs first while allowing short-lived VMs to finish their tasks naturally on the original host. By integrating survival-analysis-based predictions into the core logic of data center management, cloud providers can transition from reactive scheduling to a proactive model. This system not only maximizes resource density but also ensures that the physical infrastructure remains flexible enough to handle large, resource-intensive provisioning requests and essential system updates.

AI Machine Learning Cloud Computing Virtual Machines+5

googleOct 15, 2025Original article

Using AI to identify genetic variants in tumors with DeepSomatic (opens in new tab)

DeepSomatic is an AI-powered tool developed by Google Research to identify cancer-related mutations by analyzing a tumor's genetic sequence with higher accuracy than current methods. By leveraging convolutional neural networks (CNNs), the model distinguishes between inherited genetic traits and acquired somatic variants that drive cancer progression. This flexible tool supports multiple sequencing platforms and sample types, offering a critical resource for clinicians and researchers aiming to personalize cancer treatment through precision medicine. ## Challenges in Somatic Variant Detection * Somatic variants are genetic mutations acquired after birth through environmental exposure or DNA replication errors, making them distinct from the germline variants found in every cell of a person's body. * Detecting these mutations is technically difficult because tumor samples are often heterogeneous, containing a diverse set of variants at varying frequencies. * Sequencing technologies often introduce small errors that can be difficult to distinguish from actual somatic mutations, especially when the mutation is only present in a small fraction of the sampled cells. ## CNN-Based Variant Calling Architecture * DeepSomatic employs a method pioneered by DeepVariant, which involves transforming raw genetic sequencing data into a set of multi-channel images. * These images represent various data points, including alignment along the chromosome, the quality of the sequence output, and other technical variables. * The convolutional neural network processes these images to differentiate between three categories: the human reference genome, non-cancerous germline variants, and the somatic mutations driving tumor growth. * By analyzing tumor and non-cancerous cells side-by-side, the model effectively filters out sequencing artifacts that might otherwise be misidentified as mutations. ## System Versatility and Application * The model is designed to function in multiple modes, including "tumor-normal" (comparing a biopsy to a healthy sample) and "tumor-only" mode, which is vital for blood cancers like leukemia where isolating healthy cells is difficult. * DeepSomatic is platform-agnostic, meaning it can process data from all major sequencing technologies and adapt to different types of sample processing. * The tool has demonstrated the ability to generalize its learning to various cancer types, even those not specifically included in its initial training sets. ## Open-Source Contributions to Precision Medicine * Google has made the DeepSomatic tool and the CASTLE dataset—a high-quality training and evaluation set—openly available to the global research community. * This initiative is part of a broader effort to use AI for early detection and advanced research in various cancers, including breast, lung, and gynecological cancers. * The release aims to accelerate the development of personalized treatment plans by providing a more reliable way to identify the specific genetic drivers of an individual's disease. By providing a more accurate and adaptable method for variant calling, DeepSomatic helps researchers pinpoint the specific drivers of a patient's cancer. This tool represents a significant advancement in deep learning for genomics, potentially shortening the path from biopsy to targeted therapeutic intervention.

AI Machine Learning Deep Learning Genomics+5

googleOct 14, 2025Original article

Coral NPU: A full-stack platform for Edge AI (opens in new tab)

Coral NPU is a new full-stack, open-source platform designed to bring advanced AI directly to power-constrained edge devices and wearables. By prioritizing a matrix-first hardware architecture and a unified software stack, Google aims to overcome traditional bottlenecks in performance, ecosystem fragmentation, and data privacy. The platform enables always-on, low-power ambient sensing while providing developers with a flexible, RISC-V-based environment for deploying modern machine learning models. ## Overcoming Edge AI Constraints * The platform addresses the "performance gap" where complex ML models typically exceed the power, thermal, and memory budgets of battery-operated devices. * It eliminates the "fragmentation tax" by providing a unified architecture, moving away from proprietary processors that require costly, device-specific optimizations. * On-device processing ensures a high standard of privacy and security by keeping personal context and data off the cloud. ## AI-First Hardware Architecture * Unlike traditional chips, this architecture prioritizes the ML matrix engine over scalar compute to optimize for efficient on-device inference. * The design is built on RISC-V ISA compliant architectural IP blocks, offering an open and extensible reference for system-on-chip (SoC) designers. * The base design delivers performance in the 512 giga operations per second (GOPS) range while consuming only a few milliwatts of power. * The architecture is tailored for "always-on" use cases, making it ideal for hearables, AR glasses, and smartwatches. ## Core Architectural Components * **Scalar Core:** A lightweight, C-programmable RISC-V frontend that manages data flow using an ultra-low-power "run-to-completion" model. * **Vector Execution Unit:** A SIMD co-processor compliant with the RISC-V Vector instruction set (RVV) v1.0 for simultaneous operations on large datasets. * **Matrix Execution Unit:** A specialized engine using quantized outer product multiply-accumulate (MAC) operations to accelerate fundamental neural network tasks. ## Unified Developer Ecosystem * The platform is a C-programmable target that integrates with modern compilers such as IREE and TFLM (TensorFlow Lite Micro). * It supports a wide range of popular ML frameworks, including TensorFlow, JAX, and PyTorch. * The software toolchain utilizes MLIR and the StableHLO dialect to facilitate the transition from high-level models to hardware-executable code. * Developers have access to a complete suite of tools, including a simulator, custom kernels, and a general-purpose MLIR compiler. SoC designers and ML developers looking to build the next generation of wearables should leverage the Coral NPU reference architecture to balance high-performance AI with extreme power efficiency. By utilizing the open-source documentation and RISC-V-based tools, teams can significantly reduce the complexity of deploying private, always-on ambient sensing.

AI Machine Learning Coral Npu Edge Ai+5

googleOct 8, 2025Original article

XR Blocks: Accelerating AI + XR innovation (opens in new tab)

XR Blocks is an open-source, cross-platform framework designed to bridge the technical gap between mature AI development ecosystems and high-friction extended reality (XR) prototyping. By providing a modular architecture and high-level abstractions, the toolkit enables creators to rapidly build and deploy intelligent, immersive web applications without managing low-level system integration. Ultimately, the framework empowers developers to move from concept to interactive prototype across both desktop simulators and mobile XR devices using a unified codebase. ### Core Design Principles * **Simplicity and Readability:** Drawing inspiration from the "Zen of Python," the framework prioritizes human-readable abstractions where a developer’s script reflects a high-level description of the experience rather than complex boilerplate code. * **Creator-Centric Workflow:** The architecture is designed to handle the "plumbing" of XR—such as sensor fusion, AI model integration, and cross-platform logic—allowing creators to focus entirely on user interaction and experience. * **Pragmatic Modularity:** Rather than attempting to be a perfect, all-encompassing system, XR Blocks favors an adaptable and simple architecture that can evolve alongside the rapidly changing fields of AI and spatial computing. ### The Reality Model Abstractions * **The Script Primitive:** Acts as the logical center of an application, separating the "what" of an interaction from the "how" of its underlying technical implementation. * **User and World:** Provides built-in support for tracking hands, gaze, and avatars while allowing the system to query the physical environment for depth, estimated lighting conditions, and object recognition. * **AI and Agents:** Facilitates the integration of intelligent assistants, such as the "Sensible Agent," which can provide proactive, context-aware suggestions within the XR environment. * **Virtual Interfaces:** Offers tools to augment blended reality with virtual UI elements that respond to the user's physical context. ### Technical Implementation and Integration * **Web-Based Foundation:** The framework is built upon accessible, standard technologies including WebXR, three.js, and LiteRT (formerly TFLite) to ensure a low barrier to entry for web developers. * **Advanced AI Support:** It features native integration with Gemini for high-level reasoning and context-aware applications. * **Cross-Platform Deployment:** Developers can prototype depth-aware, physics-based interactions in a desktop simulator and deploy the exact same code to Android XR devices. * **Open-Source Resources:** The project includes a comprehensive suite of templates and live demos covering specific use cases like depth mapping, gesture modeling, and lighting estimation. By lowering the barrier to entry for intelligent XR development, XR Blocks serves as a practical starting point for researchers and developers aiming to explore the next generation of human-centered computing. Interested creators can access the source code on GitHub to begin building immersive, AI-driven applications that function seamlessly across the web and specialized XR hardware.

AI Gemini Webxr Litert+4

lineOct 3, 2025Original article

A month-long project in (opens in new tab)

This blog post explores how LY Corporation reduced a month-long development task to just five days by leveraging "vibe coding" with Generative AI tools like ChatGPT and Cursor. By shifting from traditional, rigid documentation to an iterative, demo-first approach, developers can rapidly validate multiple UI/UX solutions for complex problems like restaurant menu registration. The author concludes that AI's ability to handle frequent re-work makes it more efficient to "build fast and iterate" than to aim for perfection through long-form specifications. ### Strategic Shift to Rapid Prototyping * Traditional development cycles (spec → design → dev → fix) are often too slow to keep up with market trends due to heavy documentation and impact analysis. * The "vibe coding" approach prioritizes creating "working demos" over perfect specifications to find "good enough" answers through rapid feedback loops. * AI reduces the psychological and logistical burden of "starting over," allowing developers to refine the context and quality of outputs through repeated interaction without the friction of manual re-documentation. ### Defining Requirements and Solution Ideation * Initial requirements are kept minimal, focusing only on the core mission, top priorities, and essential data structures (e.g., product name, image, description) to avoid limiting AI creativity. * ChatGPT is used to generate a wide range of solution candidates, which are then filtered into five distinct approaches: Stepper Wizards, Live Previews with Quick Add, Template/Cloning, Chat Input, and OCR-based photo scanning. * This stage emphasizes volume and variety, using AI-generated pros and cons to establish selection criteria and identify potential UX bottlenecks early in the process. ### Detailed Design and Multi-Solution Wireframing * Each of the five chosen solutions is expanded into detailed screen flows and UI elements, such as progress bars, bottom sheets, and validation logic. * Prompt engineering is used iteratively; if an AI-generated result lacks a specific feature like "temporary storage" or "mandatory field validation," the prompt is adjusted to regenerate the design instantly. * The focus remains on defining the "what" (UI elements) and "how" (user flow) through textual descriptions before moving to actual coding. ### Implementation with Cursor and Flutter * Cursor is utilized to generate functional code based on the refined wireframes, using Flutter as the framework to ensure rapid cross-platform development for both iOS and Android. * The development follows a "skeleton-first" approach: first creating a main navigation hub with five entry points, then populating each individual solution module one by one. * Technical architecture decisions, such as using Riverpod for state management or SQLite for data storage, are layered onto the demo post-hoc, reversing the traditional "stack-first" development order to prioritize functional validation. ### Recommendation To maximize efficiency, developers should treat AI as a partner for high-speed iteration rather than a one-shot tool. By focusing on creating functional demos quickly and refining them through direct feedback, teams can bypass the bottlenecks of traditional software requirements and deliver user-centric products in a fraction of the time.

AI Large Language Models Generative AI Prompt Engineering+5

lineOct 3, 2025Original article

IUI 202 (opens in new tab)

The IUI 2025 conference highlighted a significant shift in the AI landscape, moving away from a sole focus on model performance toward "human-centered AI" that prioritizes collaboration, ethics, and user agency. The prevailing consensus across key sessions suggests that for AI to be sustainable and trustworthy, it must transcend simple automation to become a tool that augments human perception and decision-making through transparent, interactive, and socially aware design. ## Reality Design and Human Augmentation The concept of "Reality Design" suggests that Human-Computer Interaction (HCI) research must expand beyond screen-based interfaces to design reality itself. As AI, sensors, and wearables become integrated into daily life, technology can be used to directly augment human perception, cognition, and memory. * Memory extension: Systems can record and reconstruct personal experiences, helping users recall details in educational or professional settings. * Sensory augmentation: Technologies like selective hearing or slow-motion visual playback can enhance a user's natural observational powers. * Cognitive balance: While AI can assist with task difficulty (e.g., collaborative Lego building), designers must ensure that automation does not erode the human will to learn or remember, echoing historical warnings about technology-induced "forgetfulness." ## Bridging the Socio-technical Gap in AI Transparency Transparency in AI, particularly for high-risk areas like finance or medicine, should not be limited to showing mathematical model weights. Instead, it must bridge the gap between technical complexity and human understanding by focusing on user goals and social contexts. * Multi-faceted communication: Effective transparency involves model reporting (Model Cards), sharing safety evaluation results, and providing linguistic or visual cues for uncertainty rather than just numerical scores. * Counterfactual explanations: Users gain better trust when they can see how a decision might have changed if specific input conditions were different. * Interaction-based transparency: Transparency must be coupled with control, allowing users to act as "adjusters" who provide feedback that the model then reflects in its future outputs. ## Interactive Machine Learning and Human-in-the-Loop The framework of Interactive Machine Learning (IML) challenges the traditional view of AI as a static black box trained on fixed data. Instead, it proposes an interactive loop where the user and the model grow together through continuous feedback. * User-driven training: Users should be able to inspect model classifications, correct errors, and have those corrections immediately influence the model's learning path. * Beyond automation: This approach reframes AI from a replacement for human labor into a collaborative partner that adapts to specific user behaviors and professional expertise. * Impact on specialized tools: Modern applications include educational platforms where students manipulate data directly and research tools that integrate human intuition into large-scale data analysis. ## Collaborative Systems in Specialized Professional Contexts Practical applications of human-centered AI are being realized in sensitive fields like child counseling, where AI assists experts without replacing the human element. * Counselor-AI transcription: Systems designed for counseling analysis allow AI to handle the heavy lifting of transcription while counselors manage the nuance and contextual editing. * Efficiency through partnership: By focusing on reducing administrative burdens, these systems enable professionals to spend more time on high-level cognitive tasks and emotional support, demonstrating the value of AI as a supportive infrastructure. The future of AI development requires moving beyond isolated technical optimization to embrace the complexity of the human experience. Organizations and developers should focus on creating systems where transparency is a tool for "appropriate trust" and where design is focused on empowering human capabilities rather than simply automating them.

AI Large Language Models Human In The Loop Human Centered Ai+5

googleOct 1, 2025Original article

A collaborative approach to image generation (opens in new tab)

Google Research has introduced PASTA (Preference Adaptive and Sequential Text-to-image Agent), a reinforcement learning agent designed to transform image generation from a single-prompt task into a collaborative, multi-turn dialogue. By learning individual user preferences through sequential interactions, the system eliminates the frustration of trial-and-error prompting to achieve a specific creative vision. ## Data Strategy and User Simulation * Researchers collected a foundational dataset featuring over 7,000 human interactions, using Gemini Flash for prompt expansion and Stable Diffusion XL (SDXL) for image generation. * To overcome the scarcity of real-world interaction data, the team developed a user simulator that generated over 30,000 additional interaction trajectories. * The simulator is built on two primary components: a utility model that predicts how much a user will like an image, and a choice model that predicts which image a user will select from a given set. ## Latent Preference Discovery * The architecture utilizes pre-trained CLIP encoders paired with user-specific components to capture nuanced aesthetic tastes. * An expectation-maximization (EM) algorithm is employed to identify "user types," allowing the system to cluster users with similar interests, such as a preference for specific artistic styles or subject matter like "Food" or "Animals." * This approach enables the model to generalize preferences quickly, allowing it to adapt to new users based on minimal initial feedback. ## The Collaborative Generation Loop * PASTA operates as a value-based reinforcement learning model that aims to maximize cumulative user satisfaction across an entire interaction session. * The workflow begins with a candidate generator creating diverse prompt expansions; a candidate selector then picks an optimal "slate" of four variations to present to the user. * Each user selection provides a feedback signal that guides the agent’s next set of suggestions, iteratively narrowing the gap between the generated output and the user's intent. ## Training and Performance Validation * The agent was trained using Implicit Q-learning (IQL) to optimize decision-making without requiring online interaction during the training phase. * Performance was measured using several metrics, including Pick-a-Pic accuracy, Spearman’s rank correlation, and cross-turn accuracy. * Results indicated that agents trained on a combination of real-world and simulated data significantly outperformed baseline models and versions trained on only one data type. PASTA demonstrates that integrating iterative feedback loops and reinforcement learning can effectively bridge the "intent gap" in generative AI. For developers building creative tools, this research suggests that move-away from static prompting toward adaptive, simulation-trained agents can provide a more satisfying and intuitive user experience.

AI Reinforcement Learning Text To Image Clip+5