Computer Vision | Techlist.io

netflixJun 23, 20263 min readCurated summary

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Netflix explores AI video-editing tools designed to preserve artists’ creative control rather than regenerate entire clips indiscriminately. The research addresses two major problems: unintended changes to untouched footage and physically implausible results when objects are removed. Its proposed systems, Vera and VOID, generate targeted edits while preserving scene identity, performance, and continuity. ## Challenges in Generative Video Editing - Full-video regeneration can unintentionally change: - Actors’ identities and performances - Backgrounds and objects - Important scene details - Object removal often produces unnatural results because models erase the target without reconstructing realistic motion and physical interactions. - Professional editors need precise control over what changes and what remains untouched. ## Vera: Layered Video Diffusion - Vera generates: - An edit layer containing the requested visual change - An alpha matte defining where that change should appear - These layers are composited with the original footage, leaving pixels outside the edited region intact. - The approach supports tasks such as: - Adding objects - Changing backgrounds - This layered design helps preserve original identities, performances, and details. ## Training Dataset - Netflix created a custom dataset because existing public datasets lacked high-quality layered video data. - The dataset contains 486,000 frames at 832×480 resolution. - It includes: - **Synthetic composites:** Foreground objects with alpha mattes placed over generated backgrounds. - **Realistic single-object videos:** Real footage processed with segmentation, matting, background generation, and human review. - **Realistic multi-object videos with effects:** Objects isolated along with shadows, reflections, and other scene effects. ## Vera’s Model Architecture - Vera uses a Mixture-of-Transformers design with three specialized DiTs for: - The edit layer - The alpha matte - The composite video - Each branch has its own attention projections and feed-forward weights, allowing specialization while joint attention enables communication between layers. - The model is initialized from a pretrained text-to-video model. - Additional embeddings and input layers help distinguish source-video, mask, alpha, and composite information. ## Evaluation and Results - Netflix tested Vera on: - 72 object-addition video-prompt pairs - 69 background-change pairs - The benchmark included varied motion speeds, camera movements, object counts, and scene complexity. - Evaluation measured: - Preservation of untouched content - Compliance with text instructions - Temporal and per-frame video quality - Vera-1.3B and Vera-14B substantially outperformed existing methods on content preservation while achieving comparable instruction-following and visual quality. Netflix’s research favors localized, layered editing over unrestricted video regeneration. Vera demonstrates how separating edits from original footage can make generative tools safer and more controllable for professional workflows; the accompanying VOID research aims to apply similar principles to physically plausible object and interaction removal.

Read original

Computer Vision Generative AI Transformer Diffusion Models+4

googleJun 16, 20263 min readCurated summary

From pixels to planning: Earth AI for nature restoration

Google Research developed a high-resolution AI system that converts satellite imagery into vector data identifying small ecological features such as hedgerows, stone walls, and copses. These features can store carbon and support biodiversity without taking agricultural land out of production, but they are often missed by conventional forest inventories. The resulting open dataset aims to make fine-scale nature restoration measurable and actionable across the UK. ## The case for fine-scale restoration - Forests support carbon sequestration, water filtration, and biodiversity, but expanding forests can compete with food production. - Conservation efforts can also create “leakage,” shifting environmental damage elsewhere. - Hedgerows, shelterbelts, and other woody features offer ecological benefits while remaining integrated into farmland. - Google’s earlier **Farmscapes 2020** raster map detected these features across England, but pixel data was insufficient for planning, restoration, and carbon accounting. ## Challenges in mapping the countryside - Agricultural features often overlap spatially, such as hedgerows growing alongside or over stone walls. - Processing data in S2-cell tiles can split features at tile boundaries. - A generic “woody” classification does not reveal whether a feature is a forest, wildlife corridor, or isolated copse. - England’s more than 130,000 km² of high-resolution imagery created major computational demands for conventional raster-to-vector processing. ## Deep learning and vectorization - The team fine-tuned Remote Sensing Foundations’ Vision Transformer, pretrained on more than 300 million global satellite images. - Only about 247 km² of annotated British landscape data was available, so the pretrained model helped transfer broad visual knowledge to local conditions. - A dual-layer labeling system combined submeter imagery with 1-meter LiDAR to distinguish: - Ground-level boundaries such as fields and water - Above-ground features such as trees and walls - Geometry-merging algorithms reconnected features split across S2-cell borders. ## Classifying ecological features - The system used the **Polsby–Popper compactness score** to classify detected shapes by geometry. - Woodlands were defined as contiguous canopies at least 30 meters in diameter. - Woody patches included small copses and individual trees. - Linear woody features, including hedgerows and corridors, were identified using a compactness score below 0.5. - This classification turns raw detections into an inventory tied more closely to ecological function. ## Scaling the analysis - Google Earth Engine processed thousands of S2 cells in parallel. - This made it possible to generate vector geometries for millions of features across England without overwhelming traditional systems. - The resulting dataset is intended for landowners, conservationists, scientists, and policymakers. ## Future applications - The researchers plan to apply the method to silvopasture and agrisilviculture. - High-precision monitoring could help detect conservation “leakage” beyond project boundaries. - Open access to the data could support restoration while protecting agricultural productivity and food security. The dataset provides a practical foundation for planning and measuring nature recovery on working lands, demonstrating how AI can translate overlooked landscape details into conservation decisions.

Read original

Computer Vision Deep Learning Remote Sensing Geospatial Data+4

tossJun 9, 20265 min readCurated summary

The History of Facial Recognition and the Future of Face Pay

Face recognition has evolved from manually recorded facial measurements to deep-learning systems capable of identifying people with near-perfect accuracy. The technology’s progression depended on automated feature extraction, robust local descriptors, neural networks, and increasingly large datasets. Face payment represents its next major application, combining convenience with layered security designed to address biometric privacy risks. ## From Manual Measurements to Automated Recognition - In the 1960s, Woodrow Wilson Bledsoe pioneered face recognition through a classified government project. - Researchers manually recorded coordinates for features such as eye spacing, nose-to-mouth distance, and ear positions. - Computers performed calculations, but humans still identified the facial landmarks. - In 1973, Takeo Kanade developed a system that automatically detected the eyes, nose, and mouth. - His work demonstrated that computers could extract meaningful facial information without manually supplied coordinates. ## Eigenfaces and Statistical Representation - In 1991, Matthew Turk and Alex Pentland introduced Eigenfaces using Principal Components Analysis (PCA). - The system created an average face and represented individual faces by how they differed from that average. - These differences formed “Eigenfaces,” mathematical components that worked like an alphabet for describing faces. - Recognition became a matter of comparing the combination weights assigned to two facial images. ## Coping with Lighting and Real-World Conditions - Early systems struggled when the same face appeared under different lighting, angles, or shadows. - Researchers shifted from analyzing the entire face to examining local regions and textures. - Local Binary Patterns (LBP) compared each pixel with its neighbors, making facial descriptions less dependent on absolute brightness. - Hand-crafted features were combined with classifiers such as: - **SVM**, which finds separating boundaries between identities. - **AdaBoost**, which combines weak classifiers into a stronger recognition system. - These methods improved robustness even though they could not learn features automatically like modern deep-learning models. ## Deep Learning Surpasses Human-Level Benchmarks - Facebook’s 2014 DeepFace system used deep neural networks trained on approximately four million face images. - It achieved 97.35% accuracy on the LFW benchmark, approaching the reported human score of 97.53%. - Google FaceNet later introduced Triplet Loss and reached 99.63% on LFW. - SphereFace, CosFace, and ArcFace continued improving discriminative facial representations. - The field shifted from proving that recognition was possible to optimizing accuracy and reliability. ## The Role of Large-Scale Datasets - **FERET:** Established an early standardized benchmark with 1,199 people and 14,126 grayscale images. - **LFW:** Contained 13,233 images of 5,749 people collected from natural, uncontrolled environments. - **VGGFace:** Provided 2.7 million images of 2,600 people, reinforcing the relationship between data scale and deep-learning performance. - **MS-Celeb-1M:** Pursued celebrity recognition at massive scale but was later withdrawn over privacy concerns. - **WebFace260M:** Started with 260 million images and refined them into roughly 42 million images covering two million people. - These datasets improved recognition while also highlighting the tension between technical progress, data quality, and privacy. ## Face Recognition as a Payment Method - Face recognition had already reached smartphones, airports, and office access systems, but payments required stricter standards. - Unlike device unlocking, payment errors can directly cause financial loss. - Toss FacePay launched in South Korea in September 2025, aiming to combine strong security with a frictionless experience. - Face payment offers: - Availability without carrying a wallet or charged phone. - Hands-free checkout. - Faster payment without searching for cards or opening an app. ## How FacePay Works - A terminal camera identifies the customer among registered users. - The system must also reject unregistered users and apply additional authentication in ambiguous cases, such as similar-looking individuals or twins. - Once identified, the system connects the customer to a registered credit card, debit card, or another selected payment method. - Face payment adds customer identification and payment-method lookup to the conventional POS, VAN, and card-issuer approval flow. ## Edge and Cloud Processing - **Edge processing** offers low latency and better privacy because images need not leave the terminal. - Its limitations include restricted hardware, smaller models, lower potential accuracy, and difficult device-by-device updates. - **Cloud processing** enables stronger models, centralized logging, and simpler model updates. - It introduces transmission delays and increases the importance of network and server security. - FacePay combines both approaches: the terminal performs initial processing, while the server handles facial-feature extraction, recognition, and payment operations. ## Layered Biometric Security - Communications between terminals and servers are protected with TLS, while images receive additional AES-256 encryption. - Matrix Projection creates cancellable biometric vectors: changing the key produces a different representation of the same face. - If a biometric vector were compromised, it could theoretically be reissued with a new key, similar to changing a password. - The stored biometric representation is designed not to correspond one-to-one with the original face image or allow easy reconstruction. - Access to biometric data is restricted, logged, and limited to operational needs. - FacePay also underwent review by South Korea’s Personal Information Protection Commission and offers full compensation for fraudulent payments through its protection program. The development of face payment reflects six decades of progress in computer vision, machine learning, data infrastructure, and security engineering. Its practical success depends not only on recognition accuracy, but also on cancellable biometric protection, strict access controls, privacy oversight, and effective defenses against spoofing.

Read original

Computer Vision Cloud Computing Deep Learning Data Privacy+4

googleJun 4, 20263 min readCurated summary

Towards passive heart health monitoring via smartphone camera

PHRM is a Google Research system that passively estimates heart rate and daily resting heart rate using short facial videos captured by a smartphone’s front-facing camera after face unlock. It applies remote photoplethysmography and efficient deep learning, achieving wearable-comparable accuracy in laboratory and real-world settings. A central contribution is its deliberate validation across skin tones, including substantial representation of participants with dark skin. ## Passive Heart Monitoring with a Smartphone - PHRM records approximately eight seconds of facial video during everyday phone use. - It estimates heart rate from changes in how light reflects from the skin as blood pulses through facial tissue. - The system uses: - Temporal shift convolutional neural networks for efficient on-device prediction. - Confidence scores to filter unreliable measurements. - Kalman filtering and daily aggregation to estimate resting heart rate. - Unlike earlier smartphone approaches requiring a finger over the camera, PHRM works passively in the background. ## Designing for Skin-Tone Inclusivity - Existing remote PPG research often relied on small, controlled studies and underrepresented people with dark skin. - Google trained PHRM on more than 350,000 video clips from nearly 700 participants across laboratory and real-world conditions. - Dataset targets based on the Monk Skin Tone scale ensured: - At least 25% representation of light and medium skin tones. - At least 33% representation of dark skin tones. - The researchers required heart-rate error differences between skin-tone groups to remain below five percentage points. ## Laboratory Validation - The model was trained using synchronized facial video and ECG recordings from 365 diverse participants. - On a separate 104-person test set, PHRM achieved mean absolute percentage error below 10% for every skin-tone group after confidence filtering. - It outperformed 15 leading published remote-PPG models and was the only evaluated model to meet the under-10% target across all groups. ## Real-World Smartphone Study - In an eight-day free-living study, 231 participants used their own phones normally while wearing an ECG chest strap and Fitbit Charge 6. - The research app captured an average of 231 facial clips per participant per day after face unlock. - Participants reviewed and explicitly authorized uploads each day; recordings were sent to secure, encrypted servers. - On a held-out group of 101 participants: - Overall MAPE was 6.09% after confidence gating. - MAPE was 5.04% for light skin, 5.12% for medium skin, and 7.84% for dark skin. - The average underestimation was only 0.64 beats per minute. - The model again outperformed the 15 comparison systems. ## Resting Heart Rate and Research Access - By combining reliable heart-rate estimates throughout the day, PHRM produced daily resting-heart-rate estimates with mean absolute error below five beats per minute compared with a wearable tracker. - Google released the largest and most diverse publicly available smartphone-video dataset for remote PPG research, along with a pre-trained “PHRM-mini” model. - Access is available to qualified researchers. PHRM demonstrates that smartphones could broaden access to passive cardiovascular monitoring without requiring dedicated wearable hardware. Before clinical use, the system would still require continued validation, careful privacy protections, and assessment across additional devices and populations.

Read original

Computer Vision Deep Learning Convolutional Neural Networks Photoplethysmography+3

figmaApr 1, 20263 min readCurated summary

6 Designs That Reimagine How We Interact With Software | Figma Blog

This Figma Make-a-thon showcase argues that software interaction can become more expressive, playful, and socially connective when designers rethink familiar conventions. The winning projects use technology to recreate communal crafts, enable hands-free control, and expand physical experiences beyond traditional limits. Together, they suggest that creativity and emotional connection—not just speed and efficiency—should shape digital experiences. ## Collaborative Embroidery: Common Thread - Charlota Blunárová’s winning project adapts the tradition of handmade samplers into a shared online canvas. - Visitors choose thread colors and stitch types, then contribute to a communal embroidery piece alongside strangers. - The project deliberately imposes constraints: one shared canvas becomes more meaningful as people add to it over time. - More than 100,000 stitches have been added, turning the site into an evolving, collectively authored artifact. - Blunárová built the real-time collaboration and canvas interactions in Figma Make despite having no engineering background. - Her process began with the desired feeling and used visual references, including logos and color palettes created in Figma. ## Hands-Free Interaction: Pucker - Aleyna Çatak’s “Pucker” replaces tapping and voice commands with head movements and a lip gesture. - Users tilt their heads to move through an interface, hold still to select an item, and pucker their lips to confirm. - The concept is designed for situations where users’ hands are occupied, such as cooking, knitting, or designing. - It also points toward more accessible interfaces that can be operated without touch or speech. - Pucker uses a device’s front camera for real-time tracking and states that no data is stored or transmitted. - Çatak describes it as a flexible interaction layer rather than a finished product, potentially adaptable across apps and platforms. - Her advice is to understand basic coding concepts so prototypes can be refined and troubleshot more effectively. ## A Remote Photo Booth: Duet Booth - Paige Latimer reimagines the traditional photo booth as a remote, asynchronous experience. - Duet Booth allows two people in different places—or participating at different times—to take photos that are combined into one photo strip. - The project preserves the photo booth’s sense of immediacy and shared participation while removing its physical and geographic constraints. - Latimer recommends concentrating on the core interaction before polishing visual details, giving the rest of the design a strong foundation. The projects presented in the article demonstrate how Figma Make lowers the barrier to prototyping ambitious ideas. Designers can use it to turn emotional concepts, alternative input methods, and collaborative rituals into working experiences—provided they begin with a clear sense of the feeling or interaction they want to create.

Read original

Computer Vision Figma Real Time Collaboration Figma Make+2

googleMar 24, 20263 min readCurated summary

Mapping the modern world: How S2Vec learns the language of our cities

S2Vec is a self-supervised framework that converts buildings, roads, businesses, and infrastructure into general-purpose geospatial embeddings. By rasterizing these features into S2 Geometry cells and training a masked autoencoder to reconstruct missing areas, it learns the spatial “character” of neighborhoods without manually labeled data. It performs especially well for socioeconomic predictions in geographically unseen regions, while environmental tasks benefit from combining it with satellite imagery. ## Turning Geospatial Data into Images - Geospatial data is multimodal and unevenly distributed: urban blocks may contain hundreds of features, while rural areas contain few. - S2Vec uses hierarchical S2 Geometry cells to divide the Earth into regions at different resolutions. - It counts feature types within each cell—such as buildings, parks, roads, and businesses—and organizes them into multilayered raster images. - This makes complex geographic information compatible with computer vision methods developed for ordinary images. ## Learning with Masked Autoencoding - S2Vec masks portions of the rasterized map and trains a model to reconstruct the missing features from surrounding context. - Repeated training across global locations teaches relationships among urban elements, such as the likelihood of shops near residential buildings and transit stations. - The resulting embeddings are compact numerical representations of each location’s built environment. - Because training is self-supervised, S2Vec does not require worldwide labels for income, air quality, population, or other metrics. - The model can identify similar neighborhood types without being explicitly told concepts such as “financial district” or “suburban residential area.” ## Evaluation and Socioeconomic Performance - S2Vec was compared with models including SATCLIP, GEOCLIP, RS-MaMMUT, Hex2vec, and GeoVeX. - Tests covered population density, median income, carbon emissions, tree cover, and elevation. - Models were evaluated using mean squared error and both: - Interpolation, using random train/test splits - Extrapolation, predicting conditions in geographically unseen regions - S2Vec was generally the strongest individual model for zero-shot socioeconomic prediction, including population density and median income. - It performed competitively with established image-based approaches and exceeded GEOCLIP in the reported comparisons. ## Benefits of Multimodal Fusion - Combining S2Vec with satellite-image embeddings generally produced better results than either modality alone. - Built-environment data captures structures and infrastructure, while satellite imagery adds information about vegetation, terrain, and transportation patterns. - Fusion was particularly valuable for environmental prediction tasks. ## Limitations on Environmental Tasks - Built-environment features alone do not fully explain factors such as tree cover and elevation. - S2Vec was competitive for carbon-emissions prediction but weaker on some environmental metrics. - Satellite imagery embeddings improved performance by supplying information unavailable from counts of buildings, roads, and businesses. S2Vec points toward scalable geographic foundation models that replace task-specific feature engineering with reusable representations. In practice, it is most effective when its built-environment embeddings are combined with complementary imagery, especially for environmental analysis.

Read original

Computer Vision Foundation Models Geospatial Data Self Supervised Learning+4

tossDec 22, 2025Original article

Toss's AI Technology Recognized (opens in new tab)

Toss ML Engineer Jin-woo Lee presents FedLPA, a novel Federated Learning algorithm accepted at NeurIPS 2025 that addresses the critical challenges of data sovereignty and non-uniform data distributions. By allowing AI models to learn from localized data without transferring sensitive information across borders, this research provides a technical foundation for expanding services like Toss Face Pay into international markets with strict privacy regulations. ### The Challenge of Data Sovereignty in Global AI * Traditional AI development requires centralizing data on a single server, which is often impossible due to international privacy laws and data sovereignty regulations. * Federated Learning offers a solution by sending the model to the user’s device (client) rather than moving the data, ensuring raw biometric information never leaves the local environment. * Standard Federated Learning fails in real-world scenarios where data is non-IID (Independent and Identically Distributed), meaning user patterns in different countries or regions vary significantly. ### Overcoming Limitations in Category Discovery * Existing models assume all users share similar data distributions and that all data classes are known beforehand, which leads to performance degradation when encountering new demographics. * FedLPA incorporates Generalized Category Discovery (GCD) to identify both known classes and entirely "novel classes" (e.g., new fraud patterns or ethnic features) that were not present in the initial training set. * This approach prevents the model from becoming obsolete as it encounters new environments, allowing it to adapt to local characteristics autonomously. ### The FedLPA Three-Step Learning Pipeline * **Confidence-guided Local Structure Discovery (CLSD):** The system builds a similarity graph by comparing feature vectors of local data. It refines these connections using "high-confidence" samples—data points the model is certain about—to strengthen the quality of the relational map. * **InfoMap Clustering:** Instead of requiring a human to pre-define the number of categories, the algorithm uses the InfoMap community detection method. This allows the client to automatically estimate the number of unique categories within its own local data through random walks on the similarity graph. * **Local Prior Alignment (LPA):** The model uses self-distillation to ensure consistent predictions across different views of the same data. Most importantly, an LPA regularizer forces the model’s prediction distribution to align with the "Empirical Prior" discovered in the clustering phase, preventing the model from becoming biased toward over-represented classes. ### Business Implications and Strategic Value * **Regulatory Compliance:** FedLPA removes technical barriers to entry for markets like the EU or Southeast Asia by maintaining high model performance while strictly adhering to local data residency requirements. * **Hyper-personalization:** Financial services such as Fraud Detection Systems (FDS) and Credit Scoring Systems (CSS) can be trained on local patterns, allowing for more accurate detection of region-specific scams or credit behaviors. * **Operational Efficiency:** By enabling models to self-detect and learn from new patterns without manual labeling or central intervention, the system significantly reduces the cost and time required for global maintenance. Implementing localized Federated Learning architectures like FedLPA is a recommended strategy for tech organizations seeking to scale AI services internationally while navigating the complex landscape of global privacy regulations and diverse data distributions.

Computer Vision AI Machine Learning Data Privacy+5

googleNov 12, 2025Original article

Separating natural forests from other tree cover with AI for deforestation-free supply chains (opens in new tab)

Researchers from Google DeepMind and Google Research have developed "Natural Forests of the World 2020," an AI-powered global map that distinguishes natural ecosystems from commercial tree plantations. By utilizing high-resolution satellite data and machine learning, the project provides a critical 10-meter resolution baseline to support deforestation-free supply chain regulations like the EUDR. This tool enables governments and companies to monitor biodiversity-rich areas with unprecedented accuracy, ensuring that natural forests are protected from industrial degradation. **The Limitation of Traditional Tree Cover Maps** * Existing maps frequently conflate all woody vegetation into a generic "tree cover" category, leading to "apples-to-oranges" comparisons between different land types. * This lack of distinction makes it difficult to differentiate between the harvesting of short-term plantations and the permanent loss of ancient, biodiversity-rich natural forests. * Precise mapping is now a legal necessity due to regulations like the European Union Regulation on Deforestation-free Products (EUDR), which bans products from land deforested or degraded after December 31, 2020. **The MTSViT Modeling Approach** * To accurately identify forest types, researchers developed the Multi-modal Temporal-Spatial Vision Transformer (MTSViT). * Rather than relying on a single snapshot, the AI "observes" 1280 x 1280 meter patches over the course of a year to identify seasonal, spectral, and textural signatures. * The model integrates multi-modal data, including Sentinel-2 satellite imagery, topographical information (such as elevation and slope), and specific geographical coordinates. * This temporal-spatial analysis allows the AI to recognize the complex patterns of natural forests that distinguish them from the uniform, fast-growing structures of commercial plantations. **Dataset Scale and Global Validation** * The model was trained on a massive dataset comprising over 1.2 million global patches at 10-meter resolution. * The final map provides seamless global coverage, achieving a best-in-class validation accuracy of 92.2% against an independent global dataset. * The research was a collaborative effort involving the World Resources Institute and the International Institute for Applied Systems Analysis to ensure scientific rigor and practical utility. The "Natural Forests of the World 2020" dataset is publicly available via Google Earth Engine and other open repositories. Organizations should leverage this high-resolution baseline to conduct environmental due diligence, support government monitoring, and target conservation efforts in preparation for global climate milestones like COP30.

Computer Vision AI Machine Learning Remote Sensing+3

googleOct 28, 2025Original article

StreetReaderAI: Towards making street view accessible via context-aware multimodal AI (opens in new tab)

StreetReaderAI is a research prototype designed to make immersive street-level imagery accessible to the blind and low-vision community through multimodal AI. By integrating real-time scene analysis with context-aware geographic data, the system transforms visual mapping data into an interactive, audio-first experience. This framework allows users to virtually explore environments and plan routes with a level of detail and independence previously unavailable through traditional screen readers. ### Navigation and Spatial Awareness The system offers an immersive, first-person exploration interface that mimics the mechanics of accessible gaming. * Users navigate using keyboard shortcuts or voice commands, taking "virtual steps" forward or backward and panning their view in 360 degrees. * Real-time audio feedback provides cardinal and intercardinal directions, such as "Now facing North," to maintain spatial orientation. * Distance tracking informs the user how far they have traveled between panoramic images, while "teleport" features allow for quick jumps to specific addresses or landmarks. ### Context-Aware AI Describer At the core of the tool is a subsystem backed by Gemini that synthesizes visual and geographic data to generate descriptions. * The AI Describer combines the current field-of-view image with dynamic metadata about nearby roads, intersections, and points of interest. * Two distinct modes cater to different user needs: a "Default" mode focusing on pedestrian safety and navigation, and a "Tour Guide" mode that provides historical and architectural details. * The system utilizes Gemini to proactively predict and suggest follow-up questions relevant to the specific scene, such as details about crosswalks or building entrances. ### Interactive Dialogue and Session Memory StreetReaderAI utilizes the Multimodal Live API to facilitate real-time, natural language conversations about the environment. * The AI Chat agent maintains a large context window of approximately 1,048,576 tokens, allowing it to retain a "memory" of up to 4,000 previous images and interactions. * This memory allows users to ask retrospective spatial questions, such as "Where was that bus stop I just passed?", with the agent providing relative directions based on the user's current location. * By tracking every pan and movement, the agent can provide specific details about the environment that were captured in previous steps of the virtual walk. ### User Evaluation and Practical Application Testing with blind screen reader users confirmed the system's utility in practical, real-world scenarios. * Participants successfully used the prototype to evaluate potential walking routes, identifying critical environmental features like the presence of benches or shelters at bus stops. * The study highlighted the importance of multimodal inputs—combining image recognition with structured map data—to provide a more accurate and reliable description than image analysis alone could offer. While StreetReaderAI remains a proof-of-concept, it demonstrates that the integration of multimodal LLMs and spatial data can bridge significant accessibility gaps in digital mapping. Future implementation of these technologies could transform how visually impaired individuals interact with the world, turning static street imagery into a functional tool for independent mobility and exploration.

Computer Vision AI Accessibility Gemini+5

googleSep 30, 2025Original article

Introducing interactive on-device segmentation in Snapseed (opens in new tab)

Google has introduced a new "Object Brush" feature in Snapseed that enables intuitive, real-time selective photo editing through a novel on-device segmentation technology. By leveraging a high-performance interactive AI model, users can isolate complex subjects with simple touch gestures in under 20 milliseconds, bridging the gap between professional-grade editing and mobile convenience. This breakthrough is achieved through a sophisticated teacher-student training architecture that prioritizes both pixel-perfect accuracy and low-latency performance on consumer hardware. ### High-Performance On-Device Inference * The system is powered by the Interactive Segmenter model, which is integrated directly into the Snapseed "Adjust" tool to facilitate immediate object-based modifications. * To ensure a fluid user experience, the model utilizes the MediaPipe framework and LiteRT’s GPU acceleration to process selections in less than 20ms. * The interface supports dynamic refinement, allowing users to provide real-time feedback by tracing lines or tapping to add or subtract specific areas of an image. ### Teacher-Student Model Distillation * The development team first created "Interactive Segmenter: Teacher," a large-scale model fine-tuned on 30,000 high-quality, pixel-perfect manual annotations across more than 350 object categories. * Because the Teacher model’s size and computational requirements are prohibitive for mobile use, researchers developed "Interactive Segmenter: Edge" through knowledge distillation. * This distillation process utilized a dataset of over 2 million weakly annotated images, allowing the smaller Edge model to inherit the generalization capabilities of the Teacher model while maintaining a footprint suitable for mobile devices. ### Training via Synthetic User Prompts * To make the model universally capable across all object types, the training process uses a class-agnostic approach based on the Big Transfer (BiT) strategy. * The model learns to interpret user intent through "prompt generation," which simulates real-world interactions such as random scribbles, taps, and lasso (box) selections. * During training, both the Teacher and Edge models receive identical prompts—such as red foreground scribbles and blue background scribbles—to ensure the student model learns to produce high-quality masks even from imprecise user input. This advancement significantly lowers the barrier to entry for complex photo manipulation by moving heavy-duty AI processing directly onto the mobile device. Users can expect a more responsive and precise editing experience that handles everything from fine-tuning a subject's lighting to isolating specific environmental elements like clouds or clothing.

Computer Vision AI Machine Learning On Device Ai+5

googleAug 20, 2025Original article

From massive models to mobile magic: The tech behind YouTube real-time generative AI effects (opens in new tab)

YouTube has successfully deployed over 20 real-time generative AI effects by distilling the capabilities of massive cloud-based models into compact, mobile-ready architectures. By utilizing a "teacher-student" training paradigm, the system overcomes the computational bottlenecks of high-fidelity generative AI while ensuring the output remains responsive on mobile hardware. This approach allows for complex transformations, such as cartoon style transfer and makeup application, to run frame-by-frame on-device without sacrificing the user’s identity. ### Data Curation and Diversity * The foundation of the effects pipeline relies on high-quality, properly licensed face datasets. * Datasets are meticulously filtered to ensure a uniform distribution across different ages, genders, and skin tones. * The Monk Skin Tone Scale is used as a benchmark to ensure the effects work equitably for all users. ### The Teacher-Student Framework * **The Teacher:** A large, powerful pre-trained model (initially StyleGAN2 with StyleCLIP, later transitioning to Google DeepMind’s Imagen) acts as the "expert" that generates high-fidelity visual effects. * **The Student:** A lightweight UNet-based architecture designed for mobile efficiency. It utilizes a MobileNet backbone for both the encoder and decoder to ensure fast frame-by-frame processing. * The distillation process narrows the scope of the massive teacher model into a student model focused on a single, specific task. ### Iterative Distillation and Training * **Data Generation:** The teacher model processes thousands of images to create "before and after" pairs. These are augmented with synthetic elements like AR glasses, sunglasses, and hand occlusions to improve real-world robustness. * **Optimization:** The student model is trained using a sophisticated combination of loss functions, including L1, LPIPS, Adaptive, and Adversarial loss, to balance numerical accuracy with aesthetic quality. * **Architecture Search:** Neural architecture search is employed to tune "depth" and "width" multipliers, identifying the most efficient model structure for different mobile hardware constraints. ### Addressing the Inversion Problem * A major challenge in real-time effects is the "inversion problem," where the model struggles to represent a real face in latent space, leading to a loss of the user's identity (e.g., changes in skin tone or clothing). * YouTube uses Pivotal Tuning Inversion (PTI) to ensure that the user's specific features are preserved during the generative process. * By editing images in the latent space—a compressed numerical representation—the system can apply stylistic changes while maintaining the core characteristics of the original video stream. By combining advanced model distillation with on-device optimization via MediaPipe, YouTube demonstrates a practical path for bringing heavy generative AI research into consumer-facing mobile applications.

Computer Vision AI Generative AI Knowledge Distillation+5

lineAug 12, 2025Original article

The Current State of LY Corporation (opens in new tab)

Tech-Verse 2025 showcased LY Corporation’s strategic shift toward an AI-integrated ecosystem following the merger of LINE and Yahoo Japan. The event focused on the practical hurdles of deploying generative AI, concluding that the transition from experimental models to production-ready services requires sophisticated evaluation frameworks and deep contextual integration into developer workflows. ## AI-Driven Engineering with Ark Developer LY Corporation’s internal "Ark Developer" solution demonstrates how AI can be embedded directly into the software development life cycle. * The system utilizes a Retrieval-Augmented Generation (RAG) based code assistant to handle tasks such as code completion, security reviews, and automated test generation. * Rather than treating codebases as simple text documents, the tool performs graph analysis on directory structures to maintain structural context during code synthesis. * Real-world application includes a seamless integration with GitHub for automated Pull Request (PR) creation, with internal users reporting higher satisfaction compared to off-the-shelf tools like GitHub Copilot. ## Quantifying Quality in Generative AI A significant portion of the technical discussion centered on moving away from subjective "vibes-based" assessments toward rigorous, multi-faceted evaluation of AI outputs. * To measure the quality of generated images, developers utilized traditional metrics like Fréchet Inception Distance (FID) and Inception Score (IS) alongside LAION’s Aesthetic Score. * Advanced evaluation techniques were introduced, including CLIP-IQA, Q-Align, and Visual Question Answering (VQA) based on video-language models to analyze image accuracy. * Technical challenges in image translation and inpainting were highlighted, specifically the difficulty of restoring layout and text structures naturally after optical character recognition (OCR) and translation. ## Global Technical Exchange and Implementation The conference served as a collaborative hub for engineers across Japan, Taiwan, and Korea to discuss the implementation of emerging standards like the Model Context Protocol (MCP). * Sessions emphasized the "how-to" of overcoming deployment hurdles rather than just following technical trends. * Poster sessions (Product Street) and interactive Q&A segments allowed developers to share localized insights on LLM agent performance and agentic workflows. * The recurring theme across diverse teams was that the "evaluation and verification" stage is now the primary driver of quality in generative AI services. For organizations looking to scale AI, the key recommendation is to move beyond simple implementation and invest in "evaluation-driven development." By building internal tools that leverage graph-based context and quantitative metrics like Aesthetic Scores and VQA, teams can ensure that generative outputs meet professional service standards.

Computer Vision AI Large Language Models Model Context Protocol+5

lineMay 21, 2025Original article

How to evaluate AI-generated images? (opens in new tab)

To optimize the Background Person Removal (BPR) feature in image editing services, the LY Corporation AMD team evaluated various generative AI inpainting models to determine which automated metrics best align with human judgment. While traditional research benchmarks often fail to reflect performance in high-resolution, real-world scenarios, this study identifies a framework for selecting models that produce the most natural results. The research highlights that as the complexity and size of the masked area increase, the gap between model performance becomes more pronounced, requiring more sophisticated evaluation strategies. ### Background Person Removal Workflow * **Instance Segmentation:** The process begins by identifying individual pixels to classify objects such as people, buildings, or trees within the input image. * **Salient Object Detection:** This step distinguishes the main subjects of the photo from background elements to ensure only unwanted figures are targeted for removal. * **Inpainting Execution:** Once the background figures are removed, inpainting technology is used to reconstruct the empty space so it blends seamlessly with the surrounding environment. ### Comparison of Inpainting Technologies * **Diffusion-based Models:** These models, such as FLUX.1-Fill-dev, restore damaged areas by gradually removing noise. While they excel at restoring complex details, they are generally slower than GANs and can occasionally generate artifacts. * **GAN-based Models:** Using a generator-discriminator architecture, models like LaMa and HINT offer faster generation speeds and competitive performance for lower-resolution or smaller inpainting tasks. * **Performance Discrepancy:** Experiments showed that while most models perform well on small areas, high-resolution images with large missing sections reveal significant quality differences that are not always captured in standard academic benchmarks. ### Evaluation Methodology and Metrics * **BPR Evaluation Dataset:** The team curated a specific dataset of 10 images with high quality-variance to test 11 different inpainting models released between 2022 and 2024. * **Single Image Quality Metrics:** Evaluated models using LAION Aesthetics score-v2, CLIP-IQA, and Q-Align to measure the aesthetic quality of individual generated frames. * **Preference and Reward Models:** Utilized PickScore, ImageReward, and HPS v2 to determine which generated images would be most preferred by human users. * **Objective:** The goal of these tests was to find an automated evaluation method that minimizes the need for expensive and time-consuming human reviews while maintaining high reliability. Selecting an inpainting model based solely on paper-presented metrics is insufficient for production-level services. For features like BPR, it is critical to implement an evaluation pipeline that combines both aesthetic scoring and human preference models to ensure consistent quality across diverse, high-resolution user photos.

Computer Vision AI Generative AI Diffusion Models+5

googleApr 30, 2025Original article

AMIE gains vision: A research AI agent for multimodal diagnostic dialogue (opens in new tab)

Google Research and DeepMind have introduced multimodal AMIE, an advanced research AI agent designed to conduct diagnostic medical dialogues that integrate text, images, and clinical documents. By building on Gemini 2.0 Flash and a novel state-aware reasoning framework, the system can intelligently request and interpret visual data such as skin photos or ECGs to refine its diagnostic hypotheses. This evolution moves AI diagnostic tools closer to real-world clinical practice, where visual evidence is often essential for accurate patient assessment and management. ### Enhancing AMIE with Multimodal Perception To move beyond text-only limitations, researchers integrated vision capabilities that allow the agent to process complex medical information during a conversation. * The system uses Gemini 2.0 Flash as its core component to interpret diverse data types, including dermatology images and laboratory reports. * By incorporating multimodal perception, the agent can resolve diagnostic ambiguities that cannot be addressed through verbal descriptions alone. * Preliminary testing with Gemini 2.5 Flash suggests that further scaling the underlying model continues to improve the agent's reasoning and diagnostic accuracy. ### Emulating Clinical Workflows via State-Aware Reasoning A key technical contribution is the state-aware phase transition framework, which helps the AI mimic the structured yet flexible approach used by experienced clinicians. * The framework orchestrates the conversation through three distinct phases: History Taking, Diagnosis & Management, and Follow-up. * The agent maintains a dynamic internal state that tracks known information about the patient and identifies specific "knowledge gaps." * When the system detects uncertainty, it strategically requests multimodal artifacts—such as a photo of a rash or an image of a lab result—to update its differential diagnosis. * Transitions between conversation phases are only triggered once the system assesses that the objectives of the current phase have been sufficiently met. ### Evaluation through Simulated OSCEs To validate the agent’s performance, the researchers developed a robust simulation environment to facilitate rapid iteration and standardized testing. * The system was tested using patient scenarios grounded in real-world datasets, including the SCIN dataset for dermatology and PTB-XL for ECG measurements. * Evaluation was conducted using a modified version of Objective Structured Clinical Examinations (OSCEs), the global standard for assessing medical students and professionals. * In comparative studies, AMIE's performance was measured against primary care physicians (PCPs) to ensure its behavior, accuracy, and tone aligned with clinical standards. This research demonstrates that multimodal AI agents can effectively navigate the complexities of a medical consultation by combining linguistic empathy with the technical ability to interpret visual clinical evidence. As these systems continue to evolve, they offer a promising path toward high-quality, accessible diagnostic assistance that mirrors the multimodal nature of human medicine.

Computer Vision AI Large Language Models AI Agents+5

googleApr 7, 2025Original article

Geospatial Reasoning: Unlocking insights with generative AI and multiple foundation models (opens in new tab)

Google Research is introducing Geospatial Reasoning, a new framework that integrates generative AI with specialized foundation models to streamline complex geographical problem-solving. By combining large language models like Gemini with domain-specific data, the initiative seeks to make large-scale spatial analysis accessible to sectors like public health, urban development, and climate resilience. This research effort moves beyond traditional data silos, enabling agentic workflows that can interpret diverse data types—from satellite imagery to population dynamics—through natural language. ### Specialized Foundation Models for Human Activity * The Population Dynamics Foundation Model (PDFM) captures the complex interplay between human behaviors and their local environments. * A dedicated trajectory-based mobility foundation model has been developed to process and analyze movement patterns. * While initially tested in the US, experimental datasets are expanding to include the UK, Australia, Japan, Canada, and Malawi for selected partners. ### Remote Sensing and Vision Architectures * New models utilize advanced architectures including masked autoencoders, SigLIP, MaMMUT, and OWL-ViT, specifically adapted for the remote sensing domain. * Training involves high-resolution satellite and aerial imagery paired with text descriptions and bounding box annotations to enable precise object detection. * The models support zero-shot classification and retrieval, allowing users to locate specific features—such as "residential buildings with solar panels"—using flexible natural language queries. * Internal evaluations show state-of-the-art performance across multiple benchmarks, including image segmentation and post-disaster damage assessment. ### Agentic Workflows and Industry Collaboration * The Geospatial Reasoning framework utilizes LLMs like Gemini to manage complex datasets and orchestrate "agentic" workflows. * These workflows are grounded in geospatial data to ensure that the insights generated are both useful and contextually accurate. * Google is collaborating with inaugural industry partners, including Airbus, Maxar, Planet Labs, and WPP, to test these capabilities in real-world scenarios. Organizations interested in accelerating their geospatial analysis should consider applying for the trusted tester program to explore how these foundation models can be fine-tuned for specific proprietary data and use cases.

Computer Vision AI Generative AI Gemini+5