google·Mar 25, 2026·3 min readCurated summary

Vibe Coding XR: Accelerating AI + XR prototyping with XR Blocks and Gemini

Vibe Coding XR combines Gemini’s natural-language coding capabilities with the open-source XR Blocks framework to rapidly create interactive, physics-aware WebXR applications. Users can describe an experience—such as a dandelion, physics lab, or educational visualization—and receive a working Android XR prototype in under 60 seconds. The workflow supports both desktop simulation and deployment to Android XR headsets, making spatial prototyping faster and more accessible. ## Bridging AI Prototyping and XR - Traditional XR development requires fragmented perception systems, game engines, and low-level sensor integrations. - Vibe-coded prototypes let developers quickly evaluate 3D interfaces, spatial interactions, and visualizations before investing in full production. - The workflow is designed for both experienced developers and creators without prior XR expertise. - Gemini translates natural-language prompts into functional XR applications with scene setup, perception, interaction, and physics logic. ## The Vibe Coding XR Workflow - Users open the XR Blocks Gem in Chrome on an Android XR headset or desktop. - They provide a prompt by typing or using voice, such as “Create a beautiful dandelion.” - Gemini plans and implements the experience using XR Blocks examples and templates. - On Android XR, users can enter the experience with a pinch gesture and interact naturally—for example, pinching to blow away an animated dandelion. - Applications can be published through a shareable public link. - Desktop Chrome provides a simulated-reality environment for testing before deployment, while Android XR enables advanced features such as hand tracking, depth sensing, and physics. ## Technical Foundation - XR Blocks is built on WebXR, three.js, and LiteRT.js. - Its engine coordinates: - Environmental perception - XR interaction - Spatial computing - AI integration - Gemini receives a specialized system prompt containing: - XR design guidelines for room-scale environments, spatial layout, scale, and interaction distances - Package-management rules and recommended styles - Curated source code, templates, and working samples - Grounding Gemini in valid XR Blocks APIs reduces hallucinated code and encourages consistent implementation patterns. ## Educational and Interactive Applications - **Math tutor:** Visualizes Euler’s theorem using tetrahedra, cubes, and octahedra, with pinch-based highlighting of vertices, edges, and faces. - **Physics lab:** Lets users pick up and place labeled weights on a balance scale to learn about equilibrium. - **Immersive chemistry:** Simulates methane, ethylene, and acetylene combustion with educational cards and volumetric effects, offering a safer mixed-reality alternative to physical experiments. - **Schrödinger’s cat:** Uses pinch and proximity interactions to demonstrate superposition, revealing alive and dead versions of a cat before collapsing the state into one outcome. - **XR sports:** Generates interactive experiences such as hand-based volleyball, including textured balls, environmental collision, and adjustable launch behavior. ## Practical Value - Creators can test spatial ideas in minutes rather than building complete XR pipelines first. - The same prototype can be evaluated on desktop and then experienced with body and hand interactions on Android XR. - The approach is especially useful for education, interaction design, scientific visualization, and early-stage product exploration. Vibe Coding XR is best viewed as a rapid experimentation layer rather than a replacement for production XR engineering. By combining Gemini’s reasoning with XR Blocks’ specialized runtime and templates, it significantly lowers the barrier to creating and validating intelligent spatial experiences.

Read original(opens in new tab)

WebxrGeminiVibe CodingSpatial Computing+4

google·Oct 8, 2025Original article

XR Blocks: Accelerating AI + XR innovation (opens in new tab)

XR Blocks is an open-source, cross-platform framework designed to bridge the technical gap between mature AI development ecosystems and high-friction extended reality (XR) prototyping. By providing a modular architecture and high-level abstractions, the toolkit enables creators to rapidly build and deploy intelligent, immersive web applications without managing low-level system integration. Ultimately, the framework empowers developers to move from concept to interactive prototype across both desktop simulators and mobile XR devices using a unified codebase. ### Core Design Principles * **Simplicity and Readability:** Drawing inspiration from the "Zen of Python," the framework prioritizes human-readable abstractions where a developer’s script reflects a high-level description of the experience rather than complex boilerplate code. * **Creator-Centric Workflow:** The architecture is designed to handle the "plumbing" of XR—such as sensor fusion, AI model integration, and cross-platform logic—allowing creators to focus entirely on user interaction and experience. * **Pragmatic Modularity:** Rather than attempting to be a perfect, all-encompassing system, XR Blocks favors an adaptable and simple architecture that can evolve alongside the rapidly changing fields of AI and spatial computing. ### The Reality Model Abstractions * **The Script Primitive:** Acts as the logical center of an application, separating the "what" of an interaction from the "how" of its underlying technical implementation. * **User and World:** Provides built-in support for tracking hands, gaze, and avatars while allowing the system to query the physical environment for depth, estimated lighting conditions, and object recognition. * **AI and Agents:** Facilitates the integration of intelligent assistants, such as the "Sensible Agent," which can provide proactive, context-aware suggestions within the XR environment. * **Virtual Interfaces:** Offers tools to augment blended reality with virtual UI elements that respond to the user's physical context. ### Technical Implementation and Integration * **Web-Based Foundation:** The framework is built upon accessible, standard technologies including WebXR, three.js, and LiteRT (formerly TFLite) to ensure a low barrier to entry for web developers. * **Advanced AI Support:** It features native integration with Gemini for high-level reasoning and context-aware applications. * **Cross-Platform Deployment:** Developers can prototype depth-aware, physics-based interactions in a desktop simulator and deploy the exact same code to Android XR devices. * **Open-Source Resources:** The project includes a comprehensive suite of templates and live demos covering specific use cases like depth mapping, gesture modeling, and lighting estimation. By lowering the barrier to entry for intelligent XR development, XR Blocks serves as a practical starting point for researchers and developers aiming to explore the next generation of human-centered computing. Interested creators can access the source code on GitHub to begin building immersive, AI-driven applications that function seamlessly across the web and specialized XR hardware.

WebxrAIGeminiLitert+4

google·Sep 17, 2025Original article

Sensible Agent: A framework for unobtrusive interaction with proactive AR agents (opens in new tab)

Sensible Agent is a research prototype designed to move AR agents beyond explicit voice commands toward proactive, context-aware assistance. By leveraging real-time multimodal sensing of a user's environment and physical state, the framework ensures digital help is delivered unobtrusively through the most appropriate interaction modalities. This approach fundamentally reshapes human-computer interaction by anticipating user needs while minimizing cognitive and social disruption. ## Contextual Understanding via Multimodal Parsing The framework begins by analyzing the user's immediate surroundings to establish a baseline for assistance. * A Vision-Language Model (VLM) processes egocentric camera feeds from the AR headset to identify high-level activities and locations. * YAMNet, a pre-trained audio event classifier, monitors environmental noise levels to determine if audio feedback is appropriate. * The system synthesizes these inputs into a parsed context that accounts for situational impairments, such as when a user’s hands are occupied. ## Reasoning with Proactive Query Generation Once the context is established, the system determines the specific type of assistance required through a sophisticated reasoning process. * The framework uses chain-of-thought (CoT) reasoning to decompose complex problems into intermediate logical steps. * Few-shot learning, guided by examples from data collection studies, helps the model decide between actions like providing translations or displaying a grocery list. * The generator outputs a structured suggestion that includes the specific action, the query format (e.g., binary choice or icons), and the presentation modality (visual, audio, or both). ## Dynamic Modality and Interaction Management The final stage of the framework manages how the agent communicates with the user and how the user can respond without breaking their current flow. * The prototype, built on Android XR and WebXR, utilizes a UI Manager to render visual panels or generate text-to-speech (TTS) prompts based on the agent's decision. * An Input Modality Manager activates the most discreet response methods available, such as head gestures (nods), hand gestures (thumbs up), or gaze tracking. * This adaptive selection ensures that if a user is in a noisy room or a social setting, the agent can switch from verbal interaction to subtle visual cues and gesture-based confirmations. By prioritizing social awareness and context-sensitivity, Sensible Agent provides a blueprint for AR systems that feel like helpful companions rather than intrusive tools. Implementing such frameworks is essential for making proactive digital assistants practical and acceptable for long-term, everyday use in public and private spaces.

WebxrAIMultimodal AIAugmented Reality+5