Adaptive Interfaces

2 posts

figma3 min readCurated summary

What Does the Future of Software Look Like? | Figma Blog

AI may reshape software around more human, contextual interactions rather than fixed menus and mechanical commands. The post argues that future interfaces could understand intent through voice, gesture, emotion, and situation, adapting their behavior to each person. Instead of forcing users to adapt to increasingly powerful systems, software could meet users where they are while encouraging focus, presence, and healthier technology habits. ## Ephemeral Tools - Controls appear only when users select an object and indicate what they want to do. - Contextual options replace persistent menus, panels, and modes. - A video editor, for example, might show timing, pacing, alternate cuts, and sound options around a selected clip. - This lets creators focus on decisions and intent rather than remembering how software is organized. ## Magic Marker - Users interact through a combination of voice, cursor movement, gestures, sound effects, and body language. - Someone could circle an object, drag it into position, and verbally request a change. - AI would interpret these signals together, making it feel more like collaborating with a teammate. - This reduces the need for precise prompt engineering or complex document references. ## Adaptive Presence - Intelligent systems adjust their communication style and level of assistance based on user behavior. - They might offer structured guidance when someone is confused, step back when help is unnecessary, or switch between text, voice, and visuals. - Software could change pacing, simplify language, and divide information into smaller steps. - This approach is especially valuable in healthcare and education, where differences in user readiness can have serious consequences. ## Empathetic Flows - Interfaces could infer emotional states from typing speed, stylus pressure, speech patterns, facial expressions, and repeated revisions. - A food app might reduce choices when someone appears overwhelmed. - A creative tool could become quiet when the user is concentrating, while a hotel app might stop promoting upgrades when the guest seems tired. - Rather than requiring users to explicitly state what they need, systems would respond to behavioral signals. ## Situational Cues - Sound, motion, pacing, progress indicators, and visual transitions can help users understand where they are in an experience. - Earlier digital products used cues such as dial-up sounds, progress bars, and “You’ve got mail” announcements to provide orientation. - Future interfaces should counteract the overstimulation caused by attention-driven notifications. - Persistent progress indicators, transition sounds, and consistent visual language could help users regulate their attention and nervous systems. ## Spatial Tuning - Users could control software through bodily movement instead of conventional tapping and clicking. - Examples include shaping music with hand movements, navigating augmented reality by changing body orientation, or adjusting design elements through gestures. - These interactions demand attention and presence, making them harder to rush or automate. - Technology becomes an experience that intentionally slows users down rather than continually rewarding speed. ## Mash-Ups - Future systems could combine any two inputs—files, objects, sounds, locations, or physical gestures—to create something new. - The system would synthesize the combined inputs while blending their structure, tone, and meaning. - Possible examples include merging a playlist with a city map or combining digital objects through touch or gestures. The overall recommendation is to design AI-powered software around human intent, context, emotion, and physical presence. The most successful future interfaces may be those that make technology feel less like a collection of controls and more like an adaptable, considerate collaborator.

Read original(opens in new tab)
google3 min readCurated summary

How AI tools can redefine universal design to increase accessibility

Google Research proposes Natively Adaptive Interfaces (NAI), a framework that uses multimodal and agentic AI to make interfaces adapt to individual users rather than forcing everyone into a fixed design. Developed through co-design with disability communities, NAI aims to reduce the accessibility gap by embedding assistive capabilities directly into products. Early prototypes suggest that personalized, context-aware interfaces can improve experiences for disabled users while also benefiting the broader population. ## Community-led co-design - Google follows the principle “Nothing About Us, Without Us,” involving people with disabilities as co-designers from the beginning. - Partnerships include RIT/NTID, The Arc of the United States, RNID, and Team Gleason. - These collaborations focus on real-world barriers and recognize the expertise of disability communities. - The approach also aims to create employment and economic opportunities for people who help shape the technology. ## Moving from reactive accessibility to adaptive interfaces - Google identifies an “accessibility gap” between the release of new features and the development of compatible assistive tools. - NAI addresses this by making accessibility native to the interface instead of adding it afterward. - Static navigation is replaced with dynamic, agent-driven modules that can interpret context and adjust the experience. ## Multi-system agents - An Orchestrator maintains shared context and delegates tasks to specialized sub-agents. - A Summarization Agent breaks down complex documents and assigns subtasks to expert agents. - A Settings Agent dynamically adjusts interface elements such as text size. - This structure lets users accomplish tasks without navigating complicated menus or searching for the right control. ## Multimodal interaction - Gemini-based prototypes combine voice, vision, and text rather than limiting accessibility to text-to-speech. - Live video can be converted into interactive audio descriptions. - Users can ask follow-up questions about specific visual details as events unfold. - Conversational interaction provides situational awareness and may reduce cognitive load. ## Proven prototypes - **StreetReaderAI** - Supports blind and low-vision users navigating physical spaces. - Combines an AI Describer that analyzes visual and geographic information with an AI Chat system for questions. - Maintains context so users can ask about previously encountered locations, such as the position of a bus stop. - **Multimodal Agent Video Player (MAVP)** - Makes audio description interactive rather than static. - Users can change the level of detail or ask questions during playback. - Uses an offline “dense index” of visual descriptions and retrieval-augmented generation (RAG) for fast responses. - **Grammar Laboratory** - Developed by RIT/NTID with Google.org support for American Sign Language and English learners. - Provides grammar instruction through ASL videos, English captions, spoken narration, and written transcripts. - Uses adaptive AI to customize lessons according to each student’s language preferences and interactions. ## The curb-cut effect - Accessibility features designed for people with significant constraints can benefit many other users. - Voice interfaces created for blind users may help sighted people who are multitasking. - AI synthesis and learning tools designed for people with learning disabilities can also support users who want information presented more clearly or flexibly. - NAI therefore treats accessibility as a source of better universal design, not as a specialized add-on. NAI’s central recommendation is to build accessibility into interfaces from the start, using multimodal AI, persistent context, and community-led design. The most effective systems will adapt to users while remaining accountable to the people whose needs they are intended to serve.

Read original(opens in new tab)