Multi Agent Architecture

googleAug 11, 20263 min readCurated summary

Advancing AMIE towards expert-level audio-visual clinical consultations

AMIE (Video) is Google’s real-time audiovisual medical AI system, designed to overcome the limitations of text-only clinical conversations. Built on Gemini and Project Astra, it observes visual and auditory cues, guides patients through virtual examinations, and performs diagnostic reasoning during live consultations. In a randomized study involving 300 simulated consultations, the system was evaluated against text-only AMIE and board-certified primary care physicians. ## Why Audio-Visual Consultation Matters - Traditional text-based systems lose important clinical information, including: - Gait and visible physical symptoms - Breathing patterns and signs of distress - Vocal and auditory cues - Patient responses during physical examination maneuvers - Requiring patients to describe symptoms in writing can reduce diagnostic accuracy, particularly for people with limited digital or health literacy. - Audiovisual interaction may also improve trust, communication, and access to medical expertise. ## AMIE’s Broader Development - Earlier versions of AMIE demonstrated expert-level performance in: - Text-based diagnostic dialogue - Differential diagnosis support - Disease treatment and longitudinal management - Specialist evaluations in oncology, cardiology, and ophthalmology - Reasoning over medical images and clinical documents - Google has also explored physician oversight and real-world clinical feasibility studies. ## Asynchronous Multi-Agent Architecture AMIE (Video) divides the consultation among three agents operating in parallel: - **Talker agent** - Maintains natural, low-latency spoken conversation. - Incorporates information and recommendations from the other agents. - **Planner agent** - Performs deeper clinical reasoning in the background. - Updates differential diagnoses and management plans. - Identifies missing information and reprioritizes clinical objectives. - **Perception agent** - Continuously analyzes audio and video. - Detects non-verbal findings such as visible distress, physical signs, and auditory abnormalities. - Interprets observations in the context of the conversation. This separation allows AMIE to reason deeply without creating long conversational pauses. Automated tests indicated that the agents contributed to improvements in history-taking, clinical reasoning, treatment recommendations, communication quality, and response latency. ## Automated Evaluation Framework - Google created a taxonomy of audiovisual clinical competencies based on medical literature. - The taxonomy covered: - Non-verbal visual cues - Auditory signals - Physical examination maneuvers - The evaluation suite included: - Single-turn tests targeting specific perception and reasoning abilities - Multi-turn simulated consultations assessing complete conversational performance - Simulations injected visual findings as textual descriptions, such as a patient holding handwriting samples up to the camera. - These tests helped identify capabilities and failure modes before human evaluation. ## Randomized Video Study - The study used a synchronous video consultation interface and an Objective Structured Clinical Examination format. - It included: - 100 clinical scenarios - Five body systems: cardiopulmonary, abdominal, HEENT, neurological/psychiatric, and musculoskeletal - 15 trained patient actors - 300 standardized consultations - Three study arms were compared: - **AMIE (Video):** Real-time audiovisual consultations - **AMIE (Text):** Text-only AMIE used to isolate the value of audiovisual capabilities - **PCP (Video):** Board-certified primary care physicians using the same video interface - An independent panel of 20 experienced primary care physicians assessed the consultations using established clinical rubrics. AMIE (Video) represents a move from text-based medical dialogue toward interactive, multimodal consultations. Its multi-agent design and audiovisual perception are intended to preserve conversational responsiveness while supporting richer clinical reasoning, though the reported findings come from simulated consultations and require further validation in real-world clinical care.

Read original

Multi Agent Architecture Gemini Multimodal AI Telehealth+3

googleFeb 3, 20263 min readCurated summary

Collaborating on a nationwide randomized study of AI in real-world virtual care

Google and Included Health plan to launch, pending IRB approval, a nationwide randomized study of conversational AI in real-world virtual care. Unlike prior simulated or small feasibility studies, it will prospectively evaluate AI with consented patients across varied conditions and locations, comparing it with standard clinical practice. The goal is to generate rigorous evidence about safety, usefulness, limitations, and impact on patients and clinicians. ## Moving from Simulation to Real-World Evaluation - Earlier research demonstrated clinician-level capabilities in simulated consultations and retrospective analyses. - A feasibility study with Beth Israel Deaconess Medical Center began testing conversational AI in clinical workflows, using measures such as safety-supervisor interruptions. - The new study will advance beyond feasibility through: - A randomized controlled design - Nationwide recruitment - Consented participants - Real patients, clinical concerns, and virtual-care workflows - Controlled comparison with standard practice ## A Phased Approach to Medical AI Research - Google argues that medical AI should be evaluated with evidence standards similar to other medical interventions. - Each research phase adds information about: - Patient and clinician experiences - Safety - Usefulness - The AI system’s capabilities and limitations - Results from each stage are intended to guide safer, more responsible development and deployment. ## Foundational Research Behind the Study ### Diagnostic and Management Reasoning - The AMIE system was developed to handle medical interviews and clinical reasoning. - Studies with patient actors and synthetic cases found that AMIE could match or exceed primary care physicians in simulated diagnostic accuracy and conversation quality. - Later work expanded the system to: - Longitudinal disease management - Clinical-guideline and patient-history reasoning - Investigation and treatment planning - Interpretation of multimodal evidence ### Personalized Health Insights - Research on the Personal Health Agent examined how AI could interpret personal health data, including sleep and activity information from wearables. - Its multi-agent architecture combined the roles of: - Data scientist - Medical domain expert - Health coach - This work informed Fitbit Labs tools such as Symptom Checker and Medical Records Navigator and Plan for Care. ### Navigating Health Information - Google’s “wayfinding” AI research explored how conversational agents can help people find and understand health information. - The system uses proactive guidance, goal recognition, and tailored conversations to make health information searches more practical and useful. ## Practical Conclusion The partnership with Included Health represents a transition from demonstrating what medical AI can do in controlled environments to measuring how it performs at scale in actual care. A nationwide randomized trial could provide the evidence needed to determine whether conversational AI can safely improve virtual care and expand access to medical expertise.

Read original

Multi Agent Architecture Conversational Ai Healthcare Ai Diagnostic Reasoning+3