Telehealth | Techlist.io

googleAug 11, 20263 min readCurated summary

Advancing AMIE towards expert-level audio-visual clinical consultations

AMIE (Video) is Google’s real-time audiovisual medical AI system, designed to overcome the limitations of text-only clinical conversations. Built on Gemini and Project Astra, it observes visual and auditory cues, guides patients through virtual examinations, and performs diagnostic reasoning during live consultations. In a randomized study involving 300 simulated consultations, the system was evaluated against text-only AMIE and board-certified primary care physicians. ## Why Audio-Visual Consultation Matters - Traditional text-based systems lose important clinical information, including: - Gait and visible physical symptoms - Breathing patterns and signs of distress - Vocal and auditory cues - Patient responses during physical examination maneuvers - Requiring patients to describe symptoms in writing can reduce diagnostic accuracy, particularly for people with limited digital or health literacy. - Audiovisual interaction may also improve trust, communication, and access to medical expertise. ## AMIE’s Broader Development - Earlier versions of AMIE demonstrated expert-level performance in: - Text-based diagnostic dialogue - Differential diagnosis support - Disease treatment and longitudinal management - Specialist evaluations in oncology, cardiology, and ophthalmology - Reasoning over medical images and clinical documents - Google has also explored physician oversight and real-world clinical feasibility studies. ## Asynchronous Multi-Agent Architecture AMIE (Video) divides the consultation among three agents operating in parallel: - **Talker agent** - Maintains natural, low-latency spoken conversation. - Incorporates information and recommendations from the other agents. - **Planner agent** - Performs deeper clinical reasoning in the background. - Updates differential diagnoses and management plans. - Identifies missing information and reprioritizes clinical objectives. - **Perception agent** - Continuously analyzes audio and video. - Detects non-verbal findings such as visible distress, physical signs, and auditory abnormalities. - Interprets observations in the context of the conversation. This separation allows AMIE to reason deeply without creating long conversational pauses. Automated tests indicated that the agents contributed to improvements in history-taking, clinical reasoning, treatment recommendations, communication quality, and response latency. ## Automated Evaluation Framework - Google created a taxonomy of audiovisual clinical competencies based on medical literature. - The taxonomy covered: - Non-verbal visual cues - Auditory signals - Physical examination maneuvers - The evaluation suite included: - Single-turn tests targeting specific perception and reasoning abilities - Multi-turn simulated consultations assessing complete conversational performance - Simulations injected visual findings as textual descriptions, such as a patient holding handwriting samples up to the camera. - These tests helped identify capabilities and failure modes before human evaluation. ## Randomized Video Study - The study used a synchronous video consultation interface and an Objective Structured Clinical Examination format. - It included: - 100 clinical scenarios - Five body systems: cardiopulmonary, abdominal, HEENT, neurological/psychiatric, and musculoskeletal - 15 trained patient actors - 300 standardized consultations - Three study arms were compared: - **AMIE (Video):** Real-time audiovisual consultations - **AMIE (Text):** Text-only AMIE used to isolate the value of audiovisual capabilities - **PCP (Video):** Board-certified primary care physicians using the same video interface - An independent panel of 20 experienced primary care physicians assessed the consultations using established clinical rubrics. AMIE (Video) represents a move from text-based medical dialogue toward interactive, multimodal consultations. Its multi-agent design and audiovisual perception are intended to preserve conversational responsiveness while supporting richer clinical reasoning, though the reported findings come from simulated consultations and require further validation in real-world clinical care.

Read original

Telehealth Gemini Multimodal AI Multi Agent Architecture+3

googleMar 17, 20263 min readCurated summary

Google Research at The Check Up: from healthcare innovation to real-world care settings

Google Research argues that AI is entering a new phase in healthcare: moving beyond isolated tools toward personalized care, clinical collaboration, public-health planning, and scientific discovery. The post highlights research partnerships, open models, and real-world deployments designed to make healthcare more accurate, accessible, and proactive. Google emphasizes that these advances must be developed responsibly through clinical validation, peer review, and collaboration with healthcare institutions. ## AI for Personalized Healthcare - A Fitbit collaboration studied how AI could support preventative care across the United States. - The research found that a Personal Health Agent (PHA) modeled on a collaborative health team could provide more effective long-term support than single-purpose fitness or tracking apps. - The PHA combines: - Data analysis - Medical and domain expertise - Health coaching - Large multimodal models can transform wearable data into personalized guidance about sleep, fitness, and overall health. ## AI as a Clinical Collaborator - Google’s breast cancer research with Imperial College London and the UK’s NHS used diverse datasets and expert-validated ground truth data. - The experimental system identified 25% of “interval cancers”—cancers missed during screening and later detected after symptoms appeared. - Integrated into clinical workflows, the system could reduce radiologists’ workload while maintaining safe detection performance. - Google’s diabetic retinopathy screening model has been deployed through partnerships with medical institutions in India, Thailand, and Australia. - It has supported more than one million screenings. - Patients can receive results in roughly two minutes. - AMIE, a multi-agent medical AI system, can reason across medical histories, laboratory results, and medical images to identify overlooked patterns. - Google is testing AMIE with Beth Israel Deaconess Medical Center to assist with pre-visit history-taking and flag urgent symptoms. - An IRB-approved national study with Included Health will evaluate AI-supported telehealth care. ## Open Models for Healthcare Developers - Google’s Health AI Developer Foundations (HAI-DEF) provides free open-weight models and open-source tools for building healthcare applications. - MedGemma supports: - Medical text and image interpretation - High-dimensional 3D imaging - Medical-specific speech recognition - The All India Institute of Medical Sciences is using MedGemma for outpatient triage and dermatology screening. - Singapore’s Ministry of Health is adapting the model for locally relevant primary- and specialty-care applications. - The MedGemma Impact Challenge received more than 850 submissions aimed at turning AI research into practical, human-centered healthcare tools. ## AI for Public Health - Google Earth AI combines geospatial models and datasets to study connections between environmental conditions, population behavior, and health outcomes. - Researchers at Mount Sinai and Boston Children’s Hospital/Harvard used Google data and surveys to estimate childhood MMR vaccination coverage at ZIP-code resolution. - The resulting “super-resolution” maps identified pockets of under-vaccination that corresponded with recent measles outbreaks. - Such analysis could help public-health officials target outreach and prevention efforts more effectively. ## AI for Biomedical Discovery - Co-Scientist and Gemini Deep Think are being used to generate scientific hypotheses and support research across fields including single-cell analysis, public health, and neuroscience. - Google is also exploring evolutionary coding agents that run scientific-computing experiments in parallel. - DeepSomatic, a genomic analysis tool, is designed to improve the detection of cancer-related genetic mutations across multiple cancer types. Google’s broader recommendation is to treat AI as a validated collaborator and infrastructure layer rather than a replacement for clinicians or researchers. Continued clinical testing, expert oversight, transparent publication, and open developer access will be essential to translating these systems into safe, practical benefits.

Read original

Telehealth AI Data Science Public Health+4

googleMar 11, 20263 min readCurated summary

Exploring the feasibility of conversational diagnostic AI in a real-world clinical study

The study evaluated Google’s conversational medical AI, AMIE, in a real-world primary care workflow rather than simulated cases. In a prospective, IRB-approved study at Beth Israel Deaconess Medical Center, AMIE conducted supervised pre-visit history-taking with 100 patients. Results suggested that supervised deployment was feasible and conversationally safe, while AMIE’s diagnostic and management-plan quality was broadly comparable to that of primary care physicians, with physicians performing better on practicality and cost effectiveness. ## Study Design and Clinical Workflow - Patients with new, non-emergency, episodic complaints used AMIE through a secure web link before an in-person or telehealth appointment. - A physician supervised each AI-patient interaction through live video and screen-sharing. - AMIE produced a transcript and summary for the patient’s primary care physician. - Independent clinical evaluators assessed: - The quality of the AMIE conversation - AMIE’s differential diagnoses - AMIE’s management plans - Comparable outputs from physicians - The study was prospective, single-center, single-arm, pre-registered, and IRB approved. ## Participants - 100 adults completed the AMIE interaction. - 98 attended their scheduled primary care appointments. - Participants represented varied ages, racial and ethnic groups, health literacy, technology literacy, and prior chatbot experience. - Compared with all 1,452 urgent care visits during the study period, participants tended to be younger, although the sample reflected the broader population’s female and white demographic skew. ## Safety Oversight - Human supervisors could stop an interaction if they observed: - Immediate risk of harm to the patient or others - Significant emotional distress related to the AI interaction - Potential clinical harm - A patient’s explicit request to end the session - No safety stops were required across the study. - The authors interpret this as evidence that supervised AMIE interactions were conversationally safe in this setting. ## Clinical Reasoning Performance - Three independent clinical evaluators reviewed each case using blinded, randomized assessments. - AMIE and physicians showed similar overall quality for: - Differential diagnoses - Management plans - Management-plan appropriateness and safety - Physicians performed better on the practicality and cost effectiveness of management plans. - AMIE’s differential-diagnosis accuracy was reported as high, including cases where the final diagnosis was confirmed through diagnostic testing. ## Patient and Clinician Experience - The study measured trust, perceptions, and acceptance among both patients and clinicians. - Patient trust in AI increased after interacting with AMIE. - Overall findings indicated that the system was well received within the supervised pre-visit workflow. The study supports cautious, supervised testing of conversational diagnostic AI in clinical environments. It does not establish that AMIE can independently replace clinicians; rather, it suggests that pre-visit information gathering may be a practical early use case, provided rigorous oversight, safety protocols, and further evaluation in larger and more diverse settings.

Read original

Telehealth Conversational Ai Chatbot Differential Diagnosis+4