Rethinking Patient Education as Multi-turn Multi-modal Interaction

arXiv cs.AI / 4/17/2026

📰 NewsSignals & Early TrendsTools & Practical UsageModels & Research

共有:

Key Points

The paper argues that patient education in radiology is more complex than typical static medical multimodal tasks because it must ground explanations in evidence, guide what patients should look at, and adapt to confusion or distress.
It introduces MedImageEdu, a new benchmark designed for multi-turn, evidence-grounded radiology patient education using a DoctorAgent–PatientAgent interaction framework with a hidden patient profile (e.g., education level and health literacy).
The system can generate visual support by issuing drawing instructions grounded in the radiology report, case images, and the patient’s question, then uses returned images to produce a final multimodal response with accessible explanations.
Evaluations across multiple vision-language model agents show three recurring issues: language fluency can exceed faithful visual grounding, safety is the weakest dimension across disease categories, and emotionally tense interactions are particularly challenging.
MedImageEdu is positioned as a controlled testbed to assess whether multimodal agents can teach from evidence rather than only produce text-based answers.

Abstract

Most medical multimodal benchmarks focus on static tasks such as image question answering, report generation, and plain-language rewriting. Patient education is more demanding: systems must identify relevant evidence across images, show patients where to look, explain findings in accessible language, and handle confusion or distress. Yet most patient education work remains text-only, even though combined image-and-text explanations may better support understanding. We introduce MedImageEdu, a benchmark for multi-turn, evidence-grounded radiology patient education. Each case provides a radiology report with report text and case images. A DoctorAgent interacts with a PatientAgent, conditioned on a hidden profile that captures factors such as education level, health literacy, and personality. When a patient question would benefit from visual support, the DoctorAgent can issue drawing instructions grounded in the report, case images, and the current question to a benchmark-provided drawing tool. The tool returns image(s), after which the DoctorAgent produces a final multimodal response consisting of the image(s) and a grounded plain-language explanation. MedImageEdu contains 150 cases from three sources and evaluates both the consultation process and the final multimodal response along five dimensions: Consultation, Safety and Scope, Language Quality, Drawing Quality, and Image-Text Response Quality. Across representative open- and closed-source vision-language model agents, we find three consistent gaps: fluent language often outpaces faithful visual grounding, safety is the weakest dimension across disease categories, and emotionally tense interactions are harder than low education or low health literacy. MedImageEdu provides a controlled testbed for assessing whether multimodal agents can teach from evidence rather than merely answer from text.