Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
OAAgent: A Multimodal LLM Agent Clinical Assistant for Precision Osteoarthritis Care
DescriptionOsteoarthritis (OA) is a chronic condition which affects over 300 million people globally and is a leading cause of disability, yet predictive models often remain monomodal, static, and opaque to clinicians. This dissertation develops OAAgent, a multimodal large language model (LLM) clinical assistant that integrates medical images (X-ray, MRI), longitudinal clinical variables, and physician notes for personalized, interpretable OA care and prediction of progression. OAAgent employs a fusion transformer for multimodal integration, a temporal retrieval system C-TRAG with explicit cross-visit semantics to retrieve clinically similar cases (similar past trajectories), and reinforcement learning for dynamic decision-making through a Chain-of-Thought reasoning layer and Extract-and-Abstract clinical note summarization to ensure transparent, patient-specific recommendations. The dissertation addresses five critical gaps at the intersection of OA AI research:
1. Joint integration of heterogeneous modalities
2. Longitudinal temporal reasoning
3. Clinically interpretable decision support
4. Personalized treatment recommendations
5. Inclusion of underutilized narrative notes
OAAgent’s architecture is designed for extensibility through the Model Context Protocol (MCP), enabling it to interoperate with other domain-specific models, multimodal pipelines, and external reasoning agents. This creates a bridge between the LLM core and diverse analytical components, enhancing adaptability to new modalities and clinical contexts.
Developed in collaboration with the Cleveland Clinic and validated on the OAI dataset and FNIH cohort and MIMIC datasets, OAAgent demonstrates improved accuracy, temporal calibration, and interpretability. Anchored in a Trustworthy AI framework, this work advances agentic multimodal AI for healthcare, offering a scalable, ethical, and interoperable pathway toward equitable, explainable clinical decision support across chronic diseases.
1. Joint integration of heterogeneous modalities
2. Longitudinal temporal reasoning
3. Clinically interpretable decision support
4. Personalized treatment recommendations
5. Inclusion of underutilized narrative notes
OAAgent’s architecture is designed for extensibility through the Model Context Protocol (MCP), enabling it to interoperate with other domain-specific models, multimodal pipelines, and external reasoning agents. This creates a bridge between the LLM core and diverse analytical components, enhancing adaptability to new modalities and clinical contexts.
Developed in collaboration with the Cleveland Clinic and validated on the OAI dataset and FNIH cohort and MIMIC datasets, OAAgent demonstrates improved accuracy, temporal calibration, and interpretability. Anchored in a Trustworthy AI framework, this work advances agentic multimodal AI for healthcare, offering a scalable, ethical, and interoperable pathway toward equitable, explainable clinical decision support across chronic diseases.
