Multimodal Learning for Clinical Decision Support Systems

Beyond the numbers: the reality of hospital data

In real healthcare settings, patient information is split across very different formats. Structured electronic health records (EHRs) — blood pressure, heart rate, lab results — are straightforward for models to use, but much of the clinically important context sits in unstructured, free-text notes written by clinicians.

A useful clinical decision support system (CDSS) has to read and combine both the numbers and the narrative.

Predicting ICU outcomes with belief functions

Intensive care is a high-stakes, fast-moving setting. I have also worked on intelligent network infrastructure for healthcare, including telemedicine efforts such as the Cisco-Philips eICU project.

To support decisions in this setting, my collaborators and I developed a multimodal learning framework for ICU-outcome prediction, published in the Journal of Healthcare Informatics Research. It fuses structured EHRs with free-text clinical notes, and uses belief-function theory to weigh the evidence from the notes against the structured data, producing predictions with an explicit measure of reliability.

Toward broadly capable healthcare AI

Large language models and foundation models have raised the prospect of “universal” clinical AI, but clinical use demands safety, transparency, and domain adaptation. In a survey in Information Fusion, I examine whether multimodal learning has actually delivered universal intelligence in healthcare, and map how clinical foundation models need to evolve to handle real patient care — from acute ICU monitoring to chronic-disease prediction.

Clinical decision support

By connecting structured-data methods with natural-language processing, this line of work aims to ensure no part of a patient’s record is left behind, giving the next generation of decision support systems a more complete and trustworthy view of each patient.