AI systems can be used to automatically process medical data and directly gather insights based on this data, or they can be constructed as interactive models, where a medical doctor stands in direct collaboration with the system. Such an interactive design can accelerate how fast medical knowledge can be gathered from an expert to train better medical image analysis models. 

In our work on interactive models, we analyzed the literature in a systematic review, discovering a taxonomy for deep medical interactive segmentation models (TPAMI). Further, we explored how to best integrate cues given by medical doctors into interactive deep learning models (MICCAI) and explored techniques to make interactive models faster (ISBI). We regularly participate (NATURE machine intelligence) and organize medical interactive segmentation challenges.

A central problem in artificial intelligence systems is the requirement to train them on large datasets with expensive annotations. Furthermore, in the medical domain, doctors are needed to create these annotations, as they have the expertise on how to interpret, e.g., medical images. To this end, solutions are needed to make it possible to train models with only very few annotations and to make the process of training these systems as flexible as possible to best accommodate the expert’s time. 

We propose training strategies for data efficient training (CVPR, AAAI), where with only a handful of annotations, we are able to train models for semantic segmentation with only minor performance loss as compared to models trained with hundreds of annotations. Further, we research training techniques that add flexibility to the annotation process by accepting highly heterogeneous training signals that can be used to train medical segmentation models (CVPR, ECCV). We also investigate the adaptation of pre-trained models towards new data distributions without the need to collect expensive pixel-wise annotations (ISBI).

The medical domain is comprised of highly multi-modal data. A wide range of different imaging modalities are used to gather insights into a patient’s health, some of which are optical coherence tomography, computed tomography, magnetic resonance imaging or X-ray scans. On top, in medical day-to-day routines textual data in the form of reports accumulates. 

In our research, we bring together different imaging modalities to benefit from the complimentary information they offer and thereby train better deep learning models (ICCV-W). We also showed that medical images and radiological reports can be utilized to train classification models without having to provide additional explicit labels while still enabling open-set recognition (MICCAI).