Agreement between expert sonographers and artificial intelligence in the evaluation of lung ultrasound data acquired from COVID-19 patients

During the COVID-19 pandemic, lung ultrasound (LUS) has been adopted by clinicians worldwide to study the effect of the virus on the lungs. However, LUS suffers from the qualitative and subjective nature of the adopted methodologies. To this end, we were the first to design, implement and validate a standardized imaging protocol (based on 14 acquisition points) and scoring system (on 4 levels). Moreover, we trained the first artificial intelligence (AI) models capable of assessing LUS videos providing, for each video-frame, the score as well as visual feedback in the form of semantic segmentation. The latter serves as explanation, is crucial to facilitate the clinician engagement in the evaluation, and guarantees a correct integration of AI into clinical practice. In this multicentre study, we report on the level of agreement between AI and LUS experts. 1,547 LUS videos from 82 patients (43 male and 39 female, mean age of 61, 23-95), corresponding to 325,568 frames, were analysed. All patients were COVID-19 positive as confirmed by RT-PCR test. Results show a perfect agreement between LUS experts and AI for 51.7% of the videos. This percentage increases to 87.5% when allowing a disagreement up to 1 point (over a score range from 0 to 3). Figure 1 shows the agreement at video-level (top) and at the exam level (bottom). The latter score is obtained summing the scores over the 14 acquisition points. These promising results illustrate the potentials of AI in supporting clinicians in the evaluation of LUS data.