Human-to-AI interrater agreement for lung ultrasound scoring in lung disease of prematurity

Background: Lung ultrasound has gained importance in managing prematurity-related lung disease. Lung ultrasound scores (LUS) are based on the analysis of imaging patterns (e.g., horizontal and vertical artefacts and consolidation areas). These scoring systems are operator-dependent, and interrater reliability (IRR) has not been thoroughly studied in the preterm population. Artificial intelligence (AI) has demonstrated great potential in medical imaging, including in reducing inter-operator variability in scoring LUS data from COVID patients. Aims: The present study aims to 1) assess the IRR of LUS in preterm infants; 2) develop AI approaches for providing LUS scores specific for preterm infants; 3) assess the agreement between human operators (HO) and the developed AI algorithms. Methods: We analyzed 360 videos (188 frames per video) from 60 infants with a median (Q1, Q3) gestational age of 26.43 (24.84, 29.79) weeks and a postnatal age of 25 (11, 42) days. Videos were scored by 3 HOs and analyzed using two newly developed AI approaches: 1) fine-tuning a “frame-to-video-level” strategy developed for adult COVID patients, 2) developing a novel Long Short-Term Memory-based (LSTM-based) direct video classification algorithm. Results: The Fleiss' kappa among the HOs was 57%. The Fleiss' kappa between HOs and the “frame-to-video-level” approach was 53%. For the “LSTM-based” approach, the Fleiss' kappa with HOs was 47%. Conclusion: We present the first AI algorithms for automatic LUS scoring in preterm infants. The proposed approaches led to a Fleiss' kappa approaching the one observed among HOs with a maximum difference of 10%. This research provides methods that can be useful for future research.