Semantic-based autonomous context tracking in cognitive robotic surgery

Autonomous robotic surgery can benefit from advances in artificial intelligence to improve the outcome of surgical procedures, enhance situation awareness, and optimize the user experience of surgeons. In this paper, we focus on the important problem of autonomous context tracking in robotic surgery, aiming at tracking the relevant items in the surgical scene with the endoscopic camera arm (ECM) of the da Vinci Research Kit (dVRK) robot. We propose SemTrack, a novel method to track both instruments and anatomical parts of interest, and overcome the lack of interpretability and trustworthiness of recent deep learning solutions. We leverage natural language processing to extract a symbolic task formalization from surgical notes and texts. We then use this interpretable formalization to track the flow of the phases in the operation, including both inter-phase transitions and intra-phase target anatomies and instruments. In the context of the tumor removal of phantom-based lateral partial nephrectomy, we validate the feasibility and accuracy of our methodology at tracking relevant scene items for enhanced situation awareness. Moreover, our approach has better performance also in terms of task duration, even with moving targets, than static and teleoperated ECM.