Detecting Errors in Convolutional Neural Networks Using Inter Frame Spatio-Temporal Correlation

Object detection, a critical feature for autonomous vehicles, is performed today using Convolutional Neural Networks (CNNs). Errors in a CNN execution can modify the way the vehicle sense the surrounding environment, potentially causing accidents or unexpected behaviors. The high computational requirements of CNNs combined with the need to perform detection in real-time allow little margin for implementing error detection. In this paper, we present an extremely efficient error detection solution for CNN based on the observation that, in the absence of errors, the differences between the input frames and the detection provided by the CNN should be strictly correlated. In other words, if the image between two subsequent frames does not change significantly, the detection should also be very similar. Similarly, if the detection varies considerably from a frame to the next, then the input image should also have been different. Whenever input images and output detection don't correlate we can detect a error. After formalizing and evaluating the inter-frame and output correlation thresholds, we implement and validate the detection strategy, utilizing data from previous radiation experiments. Exploiting the intrinsic efficiency in processing images of devices used to execute CNNs, we can detect up to 80% of errors while adding low overhead.