Crowd Science for Hybrid AI Applications

Most AI applications are hybrid, that is, employ machines to make inferences but can fall back on humans when the algorithm is not confident enough. This is true for a wide class of applications ranging from self-driving cars to decision making and process automation in enterprise AI. In this WIP paper we present our vision and progress towards an AI and crowd service that trains, assess and refines ML systems intended to be used in hybrid context. We specifically focus on crowdsourcing as a mean to assist ML algorithm development, and on the different ways in which crowd and machine can interact before, during and after the training process in a synergic way that goes well beyond the "traditional" application of crowd workers to provide data labels for ML training.