From predictive to interactive multimodal language learning

The way humans learn the meaning of words is a fundamental question in many different disciplines and, from a computational perspective, an answer to this question could lead to important advances in artificial intelligence. While the details of the learning process are still an open question, what we do know is that humans make use of the very rich perceptual input present in the communicative setups in which learning takes place. In this work, we will present three models of human learning from naturalistic multi-modal input. We will start by introducing a model that assumes a purely predictive learner existing in a non-communicative setup and show that such a computational learner when tested displays comparable learning behaviour to human learners on a novel word learning setup. We will then relax some of the learning assumptions and present a model that, instead of exposing the computational learner to a passive environment, such as the text corpora traditionally used in semantic learning experiments, it exposes the learner to communicative episodes, simulated in our experiments by corpora capturing multi-modal interactions between children and their caregivers, allowing the learner to make use of information beyond words and passive percept during learning. Finally, we will present on-going work towards interactive learning between two agents.