Neuro-Symbolic Constraint Programming for Structured Prediction

We propose N e s t e r, a method for injecting neural networks into constrained structured predictors. N e s t e r first uses a neural network to compute an initial prediction that may or may not satisfy the constraints, and then applies a constraint-based structured predictor to refine the raw predictions according to hard and soft constraints. N e s t e r combines the advantages of its two components: the network can learn complex representations from low-level data while the constraint program on top reasons about the high-level properties and requirements of the prediction task. An empirical evaluation on handwritten equation recognition shows that N e s t e r achieves better performance than both the either component in isolation, especially when training examples are scarce, while scaling to more complex problems than other neuro-programming approaches. N e s t e r proves especially useful to reduce errors at the semantic level of the problem, which is particularly challenging for neural network architectures.