Stochastic Local Search for direct training of threshold networks

This paper investigates Stochastic Local Search (SLS) algorithms for training neural networks with threshold activation functions. and proposes a novel technique, called Binary Learning Machine (BLM). BLM acts by changing individual bits in the binary representation of each weight and picking improving moves. While brute-force implementations of SLS lead to enormous CPU times, due to the limited extent of each move, the use of incremental neighborhood evaluation, first-improving strategies, and Gray encoding accelerates SLS by a huge factor, opening the way to affordable CPU times for a wide range of problems. Comparisons with alternative methods demonstrate the effectiveness of the approach. In details, BLM outperforms state-of-the-art techniques either by achieving better generalization properties, or by allowing for more compact networks, suitable for the type of applications for which threshold neural networks were originally introduced.