Analytical aspects of non-differentiable neural networks