On the Difficulty of Hiding Keys in Neural Networks