Eva Optimizer: Escaping Low-Curvature Traps in Deep Learning

Adaptive optimization methods like Adam excel in handling sparse gradients and varying parameter scales, but risk instability in regions of low curvature due to excessive step sizes. We introduce Eva (Escaping Via Adaptive gradients), a family of novel optimizers comprising two variants—Eva+ and Eva-—that dynamically switch between Adam and scaled gradient steps based on local gradient and curvature properties. Our method implements component-wise detection to identify updates where Adam step magnitude exceeds the historical gradient norm. Eva+ employs gradient ascent to escape from degenerate regions, while Eva- provides conservative descent with magnitude damping. We set a threshold for detecting unstable updates, supported by theoretical analysis and empirical validation. Extensive experiments prove that both Eva variants achieve faster convergence and improved generalization compared to popular optimizers.