Diffusive-to-ballistic crossover of symmetry violation in open many-body systems