Robustness of gauge-invariant dynamics against defects in ultracold-atom gauge theories