On the automorphism groups of certain $p$-groups of class $4$