Performance and computational complexity optimization in configurable hybrid video coding system