Addressing combinatorial experiments and scarcity of subjects by provably orthogonal and crossover experimental designs

Context: Experimentation in Software and Security Engineering is a common research practice, in particular with human subjects. Problem: The combinatorial nature of software configurations and the difficulty of recruiting experienced subjects or running complex and expensive experiments make the use of full factorial experiments unfeasible to obtain statistically significant results. Contribution: Provide comprehensive alternative Designs of Experiments (DoE) based on orthogonal designs or crossover designs that provably meet desired requirements such as balanced pair-wise configurations or balanced ordering of scenarios to mitigate bias or learning effects. We also discuss and formalize the statistical implications of these design choices, in particular for crossover designs. Artifact: We made available the algorithmic construction of the design for ℓ=2,3,4,5 levels for arbitrary K factors and illustrated their use with examples from security and software engineering research.