On the Use of Imbalanced Datasets for Learning-Based Vulnerability Detection

Static code analysis conducted by means of learning-based methods is an essential part of Security Testing. Effective learning algorithms are crucial for training reliable models that can accurately detect weaknesses and vulnerabilities. During models’ training, however, it is also of paramount importance to use adequate datasets of vulnerable and non-vulnerable source code. Most existing learning-based methods have been evaluated by applying them to public datasets of code fragments labeled as vulnerable and non-vulnerable. However, it is recognized that such datasets contain spurious entries, and are often imbalanced, i.e., contain a large portion of non-vulnerable code. While the first issue is often fixed with a pre-processing of data cleaning operations, the second one is almost ignored. This paper reports a preliminary study that investigates the effect of adopting imbalanced datasets and imbalance techniques on the performance of learning-based vulnerability detection methods. Our results show that (i) resampling approaches, in particular, a combination of over and under sampling, can generate reliable models and corroborate the results; and (ii) imbalance loss functions can improve the performance in case of very imbalanced and variegated datasets.