Empirical Methods for Evaluating Vulnerability Models

This dissertation focuses on the following research question: “how to independently and systematically validate empirical vulnerability models?”. Based on the survey of past studies about the vulnerability discovery process, the dissertation has pointed out several critical issues in the traditional methodology for evaluating the performance of vulnerability discovery models (VDMs). Such issues did impact the conclusions of several studies in the literature. To address such pitfalls, a novel empirical methodology and a data collection infrastructure are proposed to conduct experiments that evaluate the empirical performance of VDMs. The methodology consists of two quantitative analyses, namely quality and predictability analyses, which enable analysts to study the performance of VDMs, and to compare them effectively.The proposed methodology and the data collection infrastructure have been used to assess several existing VDMs on many major versions of the major browsers (i.e., Chrome, Firefox, Internet Explorer, and Safari). The extensive experimental analysis reveals an interesting finding about the VDM performance in terms of quality and predictability: the simplest linear model is the most appropriate one for predicting vulnerability discovery trend within the first twelve months since the release date of browser versions; later than that, logistic models are more appropriate. The analyzed vulnerability data exhibits the phenomenon of after-life vulnerabilities, which have been discovered for the current version, but also attributed to browser versions out of support – dead versions. These vulnerabilities, however, may not actually exist, and may have an impact on past scientific studies, or on compliance assessment. Therefore, this dissertation has proposed a method to identify code evidence for vulnerabilities. The results of the experiments show that a significant amount of vulnerabilities has been systematically over-reported for old versions of browsers. Consequently, old versions of software seem to have less vulnerabilities than reported