Software Vulnerabilities in FOSS: From Assessment to Forecasting and Effective Usability

Free Open Source Software libraries are ubiquitous and their security is crucial. Many assessment methodologies and tools have been proposed to evaluate the security of these libraries. However, many of these methodologies are hard to be applied in practice as they lack context information. This dissertation aims to develop methodologies that take contexts into account to increase their applicability and allow a better understanding of the assessment results in the field. The first part of this dissertation addresses the problem of providing a security indicator. It extends the idea of technical leverage to fully include transitive dependencies and reports an extensive validation on the top 21,205 Python library versions. It shows that the chance of picking a safe version in Python is higher than the traditionally reported ratio of safe package versions. The second part addresses the problem of forecasting future vulnerability. It proposes a methodology for forecasting the probability of a library being vulnerable, leveraging the evolution of the library's source codes and dependencies. This methodology is supported by another methodology that categorizes libraries across ecosystems. It was validated on 768 Java libraries with 1255 CVEs to demonstrate its effectiveness. The third part addresses the problem of evaluating vulnerability detection tools' effectiveness with available data in practice. It proposes a methodology to extract from a dataset the evolving knowledge a model will encounter in time. It reports a validation with 4 datasets and 5 ML-based tools. It shows that when trained on available data only, ML-based vulnerability detection tools performed worse than when trained with all retrospective data. Finally, the last part addresses the problem of evaluating the usability of vulnerability detection tools. It proposes an experiment design and workflow to allow human reviewers to evaluate the usability of ML-based vulnerability detection tools. The experiment was conducted with a sample of computer science master’s students and shows that the usability of the tools still needs to be improved.