BACKGROUND: Static analysis security testing (SAST) tools may be evaluated using synthetic micro benchmarks and benchmarks based on real-world software. AIMS: The aim of this study is to address the limitations of the existing SAST tool benchmarks: lack of vulnerability realism, uncertain ground truth, and large amount of findings not related to analyzed vulnerability. METHOD: We propose Delta-Bench - a novel approach for the automatic construction of benchmarks for SAST tools based on differencing vulnerable and fixed versions in Free and Open Source (FOSS) repositories. To test our approach, we used 7 state of the art SAST tools against 70 revisions of four major versions of Apache Tomcat spanning 62 distinct Common Vulnerabilities and Exposures (CVE) fixes and vulnerable files totalling over 100K lines of code as the source of ground truth vulnerabilities. RESULTS: Our experiment allows us to draw interesting conclusions (e.g., tools perform differently due to the selected benchmark). CONCLUSIONS: Delta-Bench allows SAST tools to be automatically evaluated on the real-world historical vulnerabilities using only the findings that a tool produced for the analysed vulnerability.

Delta-Bench: Differential Benchmark for Static Analysis Security Testing Tools / Pashchenko, Ivan; Dashevskyi, Stanislav; Massacci, Fabio. - ELETTRONICO. - (2017), pp. 163-168. (Intervento presentato al convegno ESEM 2017 tenutosi a Toronto nel 9th-10th November 2017) [10.1109/ESEM.2017.24].

Delta-Bench: Differential Benchmark for Static Analysis Security Testing Tools

Pashchenko, Ivan;Stanislav Dashevskyi;Fabio Massacci
2017-01-01

Abstract

BACKGROUND: Static analysis security testing (SAST) tools may be evaluated using synthetic micro benchmarks and benchmarks based on real-world software. AIMS: The aim of this study is to address the limitations of the existing SAST tool benchmarks: lack of vulnerability realism, uncertain ground truth, and large amount of findings not related to analyzed vulnerability. METHOD: We propose Delta-Bench - a novel approach for the automatic construction of benchmarks for SAST tools based on differencing vulnerable and fixed versions in Free and Open Source (FOSS) repositories. To test our approach, we used 7 state of the art SAST tools against 70 revisions of four major versions of Apache Tomcat spanning 62 distinct Common Vulnerabilities and Exposures (CVE) fixes and vulnerable files totalling over 100K lines of code as the source of ground truth vulnerabilities. RESULTS: Our experiment allows us to draw interesting conclusions (e.g., tools perform differently due to the selected benchmark). CONCLUSIONS: Delta-Bench allows SAST tools to be automatically evaluated on the real-world historical vulnerabilities using only the findings that a tool produced for the analysed vulnerability.
2017
11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement ESEM 2017 Proceedings
Piscataway, NJ
Institute of Electrical and Electronics Engineers
978-1-5090-4039-1
Pashchenko, Ivan; Dashevskyi, Stanislav; Massacci, Fabio
Delta-Bench: Differential Benchmark for Static Analysis Security Testing Tools / Pashchenko, Ivan; Dashevskyi, Stanislav; Massacci, Fabio. - ELETTRONICO. - (2017), pp. 163-168. (Intervento presentato al convegno ESEM 2017 tenutosi a Toronto nel 9th-10th November 2017) [10.1109/ESEM.2017.24].
File in questo prodotto:
File Dimensione Formato  
ESEM-final.pdf

Solo gestori archivio

Descrizione: Main article
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 369.41 kB
Formato Adobe PDF
369.41 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/199027
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 15
  • ???jsp.display-item.citation.isi??? 14
  • OpenAlex ND
social impact