A new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for “non-developer” software testers as well. Recently, many vendors have launched on the market many proposals of NLP-based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP-based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach— measured in terms of test cases development and test cases evolution—with respect to more consolidated approaches, that is, programmable (or script-based) testing and capture&replay testing. The results of our study show that NLP-based test automation appears to be competitive for small- to medium-sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills.

An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay / Leotta, Maurizio; Ricca, Filippo; Marchetto, Alessandro; Olianas, Dario. - In: JOURNAL OF SOFTWARE MAINTENANCE AND EVOLUTION: RESEARCH AND PRACTICE. - ISSN 2047-7473. - 2023:(2023), pp. 1-24. [10.1002/smr.2606]

An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay

Ricca, Filippo;Marchetto, Alessandro;
2023-01-01

Abstract

A new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for “non-developer” software testers as well. Recently, many vendors have launched on the market many proposals of NLP-based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP-based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach— measured in terms of test cases development and test cases evolution—with respect to more consolidated approaches, that is, programmable (or script-based) testing and capture&replay testing. The results of our study show that NLP-based test automation appears to be competitive for small- to medium-sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills.
2023
Leotta, Maurizio; Ricca, Filippo; Marchetto, Alessandro; Olianas, Dario
An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay / Leotta, Maurizio; Ricca, Filippo; Marchetto, Alessandro; Olianas, Dario. - In: JOURNAL OF SOFTWARE MAINTENANCE AND EVOLUTION: RESEARCH AND PRACTICE. - ISSN 2047-7473. - 2023:(2023), pp. 1-24. [10.1002/smr.2606]
File in questo prodotto:
File Dimensione Formato  
SI-JSEP.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Creative commons
Dimensione 1.09 MB
Formato Adobe PDF
1.09 MB Adobe PDF Visualizza/Apri
J Software Evolu Process - 2023 - Leotta.pdf

accesso aperto

Descrizione: first online
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 2.99 MB
Formato Adobe PDF
2.99 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/388969
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? 1
social impact