Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AIbased scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.

A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications / Padovani, Gabriele; Anantharaj, Valentine; Sacco, Ludovica; Kurihana, Takuya; Bunino, Matteo; Tsolaki, Kalliopi; Girone, Maria; Antonio, Fabrizio; Sopranzetti, Carolina; Fronza, Massimiliano; Fiore, Sandro L.. - (2024), pp. 2024-2031. (Intervento presentato al convegno International Conference for High Performance Computing, Networking, Storage and Analysis (WORKS24 Workshop) tenutosi a Atlanta, Georgia nel 17-22 November 2024 (Workshop 18 November 2024)) [10.1109/SCW63240.2024.00253].

A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications

Padovani, Gabriele
;
Sacco, Ludovica;Fiore, Sandro L.
2024-01-01

Abstract

Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AIbased scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.
2024
SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
Atlanta, Georgia
Institute of Electrical and Electronics Engineers Inc.
979-8-3503-5554-3
Settore ING-INF/05 - Sistemi di Elaborazione delle Informazioni
Settore IINF-05/A - Sistemi di elaborazione delle informazioni
Padovani, Gabriele; Anantharaj, Valentine; Sacco, Ludovica; Kurihana, Takuya; Bunino, Matteo; Tsolaki, Kalliopi; Girone, Maria; Antonio, Fabrizio; Sop...espandi
A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications / Padovani, Gabriele; Anantharaj, Valentine; Sacco, Ludovica; Kurihana, Takuya; Bunino, Matteo; Tsolaki, Kalliopi; Girone, Maria; Antonio, Fabrizio; Sopranzetti, Carolina; Fronza, Massimiliano; Fiore, Sandro L.. - (2024), pp. 2024-2031. (Intervento presentato al convegno International Conference for High Performance Computing, Networking, Storage and Analysis (WORKS24 Workshop) tenutosi a Atlanta, Georgia nel 17-22 November 2024 (Workshop 18 November 2024)) [10.1109/SCW63240.2024.00253].
File in questo prodotto:
File Dimensione Formato  
A_software_ecosystem_for_multi-level_provenance_management_in_large-scale_scientific_workflows_for_AI_applications.pdf

accesso aperto

Descrizione: Accepted Manuscript
Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 500.49 kB
Formato Adobe PDF
500.49 kB Adobe PDF Visualizza/Apri
A_software_ecosystem_for_multi-level_provenance_management_in_large-scale_scientific_workflows_for_AI_applications.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 649.18 kB
Formato Adobe PDF
649.18 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/443550
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact