Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AIbased scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.
A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications / Padovani, Gabriele; Anantharaj, Valentine; Sacco, Ludovica; Kurihana, Takuya; Bunino, Matteo; Tsolaki, Kalliopi; Girone, Maria; Antonio, Fabrizio; Sopranzetti, Carolina; Fronza, Massimiliano; Fiore, Sandro L.. - (2024), pp. 2024-2031. (Intervento presentato al convegno International Conference for High Performance Computing, Networking, Storage and Analysis (WORKS24 Workshop) tenutosi a Atlanta, Georgia nel 17-22 November 2024 (Workshop 18 November 2024)) [10.1109/SCW63240.2024.00253].
A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications
Padovani, Gabriele
;Sacco, Ludovica;Fiore, Sandro L.
2024-01-01
Abstract
Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AIbased scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.File | Dimensione | Formato | |
---|---|---|---|
A_software_ecosystem_for_multi-level_provenance_management_in_large-scale_scientific_workflows_for_AI_applications.pdf
accesso aperto
Descrizione: Accepted Manuscript
Tipologia:
Post-print referato (Refereed author’s manuscript)
Licenza:
Tutti i diritti riservati (All rights reserved)
Dimensione
500.49 kB
Formato
Adobe PDF
|
500.49 kB | Adobe PDF | Visualizza/Apri |
A_software_ecosystem_for_multi-level_provenance_management_in_large-scale_scientific_workflows_for_AI_applications.pdf
Solo gestori archivio
Tipologia:
Versione editoriale (Publisher’s layout)
Licenza:
Tutti i diritti riservati (All rights reserved)
Dimensione
649.18 kB
Formato
Adobe PDF
|
649.18 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione