A Dual Validation Framework for Curating Machine Learning-Ready Satellite Datasets: A Scalable Pipeline and Stratified Analysis

Earth observation foundation models (FMs) require vast amounts of high-quality training data, and the remote sensing community has responded by developing numerous benchmark datasets. However, these benchmarks are predominantly static and often lack rigorous quality assessments and difficulty measures. The convergence of petabyte-scale satellite archives and large FMs therefore necessitates rigorous validation frameworks for dataset quality and complexity. To address these limitations, we introduce a dual validation framework supported by a scalable, on-demand dataset curation pipeline. The framework integrates two complementary components. First, intrinsic validation characterizes dataset quality through radiometric fidelity checks and a composite dataset difficulty index (DI). This index synthesizes spatial heterogeneity, phenological variability, and data scarcity into a single normalized metric. Second, extrinsic validation utilizes the DI to enable stratified model performance analysis across distinct difficulty levels of the dataset, providing deeper insights than conventional aggregate evaluation. To operationalize this framework, the pipeline employs a cloud-native Zarr architecture with distributed computing to enable efficient parallel data access and automated quality supervision. We validate this framework on the cloud-gap imputation task, demonstrating that stratified analysis reveals severe performance degradation patterns completely obscured by aggregate metrics. Specifically, models evaluated on high-difficulty scenes exhibited over a 50% increase in error (RMSE) and a notable degradation in structural similarity, with SSIM falling from 0.83 to 0.77. This study provides a comprehensive foundation for transforming standard analysis-ready satellite data into rigorously validated, machine learning-ready benchmarks, advancing data-centric artificial intelligence in remote sensing through the quantitative assessment of dataset characteristics and model robustness.