The Square Kilometre Array Observatory (SKAO) is constructing the world's largest and most sensitive radio observatory, with SKA-Low in Australia and SKA-Mid in South Africa. The SKA telescopes will produce data volumes that require reliable processing pipelines capable of transforming raw visibility data into science-ready products such as continuum and spectral line image cubes and mechanisms to make the large data accessible to users across the world. While real-time processing supports telescope operations during observations and formation of coherent tied array beams, the computationally intensive calibration and imaging workflows are performed as “batch pipelines” after data acquisition. These workflows comprise multiple stages—including pre-processing, self-calibration, continuum imaging and spectral imaging—that must be orchestrated across heterogeneous compute and storage resources while maintaining reproducibility, robustness and operational efficiency.
This paper focuses on the design challenges and emerging architecture of the SKAO Science Data Processor (SDP) batch pipeline framework, developed to support reliable, observatory-scale scientific workflows in the petabyte era.. The framework is intended to support a diverse range of observing modes through modular, configurable workflows that separate scientific processing from execution infrastructure. The SKAO is constructing the Low and Mid telescopes in stages called Array Assemblies (AA). The major array assembly stages are labelled as AA2, AA and AA4. A key design objective is to provide an architecture that scales across successive processing capabilities as the observatory grows from AA2 through AA to the full AA4. As computational and domain complexity increase across these capability levels, efficient management of visibility data becomes as important as the execution of the scientific algorithms themselves.
These requirements have driven several key architectural decisions. An example is the adoption of the Measurement Set version 4 (MSv4) data model, developed in collaboration with the National Radio Astronomy Observatory (NRAO), that provides a data representation designed for parallel access to large visibility datasets. Together with workflow decomposition, configuration-driven execution, and well-defined interfaces between pipeline stages, these design choices establish a robust and extensible foundation for reliable batch processing while allowing calibration and imaging algorithms to evolve independently of the execution framework. Beyond addressing the immediate requirements of the SKAO, the proposed architecture establishes a foundation for reliable, modular and scalable batch processing that can evolve with the needs of future data-intensive astronomical observatories.