Galactic Plane Imaging and Polarimetry from MeerKAT Data on High-Performance and Memory-Based Computing Resources

Authors:
L. Haupt1, A. Basu1, S. Borra1, E. Buchholz1, H. Heßling1, Y. K. Ma2

Affiliation:
1 German Center for Astrophysics (DZA), Görlitz, Germany
2 Max Planck Institute for Radio Astronomy (MPIfR), Bonn, Germany

Date:
July 28, 2026

Abstract

The MPIfR–MeerKAT Galactic Plane Survey (MMGPS) produces large multidimensional imaging datasets, including full-polarisation image cubes and Faraday cubes. Processing these datasets requires high-capacity storage together with high-performance computing (HPC) resources, as image calibration and reconstruction demand exceptionally large amounts of memory. Unfortunately, processing the complete MMGPS workflow currently takes several weeks.

At the German Center for Astrophysics (DZA), a multidisciplinary team with expertise in astrophysics, computer science, and mathematics performed a detailed analysis of the MMGPS processing pipeline with respect to memory, storage, and computational requirements. The study revealed significant I/O bottlenecks that limit the overall processing performance. Our objective is to optimise resource allocation by mapping each pipeline stage to the hardware architecture best suited for its requirements, thereby significantly improving efficiency without modifying the underlying scientific software.

Memory-Based Computing (MBC) plays a central role in this approach. Unlike conventional HPC systems, which are primarily processor-centric, MBC is designed around a large shared-memory architecture. The DZA operates an MBC prototype equipped with 48 TB of shared main memory, providing an ideal platform for memory-intensive radio astronomy applications.

By combining HPC and MBC resources, we successfully reconstructed a complete three-dimensional MeerKAT image cube. Furthermore, the large memory capacity of the MBC system enabled the generation of a Faraday cube and its direct visualisation as a movie. Overall, the optimised mapping of pipeline stages to the appropriate hardware improved processing performance by approximately one order of magnitude. In addition to these hardware-related improvements, we present first considerations for potential software-level optimisations.

Future work will focus on identifying remaining bottlenecks and evaluating parallelisation strategies. The MMGPS pipeline serves as an ideal prototype for preparing data-processing workflows for the substantially larger datasets expected from the Square Kilometre Array Observatory (SKAO).