Elsa Buchholz (DZA)
Hold a Master’s degree in Computer Science.
Working for German Center for Astrophysics as a Data Scientist.
Sessions
Astronomical Projects with Multiple Repositories: A Portal for Consolidated Information and Kubernetes-Based Operations
Authors:
E. Buchholz1, S. Borra1, M. Drobek1, L. Haupt1, H. Heßling1
Affiliation:
1 German Center for Astrophysics (DZA), Görlitz, Germany.
Date:
July 30, 2026
Abstract
The antennas and telescopes of the Square Kilometre Array Observatory (SKAO) will produce enormous data streams, making long-term storage of the raw telescope data impractical. Instead, projects such as ‘Transients and Pulsars with MeerKAT’ (TRAPUM) process the data streams in real time. They search for pulsars and transients, storing only data of scientific relevance. This real time processing is performed by a largely automated pipeline, though final steps still require manual inspection.
The Machine-Learning based Pipeline for Pulsar Analysis (ML-PPA) project aims to eliminate the manual steps by providing a fully automated analysis pipeline for pulsar searches, developed by an interdisciplinary team of astronomers, physicists and data scientists.
As the ML-PPA project continues to grow, the increasing involvement of researchers and the expansion of software have introduced challenges in maintaining a well-organised and scalable software structure.
Such project needs to adopt modern engineering practices, including a Continuous Integration and Continuous Deployment (CI/CD) pipeline for each repository, test coverage and automated deployment of documentation. In addition to eliminating isolated information silos within the infrastructure, a single portal was set up to centralise all information.
This presentation provides an overview of the software architecture of the ML-PPA project and explains how the portal and documentation for each of the sub-projects are automatically generated from the source code and made available online. In addition, containerisation and a Kubernetes infrastructure are used to operate the entire ML-PPA system. Both the repositories and the documentation remain in-house, which enables automated updates, thereby reducing maintenance costs in particular.
Future additions to ML-PPA will follow this workflow to ensure high-quality, well-documented code that is seamlessly deployed on a modern infrastructure. The already substantial volume of software documentation makes it difficult to extract relevant information. Therefore, LLMs (Large Language Models) are to be integrated in future to facilitate the use of ML-PPA.
Galactic Plane Imaging and Polarimetry from MeerKAT Data on High-Performance and Memory-Based Computing Resources
Authors:
L. Haupt1, A. Basu1, S. Borra1, E. Buchholz1, H. Heßling1, Y. K. Ma2
Affiliation:
1 German Center for Astrophysics (DZA), Görlitz, Germany
2 Max Planck Institute for Radio Astronomy (MPIfR), Bonn, Germany
Date:
July 28, 2026
Abstract
The MPIfR–MeerKAT Galactic Plane Survey (MMGPS) produces large multidimensional imaging datasets, including full-polarisation image cubes and Faraday cubes. Processing these datasets requires high-capacity storage together with high-performance computing (HPC) resources, as image calibration and reconstruction demand exceptionally large amounts of memory. Unfortunately, processing the complete MMGPS workflow currently takes several weeks.
At the German Center for Astrophysics (DZA), a multidisciplinary team with expertise in astrophysics, computer science, and mathematics performed a detailed analysis of the MMGPS processing pipeline with respect to memory, storage, and computational requirements. The study revealed significant I/O bottlenecks that limit the overall processing performance. Our objective is to optimise resource allocation by mapping each pipeline stage to the hardware architecture best suited for its requirements, thereby significantly improving efficiency without modifying the underlying scientific software.
Memory-Based Computing (MBC) plays a central role in this approach. Unlike conventional HPC systems, which are primarily processor-centric, MBC is designed around a large shared-memory architecture. The DZA operates an MBC prototype equipped with 48 TB of shared main memory, providing an ideal platform for memory-intensive radio astronomy applications.
By combining HPC and MBC resources, we successfully reconstructed a complete three-dimensional MeerKAT image cube. Furthermore, the large memory capacity of the MBC system enabled the generation of a Faraday cube and its direct visualisation as a movie. Overall, the optimised mapping of pipeline stages to the appropriate hardware improved processing performance by approximately one order of magnitude. In addition to these hardware-related improvements, we present first considerations for potential software-level optimisations.
Future work will focus on identifying remaining bottlenecks and evaluating parallelisation strategies. The MMGPS pipeline serves as an ideal prototype for preparing data-processing workflows for the substantially larger datasets expected from the Square Kilometre Array Observatory (SKAO).