verbenajl
Instrument scientist for the GREAT instrument on board SOFIA, currently lead of the GREAT-support project, focused on GREAT archival data.
Sessions
Background
Astronomical research increasingly requires access to datasets from multiple observatories and substantial computational resources. Among the challenges we face, two related problems particularly limit efficient use of available infrastructure. First, data often remains scattered across individual archives with limited cross-center access. While IVOA standards and FAIR principles provide a great foundation, researchers still spend significant effort downloading, organizing, and combining datasets manually.
Modern science platforms like that of Rubin Observatory show promising approaches. Initiatives like PUNCH and NFDI in Germany are developing access layers and federation approaches. New astronomical data centers are being established, such as the one for the CCAT observatory in Cologne or at the German Center for Astrophysics (DZA). However, we would benefit from community agreement and blueprints on practical implementation at the data center level. For example, building CCAT's data center in Cologne, we find ourselves collecting lessons learned and advice from many people rather
than following established blueprints.
Second, HPC resources are fragmented across institutions, often with bureaucratic access barriers. Taking our experience of developing the CCAT Data Center in Cologne as an example again: we face a change in policies of the funding agencies, that now require community use for large computing infrastructure, impeding us from requesting funding for local HPC resources for our data center. This shows that our specific use case is not well represented in these policies, not least because using external HPC resources involves complex user access procedures, software deployment difficulties, and priority conflicts that make seamless resource sharing impractical or at least very burdensome.
Objectives
This BOF will examine these inter-operability challenges from both perspectives and identify practical next steps. We distinguish this from IVOA efforts by focusing on operational implementation and resource sharing policies rather than technical protocols. The goal is to understand what must be implemented at the core level when designing data centers and science platforms to enable easier, less bureaucratic cooperation between centers and institutions.
Format
We propose splitting into two working groups - one examining data inter-operability between archives and platforms, the other exploring compute resource sharing across HPC centers. Each group will identify existing solutions, ongoing initiatives, and priority areas for development. We can also explore differences in approaches between countries and communities. Groups will reconvene to share findings and develop a joint summary document outlining concrete recommendations for the community.
Expected Outcomes
- Overview of existing solutions and ongoing initiatives for data and compute inter-operability
- Differences in approaches between countries and communities
- Actions needed to improve inter-operability
- Successful models and technical requirements
- Policy considerations for enabling better resource sharing across astronomical centers
We present grex, a Python pipeline for the reduction
and analysis of spectral line data obtained with the GREAT
instrument aboard the Stratospheric Observatory for Infrared Astronomy (SOFIA).
The pipeline operates directly on FITS files
produced by the GREAT data calibration program (kalibrate),
and is designed to handle the full complexity of large
heterodyne datasets. It covers all stages from
initial data filtering and quality control, to baseline subtraction,
spatial gridding, and spectral cube construction. A central feature is a
Principal Component Analysis (PCA) based baseline correction,
which effectively removes complex spectral baselines even in the
presence of telluric contamination, using a per-mission decomposition
to account for the varying atmospheric conditions across observing flights.
This is particularly important for datasets where a telluric feature
overlaps in velocity with the target line. The pipeline was developed for
and validated on the M51 [CII] SOFIA/GREAT dataset, a large, multi-flight
survey comprising over one million spectra obtained across multiple
observing campaigns, which represents one of the most challenging
reduction problems for the GREAT instrument. The pipeline enables
the reduction of the underexploited [OI] 63 \textmu m dataset and it is built
as a general-purpose tool applicable to any SOFIA/GREAT
data available in the archive.