2026-11-03 –, Banquet Hall
The next generation of radio astronomy arrays are challenging existing data analysis paradigms, as they have an order of magnitude more antennas and larger bandwidth.
For these instruments, data storage will be the major cost driver and will constrain the processing.
The traditional imaging methods for deep (multi-epoch) observations require more storage than is available for ASKAP and than will be available for SKA.
On the other hand reducing the data volume by producing an image cube from every observing epoch bakes in systematic errors and (if the errors are non-Gaussian) imposes a sensitivity limit that can be significantly higher than the science requirement.
For this reason the radio astronomers are demanding that they retain access to the visibilities for processing the data. The only way that this can be affordable is if the data volumes for visibilities are reduced to a level that is manageable.
We have been testing two methods of data compression, which will dramatically reduce the storage requirements: grid-stacking, a two-stage lossless compression solution; and lossy compression of the raw visibilities, before traditional processing. Both are providing excellent results.
The lossless grid stacking data product, after compression, are twenty times smaller than the individual measurement sets.
The introduced losses in the recovered HI spectra are about 1\%. We are using this for DINGO.
The lossy compression can provide measurement-sets that are ten times smaller than the individual measurement sets and the losses introduced are much less than 0.1\%.
We are using this on MeerKAT.
I am the Survey Science Project lead, connecting the compute expertise in the Data-Intensive Astronomy group to the needs of the astronomers.