We archive massive datasets to map the stars in ESA’s Gaia mission, unlock the secrets of dark matter and dark energy with Euclid, and discover new worlds with PLATO. Over the past few years, these workloads ran on Greenplum, but when Greenplum stopped being open source, we had to evaluate all available options—open source or not—exploring new directions.
In this talk, we’ll share how we’re rethinking architecture, tools, and operations to handle petabyte-scale astronomy workloads. We’ll cover the practical pros and cons of the candidate solutions we evaluated, the administrative shifts required, and the hurdles we face moving such critical operations between engines.
We have selected a solution that offers both Open Source and Enterprise support options. We'll discuss how this option emerged as the best fit for our petabyte-scale workloads, including its ability to handle complex cross-match queries essential for astronomical research.
Attendees will leave with real-world lessons from our migration journey, including performance benchmarks from Gaia catalog queries, operational considerations for distributed PostgreSQL at scale, and key questions to ask before starting a migration of their own.