The Keck Observatory Archive Hurtles Towards the 2030s
2026-11-04 –, Conference Foyer 2

KOA is undergoing an ambitious architectural transformation scheduled for completion in Summer 2027, intended to support accelerating data volumes, efficient and automated operations, and new user services to meet the needs of modern analysis. Some components have been discussed at earlier ADASS meetings: this presentation will give an holistic view of the architecture, and the benefits of new technologies in developing it.
Automated real-time ingestion of newly acquired data is being extended to include a dedicated data transfer node to support ingestion of large data volumes (>100 GB nightly) created by new instruments (LIGER, SCALES, and HISPEC) under development. They that will propel archive holdings from 200 TB to 1 PB, held on-premises at NExScI . At the same time, management of metadata, creation of database schema - including TAP compliant schema - and creation of the database tables themselves have been fully automated, with AI support. The creation and management of user accounts for Keck Investigators have also been fully automated. A new interactive dashboard-based metrics system will report archive content and usage.
The archive services are under active migration to a query infrastructure based on the Open Source "low-code" Plotly-Dash framework widely used in business, augmented with R-tree based indexing and interfacing with a TAP-server. Altogether, this new query infrastructure is x20 faster than is the case now, and is much more compact and sustainable. It will sit beneath a new landing page. KOA is participating in an institutional effort to investigate the value of use Agentic AI in new, natural language end-user services. Thus far, KOA has investigated the use of harness engineering in building AI applications and in synthesizing documentation for users: the harness includes doom loop detection, a deterministic rule engine and a self-improving verifier training corpus. The harness has proved to take up 80% of the effort in delivering a prototype LLM interface.
Underpinning the effort just described are migrations to Red Hat Enterprise and PostgreSQL, support for archive security and access controls, and installations of new hardware. These efforts are coordinated at an institutional level. Given that KOA is operated by a small team, we will summarize the management approach and lessons learned in managing this ambitious enterprise.

See also: The Keck Observatory Archive (KOA) Hurtles Into the 2030s (895.7 KB)

Bruce Berriman is a Senior Scientist at Caltech/IPAC-NExScI. He is the Lead for KOA, and the NASA representative on the IVOA Executive Committee. He also builds mosaics of the sky with the Montage image mosaic engine.