As implementing complex AI models with massive datasets becomes increasingly accessible, at what point do we reach the limits of explainability? "Simple" problems are easily tackled by AI when input parameters are well-defined and the underlying processes are understood. However, as datasets grow and the underlying processes become exponentially ill-defined, we must question whether we are too quick to abandon explainability in favor of rapid results. While societal pressures often encourage this trade-off, the scientific community must resist it. Furthermore, our human drive to seek out rules that lead to definitive conclusions can be misleading. Studies have shown that rats can easily outperform humans at certain pattern recognition tasks, illustrating how our own cognitive biases can interfere with objective analysis.

This presentation will explore these challenges using examples of analysing over 100 million astronomical sources via multi-modal AI models, demonstrating how an abundance of information can sometimes yield increasingly misleading results. I will also discuss how we utilise AI for our SRCnet software and data management stack at SKAO, specifically examining the consequences of agentic coding and how solving one problem inevitably introduces unintentional side effects. Ultimately, from analysing data to developing software, mitigating these unintentional consequences and inherent biases has become the primary focus and responsibility of human problem-solving in the era of AI.