Most astronomical machine learning workflows begin only after astronomy specific processing has already taken place: selecting archive products, interpreting FITS metadata, transforming coordinates, matching catalogues, resampling exposures, and combining observations across filters. These operations usually sit outside the model, making them difficult to optimize jointly, test within the same computation, or execute efficiently on GPUs.

We present torchfits and torchsky, a tensor native software stack that connects FITS data with differentiable models of the observed sky. torchfits provides selective access to local and remote FITS images and tables, exposing tensors and columnar batches for analysis and training. torchsky provides celestial geometry, catalogue association, mapmaking, spectral response, PSF convolution, and other observation operators on a common PyTorch runtime, with execution on CPUs and GPUs.

Together, these tools support multispectral representations of the sky as a function of position and wavelength, fitted to heterogeneous observations rather than restricted to precomputed coadds or catalogues. The same components also support conventional survey processing and large-scale cross-matching.

We will describe the software architecture, numerical validation, performance, and initial applications to multispectral mapmaking and survey-scale catalogue association. We will also show how torchsky’s geometric primitives are being reused in a new cross-matching system. More broadly, placing data access, celestial geometry, and observation operators on a common tensor runtime opens new possibilities for joint inference across surveys and wavelengths, differentiable calibration and mapmaking, and foundation models that learn from astronomical observations rather than only from preprocessed products.