Towards universal building blocks for cloud-native digital-twins

Abstract. The exponential growth of Earth Observation (EO) data presents significant challenges for efficient data access, processing, and analysis. Current approaches often involve disparate data formats, coordinate systems, and access patterns, limiting interoperability and scalability. Firstly, the Zarr and Parquet data storage formats have seen wide adoption as a unifying cloud-native foundation for various domains in recent years, including climate, EO, bio-imaging, and genomics. Secondly, Discrete Global Grid Systems (DGGS) such as HEALPix, or ISEA-based hexagonal DGGS are being increasingly used to enable spatial data indexing beyond traditional grids, by using equal-area pixels and location- and refinement level encoding indices. Lastly, the recently published OGC API DGGS standard specifies a lightweight web service API for clients accessing data organised according to Discrete Global Grid Reference Systems (DGGRS). 
We describe a scalable, interoperable, and extensible FOSS architecture for a modern geospatial data ecosystem, based on DGGS. As an example, we introduce pydggsapi, a Python server implementing this new OGC standard to serve large geospatial datasets from cloud-native Zarr and Parquet data stores indexed by a DGGS. This novel architecture combines the DGGS data cube paradigm, standardised OGC API DGGS web service access, and cloud-optimised data formats such as Zarr and Parquet as universal building blocks for geospatial data management, enabling seamless transitions between high-performance computing environments and lightweight client applications.