Zarr has emerged as a flexible data format for storing cloud-optimized self-describing n-dimensional data cubes. It’s great! There is just one problem. The vast majority of data that is being generated and distributed right now is not in Zarr stores — it’s in NetCDF, or HDF5, or COG. Those legacy data formats all support partial reads, but they aren’t optimized for cloud access (except for COG). What this means is that they require many small network requests to fetch bits of metadata that are sprinkled throughout the file and — when concatenating — across many different files. One solution is to copy data over to Zarr. But this means you double your storage cost and introduce a reprocessing step.

That’s where virtual Zarr stores come in. Virtual Zarr stores capture metadata about where particular chunks of data are stored and allows tools like xarray to access the data from the chunks directly using HTTP range requests. You never need to duplicate the data to Zarr. These virtual Zarr stores can be written to disk using kerchunk or icechunk and distributed alongside data.

So how does VirtualiZarr fit in? VirtualiZarr is the easiest way to construct virtual Zarr stores out of thousands of individual data files. It takes advantage of the powerful merging and concatenation logic that xarray already has and lets you apply those same methods to virtual Zarr stores. VirtualiZarr provides readers for many legacy data formats so you can read the metadata from the original files and produce a virtual Zarr store with the combined metadata and access paths.