Working examples of Zarr datasets

[Alessandro De Donno]

Dear All,

do you have any working XML configuration of Zarr datasets?

Local datasets or remote S3 ones.

Thanks in advance,

  Alessandro

[Roy Mendelssohn - NOAA Federal]

I do not have a sample datasets.xl snippet, but note from the docs:

> As of version 2.25 ERDDAP™ can read local Zarr files using EDDTableFromNcFiles and EDDGridFromNcFiles.
>

If you have a sample Zarr dataset, just use GenerateDatasetsXml.sh with EDDGridFromNcFiles and that should get you most of the way.

A reminder that right now ERDDAP™ only works with Zarr v.2, Zarr v3 support will be added when there is a stable parr-java and of course when there is time. If anyone wants to start working on that using the preview zarr-java it would be most welcome, but for many reasons we do not want to use it until there is a stable enough version.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/f0160e25-1435-4537-b374-82f9c28783a3n%40googlegroups.com.

[Chris John - NOAA Affiliate]

There are several test datasets. I've attached a text file with those. As for the files, they are in the data.zip here. They are in the zarr folder once you've extracted the files from data.zip.

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

To view this discussion, visit https://groups.google.com/d/msgid/erddap/712B8AFA-B11D-47C3-BB61-22B6F8CB4253%40noaa.gov.

[Alessandro De Donno]

Dear Roy, dear Christopher,

thanks for your prompt replies.

I'm not able to generate a valid configuration through GenerateDatasetsXml.sh, even for the data provided by Christopher: once I select the EDDType (EDDGridFromNcFiles), the parent directory, the file name regex (the .zarr directory), and leaving the other fields empty, there are at least a couple of problems: 1) the sampleFileName is the first file it founds in the zarr directory, but it cannot be used as "sample" filename, and 2) 'ERROR in NcHelper.findMaxDVariables: None of the variables in group="" have dimensions'. Listing the dimensions in DimensionsCSV does not help.

Anyways I did a step forward, I can load the data provided by Christopher onto the ERDDAP node (v2.28.1), although they don't seem geographical data.
And, in general, it's not clear to me how to handle dimensions.

I'm trying to configure a sample Copernicus dataset downloaded as explained here: https://help.marine.copernicus.eu/en/articles/8176692-how-to-choose-between-netcdf-and-zarr-format-using-the-toolbox
Starting from the examples by Christopher, I can manually rearrange the configuration to ensure that the dataset is loaded onto ERDDAP, even if there are still some issues: longitude missing, incorrect ranges, etc.
And I don't know why I have to set "time/dim0" etc. in the source names.

I attach data and current XML config, if anyone can take a look at it and give me some tips.
And, again, if anyone has a working example, it would be appreciated! :-)

Thanks, bye,
  A.

[Roy Mendelssohn - NOAA Federal]

HI Alessandro:

Unless Copernicus has changed their access policies, in order to get files you need a login/password or a token, and ERDDAP™ doesn’t support that. The Copernicus toolbox does require such. My guess. unless I am mistaken, is this is why your access is failing.

What I would suggest to test this is use the Copernicus toolbox to download 2-3 zarr files, and then try to load those into ERDDAP™. If that fails, then e have something we can test against.

Thanks,

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/2556e07c-b0e5-44c0-b14c-be9908987510n%40googlegroups.com.
> <zarr-test.xml><global_so.zarr.zip>

[Alessandro De Donno]

Hi Roy,

yes, you need Copernicus credentials to get datasets from their datastores, but - as you suggested - I have downloaded them locally and I tried to configure them in ERDDAP as local files.

They are the data I attached in my previous message (global_so.zarr.zip), and "zarr-test.xml" is the configuration I'm working on.

Thanks anyway for your time!

Bye,

  A.

[Roy Mendelssohn - NOAA Federal]

Hi Alessandro:

Please understand we have very small staff who do a lot of other work and we try our best to support users.

Thanks for the file. I have Python 3.3.14 running python zarr version 3.1.5. which I just updated: I get:

> >>> import zarr
> >>> arr = zarr.open_array("global_so.zarr", mode="r")
> Traceback (most recent call last):
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 1282, in open_array
> return await AsyncArray.open(store_path, zarr_format=zarr_format)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/core/array.py", line 1006, in open
> metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/core/array.py", line 254, in get_array_metadata
> raise ArrayNotFoundError(msg)
> zarr.errors.ArrayNotFoundError: Neither Zarr V3 nor Zarr V2 array metadata documents were found in store LocalStore('file://global_so.zarr') at path ''.
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "<python-input-1>", line 1, in <module>
> arr = zarr.open_array("global_so.zarr", mode="r")
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/api/synchronous.py", line 1392, in open_array
> sync(
> ~~~~^
> async_api.open_array(
> ^^^^^^^^^^^^^^^^^^^^^
> ...<6 lines>...
> )
> ^
> )
> ^
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/core/sync.py", line 159, in sync
> raise return_result
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/core/sync.py", line 119, in _runner
> return await coro
> ^^^^^^^^^^
> File "/Users/rmendels/micromamba/envroot/envs/work/lib/python3.14/site-packages/zarr/api/asynchronous.py", line 1294, in open_array
> raise ArrayNotFoundError(msg) from err
> zarr.errors.ArrayNotFoundError: No array found in store file://global_so.zarr at path
>
>

Which probably explains why ERDDAP™ is failing.

HTH,

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/18ee3399-f28f-4269-99a1-01180602478dn%40googlegroups.com.

[Alessandro De Donno]

Hi Roy,

you're absolutely right and my intention was not to ask you directly for support!

My question was directed at someone who had already configured some Zarr (geo)datasets on ERDDAP and who may have experienced similar problems.

Thanks a lot for your suggestions and your time.

I will also try to read the files via Python libs.

BTW, in both files ".zgroup" and ".zmetadata", there is the zarr_format specified (2).

If I have any updates, I will share them here.

Bye,

  A.

[Roy Mendelssohn - NOAA Federal]

Hi Alessandro:

I tried to open what you sent. If you have a file that works in Python please send that. What I was looking for in the Python is the zarr version and the compressor used. ERDDAP™ uses the netcdf-java library to read zarr files. That library can only read zarr version 2 and only certain compressors. The following python program will print out that information:

import zarr
arr = zarr.open_array("path/to/data.zarr", mode="r")
print(arr.zarr_format) # Zarr version
print(arr.compressor) # Compression algorithm and params


HTH,

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/0bd775a0-eaf9-4da4-88a7-792e6849bca8n%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

I just opened the information by hand.  The .zmetadata file says they are using the Blosc compression algorithm.  That algorithm is not supported in netcdf-java.  You can use xarray for example to read in the file and then write it out using one of the compression algorithms supported by netcdf-java,   When I search for that info I get:

From the official docs, the filters/compressors that ship natively are: docs.unidata.ucar.edu+1

• Deflate (zlib) – this is the only actual compressor included out of the box.

• Shuffle – a pre-conditioning filter that reorders bytes to improve zlib compression.

• 32-bit checksum filters – CRC, Fletcher, and Adler.

• ScaleOffset / FixedScaleOffset – quantization filter (often used for packing floats into integers)

So that is why the problems in ERDDAP™.  There is a new parr-java library that is just reaching the point where it is stable enough that support both zarr v2 and  v3 as well as more compression algorithms.  Down the road we hope to support better zarr options.

HTH,

-Roy

[Alessandro De Donno]

Hi Roy,

indeed I was just looking for the supported filters here:
https://docs.unidata.ucar.edu/netcdf-java/5.9/userguide/reading_zarr.html#filters

:-)

Again, thanks for your very helpful suggestions: I'll try to change compression and let you know in the coming days.

Many many thanks,

  A.

[Roy Mendelssohn - NOAA Federal]

I haven’t tested it but you can try using toolsUI (https://downloads.unidata.ucar.edu/netcdf-java/) after you convert the zarr file to see if that can read the file, as it uses the same library as ERDDAP™

HTH,

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/bc97882b-709b-469c-9585-51f1151d4551n%40googlegroups.com.