AWS hosted Hyrax with DMR++

[Myles McManus - NOAA Affiliate]

Hello OpenDAP!

I have been tasked with getting some data into DMR++ and to use VirtualliZarr to host some type of Zarr endpoint API. I believe Hyrax is the right tool for this, but I think I'm bumping up against the newness of these capabilities as far as the documentation is concerned.

I am currently running the Hyrax container on an AWS EC2 instance, and have netCDF data in S3. I used the ingest_s3bucket tool to create dmrpp files on the EC2 filesystem that is then a mounted volume within the container using the docker run parameter: --volume ~/tmp/data:/usr/share/hyrax

NOAA NCEI archives a large amount of data, and the datasets I need to host with Hyrax number in the hundreds of millions of netCDF files. After a thorough review of the DMR++ documentation (https://opendap.github.io/DMRpp-wiki/ & https://github.com/OPENDAP/bes/blob/master/modules/dmrpp_module/data/README.md) I have a question:

Is there anyway to set the dmrpp files to exist only on S3? 

Although orders of magnitude smaller than the netCDF data, I'm seeing file sizes ~60kb per dmrpp which adds up to about 20TB. That is simply too much data for the EC2's local storage, and really expensive for an EFS (Elastic File System). Also if we try to use s3fs to mount the s3 to the container, we would see a performance impact from traversing such large buckets during catalog building/loading. So this becomes a real architectural problem, and I hope there is a work-around for allowing object-storage-hosted DMR++ files to be used by the Hyrax BES.

Does Hyrax have a Zarr API endpoint?

Or perhaps one is in development? Given that the VirtualiZarr library can parse DMR++ files (Source:https://ntrs.nasa.gov/api/citations/20240007273/downloads/VirtualiZarr_JG_R1.pdf), it would seem like a great additional output option.

Thanks,

Myles McManus, P.E.

Data Scientist

Contractor - Team Alpha Omega for NCEI
NOAA's National Centers for Environmental Information (NCEI)

NCEI Data Stewardship Division

151 Patton Avenue, Asheville, NC 28801-5001 (E/NE5)
Email:  myles,b,mcm...@noaa.gov | Voice/Text:  (828) 419-1569

[Miguel Jimenez]

Hi Myles McManus, 

Thanks for reaching out! It sounds pretty fascinating what you are doing and we would love to help. I can help quickly answer your second question:

Does Hyrax have a Zarr API endpoint?

The short answer is no. But we think with the current state of client API, it is all about Chunk Manifests, and with that DMR++ is a new way to become interoperable with Zarr. You seem very familiar with Virtualizarr which is the approach we would recommend.

Is there anyway to set the dmrpp files to exist only on S3? 

Perhaps a better way to approach this would be to set up a meeting. WE certainly would love to learn more about the approach you are following.

Are there a times next week that work best for you?

Miguel Jimenez-Urias, PhD
Computational Oceanographer | Sc. Community Director
OPeNDAP, Inc

https://www.linkedin.com/in/MiguelJimenezUrias/

[H. Joe Lee]

Hi, Myles!

Thank you for raising a very interesting question and sharing a
practical use case!

> Is there anyway to set the dmrpp files to exist only on S3?

Yes, I was able to patch Hyrax and make the DMR++ exist only on S3.
Here's patched demo Hyrax server screenshot:

https://github.com/hyoklee/hrx/wiki/S3

Please note that it uses '/opendap/s3/' for S3
instead of typical '/opendap/data/' for the local file system.
Therefore, it works for both S3 and local DMR++.

My next question or goal is
how to optimize the catalog for millions of files
using Parquet and serve it via Hyrax Parquet handler.

Regards,

---
hyo...@hdfgroup.org
http://hdfeos.org

> To unsubscribe from this group and stop receiving emails from it, send an email to support+u...@opendap.org.