A Claudian Proposal on Handling Data from S3

[Joeseph Smith]

Hey folks,

Doing some (a lot) of work on ERDDAP recently here. Trying to make our platform observation datasets generally available from NetCDF stores as RedShift needs to get phased out of our lineup and Databricks is probably (safely) too expensive for materialized view based ERDDAP datasets. Working between what I have, the ERDDAP repo (fairly recent), and Claude to figure out the best arrangement and ran into a wall. Resolving it may be beneficial to the user base, but it looks like there is some work to do to get there. I've compiled notes in an issue on the ERDDAP Repo here: https://github.com/ERDDAP/erddap/issues/592

Please have a read and let me know your thoughts. Would be happy to help work on a fix for this per the notes and other guidance I can gather here.

Thanks so much and hope all is well. Take care

-Joe

[Roy Mendelssohn - NOAA Federal]

Hi Joe:

Try using rclone, and then just the regular netcdf related data types. That is what we are using at the moment and are quite pleased with it. I wrote up something awhile back (https://groups.google.com/g/erddap/c/H-vJoGP42TI/m/RC7D84liBwAJ).

The issue on the ERDDAP™ GitHub is for Chris, I sort of remember there being some discussion about this when the present S3 version was added.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/a201af62-93d9-4c7d-9b97-1e7066519305n%40googlegroups.com.

[Joeseph Smith]

Very well, I'll have a look at RClone. Thanks!

-Joe

[Joeseph Smith]

As our ERDDAP currently runs on a Kubernetes Pod with Fargate Compute, we'll need to switch to an EC2 node group for it for Rclone to work it appears. "Fargate does not support FUSE
(`/dev/fuse`, `SYS_ADMIN`) or any CSI driver besides the EFS one"

More later

-Joe

[Roy Mendelssohn - NOAA Federal]

I would add that if you look at https://groups.google.com/g/erddap/c/4waq_lD7oWw/m/YA1SfUcREAAJ where I ask for help testing our GCP instance, that ERDDAP™ has all the data on an S3 bucket and is using rclone to “mount” the data on the Ubuntu VM. What is nice about this approach is rclone handles all the authorization issues, and has any number of options to try to optimize behavior, and is cross-platform, and is actively being maintained and developed/

Also the dataset at https://upwell.pfeg.noaa.gov/erddap/griddap/oisst_rclone.html is an example - that data is on an AWS S3 instance, the mount is on one of our servers (and our internet speed sucks - thanks NMFS!) but even then performance is pretty good and that is a large dataset.

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/b3c52775-7882-48e3-b082-670d09327226n%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

Just saw this. beyond what I know. My searches did confirm what you said about rclone mount not working on that setup. Chris also may have some ideas. As I said, my memory is there was some discussion about the new S3 support. Anyway thanks for bringing this up, yes I know you never want to learn about problems but that is how we make a more useful product.

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/3c4d4447-20c6-445a-acd1-509d9bf746c3n%40googlegroups.com.

[Joeseph Smith]

Well of course! Happy to air it!

Cheers

-Joe