Re: DMR++

[James Gallagher]

Well, yes. That is, you need to use a docker container to build these. It uses a combination of things, python being part of the mix.

But the instructions are pretty good.

What is it you want to do?

James

Sent from my iPhone

> On May 23, 2025, at 12:42, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:
>
> Hi James:
>
> Just as a follow-up am I correct that we need either DMR++ or Hyrax to create the DMR++ files, or is there some way to do this in Python say.
>
> Thanks,
>
> -Roy
>
>
>>> On May 19, 2025, at 1:57 PM, James Gallagher <jgall...@opendap.org> wrote:
>>>
>>>
>>>
>>>> On May 19, 2025, at 14:35, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:
>>>
>>> Thanks James. I am curious as to how this compares to kerchunk or virtualizarr, which I think are doing similar things.
>>
>> VirtualiZarr can read DMR++ files. That’s the DMR++ reading in Python I was talking about.
>>
>> Kerchunk is more limited than DMR++. DMR++ can be used with HDF4 and essentially all of HDF5 that NASA has ever used. Kerchunk can work with NetCDF3 while DMR++ cannot.
>>
>> James
>>
>>>
>>> -Roy
>>>
>>>
>>>
>>>> On May 19, 2025, at 1:30 PM, James Gallagher <jgall...@opendap.org> wrote:
>>>>
>>>> Roy,
>>>>
>>>> I hope all is well with you.
>>>>
>>>> Combing back through my email after a proposal and a short vacation.
>>>>
>>>> Thanks for listening, and yes, other languages and more widespread support are both goals. I think Java is possible - there is already a Python reader for DMR++, although it’s not 100 current the latest DMR++ features.
>>>>
>>>> We’re proposing an ESIP session where we will try to get more people interested in this idea and working on the first version of a common design. That will be in July.
>>>>
>>>> Best,
>>>> James
>>>>
>>>>> On Apr 23, 2025, at 12:28, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:
>>>>>
>>>>> Listening to the seminar. Would be great is if the ability to create and especially read files using DMR++ can be done outside of Hyrax, in particular using other languages than DMR++, especially for the read (clearly I have a bias that this can be done in Java),
>>>>>
>>>>> -Roy
>>>>
>>>>
>>>> --
>>>> James Gallagher
>>>> jgall...@opendap.org
>>>>
>>>>
>>>>
>>>
>>
>> --
>> James Gallagher
>> jgall...@opendap.org
>
>

[Roy Mendelssohn - NOAA Federal]

Hi James:

Sorry to be so long in replying. We likely will have to move our ERDDAP services to the cloud, and are looking at ways to speed up netcdf access without having to actually convert the files. So we would want to create the files that DMR++ does outside of Hyrax, and then if possible have a java library that reads those for ERDDAP. The kerchunk/virtualizaar approach seems to be pretty tied to python.

-Roy

[James Gallagher]

On Jun 6, 2025, at 17:08, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:

Hi James:

Sorry to be so long in replying.  We likely will have to move our ERDDAP services to the cloud,  and are looking at ways to speed up netcdf access without having to actually convert the files.  So we would want to create the files that DMR++ does outside of Hyrax,  and then if possible have a java library that reads those for ERDDAP.    The kerchunk/virtualizaar approach seems to be pretty tied to python.

Yes, it is Python.

The DMR++ files are not made by Hyrax. We reused a bunch of the code, but that was just to cut down on the time to develop the command line tools. To make DMR++ files, you build the DMR++ documents and then store them in the cloud (aka, S3, Google Cloud, …) and point Hyrax at them. The code to read data using these is C++ for Hyrax and Python for VirtualiZarr. But ERDDAP can read from Hyrax, yes? Also, the possibility exists to code a DMR++ reader in Java. Not for us, but it’s doable.

We are running a session at ESIP (Seattle, July) about developing some level of community standards around this technique of working with data in cloud data stores. Maybe that would be a good place to talk about it?

James

--

James Gallagher

jgall...@opendap.org

[Roy Mendelssohn - NOAA Federal]

Hi James:

These days travel is almost impossible unless it is “mission critical” which I doubt this would be considered that by my agency, but even if they did deem it as “mission critical” for some personal reasons I essentially don’t travel anymore. I would say it sounds like a good session, and even more we need start defining solutions that are language agnostic as much as possible.

-Roy

[James Gallagher]

Right, travel, always a hassle, has been made insurmountable.

However, the meeting is hybrid, and while it’s hard to focus, ESIP has done a good job with the hybrid format in the past.

But I get your point..

James

--

James Gallagher

jgall...@opendap.org