xr.open_mfdataset()

[Ranjan Kumar Sahu]

Hi team,

I am trying to open a bunch of files (~1000) each 30 mb. query 1) would mfdataset of xarray support such large operation. I ask as cdo fails the merging operation due to memory issue.

query 2) if it does work. I had tried this code

file_pattern = "/path/to/folder/*.nc"
ds = xr.open_mfdataset(file_pattern, combine='nested', concat_dim='time')

which seems to throw the error:

TypeError: from_array() got an unexpected keyword argument 'inline_array'

Any help would be appreciated.

regards,

Ranjan

[Marco Miani]

Hi.

"cdo fails the merging operation due to memory issue" - did you try using parallel background process?

cdo -P 4 somecommand filein fileout

Happy coding.

--
You received this message because you are subscribed to the Google Groups "xarray" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xarray+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/5c8110cb-ea7e-4e8a-bdca-39f7abcdf9f4n%40googlegroups.com.

[Ranjan Kumar Sahu]

Thank you for your responses.

To,

Nagaraju Gaddam:

Apparently, the dask latest version wasn't there for Python 3.8 (which I had on my machine). Upgrading Python to python 3.10 and corresponding libraries worked.

To,

Marco Miani

In CDO, some of the operators are parallelized with OpenMP. Mergefile operator is not among them.

It can be used for other parallelized operations.

Thank you both.

regards,

Ranjan

You received this message because you are subscribed to a topic in the Google Groups "xarray" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xarray/NKUvdnLC9JQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xarray+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/CABjf4Fj4jqP1roWqXAi2TXrLTdfqRV-Fm6cNJ0Ss9rf9Y70YGA%40mail.gmail.com.