THREDDS/OPeNDAP server errors

84 views

Skip to first unread message

Eric Younkin

unread,

Sep 25, 2024, 12:43:49 PMSep 25

to HYCOM.org Forum

Hello,

I wanted to report errors that I've seen accessing the ESPC datasets via OPeNDAP and Xarray:

https://tds.hycom.org/thredds/dodsC/ESPC-D-V02/u3z

>>> import xarray as xr
>>> ds = xr.open_dataset("https://tds.hycom.org/thredds/dodsC/ESPC-D-V02/u3z", decode_times=False)
oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"oc_open: server error retrieving url: code=? message="Error {
code = 500;
message = "NcSDArray java.lang.ArrayIndexOutOfBoundsException=arraycopy: last destination index 368 out of bounds for double[360] for request= NcSDArray read tau time(0,1,359) dataset= ESPC-D-V02/u3z";
}"Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/api.py", line 572, in open_dataset
backend_ds = backend.open_dataset(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 607, in open_dataset
ds = store_entrypoint.open_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/store.py", line 58, in open_dataset
ds = Dataset(vars, attrs=attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/dataset.py", line 701, in __init__
variables, coord_names, dims, indexes, _ = merge_data_and_coords(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/dataset.py", line 430, in merge_data_and_coords
return merge_core(
^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/merge.py", line 718, in merge_core
collected = collect_variables_and_indexes(aligned, indexes=indexes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/merge.py", line 358, in collect_variables_and_indexes
variable = as_variable(variable, name=name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 159, in as_variable
obj = obj.to_index_variable()
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 572, in to_index_variable
return IndexVariable(
^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 2641, in __init__
self._data = PandasIndexingAdapter(self._data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 1481, in __init__
self.array = safe_cast_to_index(array)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexes.py", line 469, in safe_cast_to_index
index = pd.Index(np.asarray(array), **kwargs)
^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 474, in __array__
return np.asarray(self.get_duck_array(), dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 554, in get_duck_array
array = self.array[self.key]
~~~~~~~~~~^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 100, in __getitem__
return indexing.explicit_indexing_adapter(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 861, in explicit_indexing_adapter
result = raw_indexing_method(raw_key.tuple)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 113, in _getitem
array = getitem(original_array, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/common.py", line 151, in robust_getitem
return array[key]
~~~~~^^^^^
File "src/netCDF4/_netCDF4.pyx", line 4940, in netCDF4._netCDF4.Variable.__getitem__
File "src/netCDF4/_netCDF4.pyx", line 5898, in netCDF4._netCDF4.Variable._get
File "src/netCDF4/_netCDF4.pyx", line 2014, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: file not found

I have not seen this issue with the v3z dataset. I've found that most of the time, this issue can be solved with a number of retries.

https://tds.hycom.org/thredds/dodsC/ESPC-D-V02/ice

>>> ds = xr.open_dataset("https://tds.hycom.org/thredds/dodsC/ESPC-D-V02/ice", decode_times=False)
Traceback (most recent call last):
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 113, in _getitem
array = getitem(original_array, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/common.py", line 151, in robust_getitem
return array[key]
~~~~~^^^^^
File "src/netCDF4/_netCDF4.pyx", line 4940, in netCDF4._netCDF4.Variable.__getitem__
File "src/netCDF4/_netCDF4.pyx", line 5896, in netCDF4._netCDF4.Variable._get
IndexError: index exceeds dimension bounds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/api.py", line 572, in open_dataset
backend_ds = backend.open_dataset(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 607, in open_dataset
ds = store_entrypoint.open_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/store.py", line 58, in open_dataset
ds = Dataset(vars, attrs=attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/dataset.py", line 701, in __init__
variables, coord_names, dims, indexes, _ = merge_data_and_coords(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/dataset.py", line 430, in merge_data_and_coords
return merge_core(
^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/merge.py", line 718, in merge_core
collected = collect_variables_and_indexes(aligned, indexes=indexes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/merge.py", line 358, in collect_variables_and_indexes
variable = as_variable(variable, name=name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 159, in as_variable
obj = obj.to_index_variable()
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 572, in to_index_variable
return IndexVariable(
^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/variable.py", line 2641, in __init__
self._data = PandasIndexingAdapter(self._data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 1481, in __init__
self.array = safe_cast_to_index(array)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexes.py", line 469, in safe_cast_to_index
index = pd.Index(np.asarray(array), **kwargs)
^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 474, in __array__
return np.asarray(self.get_duck_array(), dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 554, in get_duck_array
array = self.array[self.key]
~~~~~~~~~~^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 100, in __getitem__
return indexing.explicit_indexing_adapter(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/core/indexing.py", line 861, in explicit_indexing_adapter
result = raw_indexing_method(raw_key.tuple)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/eric.younkin/miniconda3/envs/wx-products-common/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 123, in _getitem
raise IndexError(msg)
IndexError: The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load().

This can sometimes be solved with a number of retries, but often times all of my retries fail in this way.

Thanks,
Eric

Michael McDonald

unread,

Oct 15, 2024, 6:19:11 PMOct 15

to Eric Younkin, HYCOM.org Forum

Eric,
If you are setting this fail and then working on refresh/retry then
this is likely due to one (or more) of the servers providing THREDDS
in the pool having not updated/restarted yet. So this might just be
poor timing to get the data when the servers are updating
(restarting). Next time please note the "time/date and timezone" when
these errors occur and we can see if this coincides with a dataset
update.

> --
> --
> You received this message because you are a member of HYCOM.org
> To ask a question, send an email to fo...@hycom.org
>

--
Michael McDonald
HYCOM.org Administrator

Reply all

Reply to author

Forward

0 new messages