Interpolated Coordinates

[Alister Trabattoni]

Dear xarray community,

I would like to implement a feature that is part of the NetCDF CF convention. It consists in storing coordinates as the interpolation of some tie points (interpolated coordinates) instead of storing those points by points as a list of values. I’m actually facing a case where I cannot store the whole coordinate vector in memory and this would solve my issue.

To make it simple let focus on the simplest case: I would like to be able to pass for example a range(0, 1000, 1) like object as a coordinate instead of np.array([0, 1, …, 999]). I read a lot of things about the explicit indexing thing that is being worked on but I’m not sure this will solve my use case (or will it?). It feels that what I need are more like "custom coordinates" as for now coordinates are still stored as xarray variables that must have the same shape as the data (if I'm not wrong). If using explicit indexing is the way to go, can I start working on a solution today or is it too early to do so.

Thanks

Alister Trabattoni

[Chris Barker]

I think this is a pretty cool idea.

I think what you are going for is a lazy-evaluated array (as you say, like a range object).

I would encourage you to think of it in this more generic way -- a lazy evaluated array could be useful for all sorts of things.

However, I'm not that familiar with xarray internals, but this would be very tough to do with numpy -- there are a LOT of assumptions with numpy arrays about the internal data storage.

Dask might help here, though -- indeed it might already have such a thing.

-CHB

--
You received this message because you are subscribed to the Google Groups "xarray" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xarray+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/8f56bff2-472e-451e-a690-5041e0f8ea5an%40googlegroups.com.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Benoît Bovy]

Hi Alister,

The use case that you describe is a good one for explicit (flexible)
indexes I think. At least if you want to use the coordinate for
label-based selection, alignment, etc. Otherwise, Chris' suggestion of
using lazy arrays may be enough.

Out-of-core indexes are discussed here:
https://github.com/pydata/xarray/issues/1650.

Somewhat related (functional coordinates):
https://github.com/pydata/xarray/issues/3620.

In theory a coordinate variable could wrap any "duck" array (i.e.,
exposing some common API with .shape and .dtype properties, etc.). So it
should be possible to create a wrapper around the range object (or a
more advanced object representing interpolation tie points) so that it
could be reused as Xarray coordinate data. You might also want to create
an Xarray custom index in which you implement all the data selection and
alignment logic based on the range / tie points.

This is similar to what Xarray does internally with pandas indexes
("PandasIndex" xarray index wrapper + "PandasIndexingAdapter" coordinate
data wrapper). This is not yet ready for custom, non-pandas objects,
though. There are still some parts of Xarray internals where pandas
coordinate data wrappers are hard-coded (e.g., for dimension
coordinates) and that would require further refactoring. But the goal is
to make it ready at some point.

Benoît

[Alister Trabattoni]

Dear Benoit,

 

I tried to implement a dummy duck array class but unfortunately it get converted into a Pandas object at initialization. This happens in the IndexVariable class which is used to store coordinate data.

 

        # Unlike in Variable, always eagerly load values into memory

        if not isinstance(self._data, PandasIndexingAdapter):

            self._data = PandasIndexingAdapter(self._data)

 

So I’m stuck for now.

 

Alister

You received this message because you are subscribed to a topic in the Google Groups "xarray" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/xarray/z3XHfwUVeWs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to xarray+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/m235e3wvav.fsf%40gmail.com.

 

[Deepak Cherian]

You could try starting with https://github.com/dcherian/crsindex/blob/main/crsindex.ipynb and primarily modify the use of `spatial_ref` and the `.sel` method. In .sel you would do the interpolation rather than defer to the underlying pandas index which is what it does currently.

On a higher level, I'd be really happy to merge this functionality (CFInterpolatedIndex?) in cf-xarray. We could start by merging the interpolator functions and then work from there to more convenient abstractions.

Deepak

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/D9A49F64-63D1-4AA7-8642-251EEF603BF6%40hxcore.ol.

[Alister Trabattoni]

@Deepak Cherian: I would be happy to merge this functionality in cf-xarray.

I played a bit with your notebook but I have the feeling that your solution cannot be used as it. If a understand well, the new index class takes as input the coordinates. In my case I don’t want to use array coordinates but special objects containing a few tie_indices and tie_values. For now I don’t see how to pass those to the indexer. So I guess what Benoit proposes is the way to go: create a duck array class that can be accepted as coordinate.

 

@Benoît Bovy: Is this duck array coordinate feature planned for soon ? Is it included in the explicit indexing refactoring ? Can a newcomer like me can help on this or is it a quite tough problem ?   

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/f14e705f-0011-f0ee-c63f-81016df89b8c%40gmail.com.