xarray-dsp : xarray-compatible digital signal processing tools - possible merge with xr-scipy?

[Ondřej Grover]

Hello,

me and may colleagues have been using a small package I wrote for digital signal processing (DSP) of DataArrays.

The package is available here: https://github.com/smartass101/xarray-dsp

It mainly contains wrappers for IIR and FIR filters and spectral analysis tools from scipy, wavelets (wrapper for pycwt).

It also had some interpolation utils, but those are mostly redundant now with DataArray.interp().

Our common usecase is mostly filtering and analyzing signals where one of the dimensions is the time domain with several milion datapoints. They are usually records from sensors and diagnostics acquired with 2-5 MHz sampling in tokamak discharges.

I've noticed that there is the xr-scipy project which has a similar aim. The contact email for this project was this mailing list, so I'm asking here.

I'd like to ask what would you think about merging our package into xr-scipy. Another possible candidate might be xscale.

My main reasons for seeking a merge with xr-scipy are:

- xr-scipy already uses some similar infrastructure for coords handling and gradient calculation, so why roll my own

- xr-scipy already has a decent documentation set up to which we could contribute

- one project (especially since xr-scipy looks semi-official) will make it more easily discoverable by users and easier to install than hunting down several different projects for all the scipy wrappers

- easier to maintain in a larger team

Let me know what you think.

Kind regards,

Ondřej Grover

[Keisuke FUJII]

Hi, Ondřej

Thanks for the contact.

xr-scipy is actually also a small project to simplify my own data analysis procedure.

I really welcome to any contribution, or even co-managing the project.

Some apologies.

The contact e-mail address of this project should be mine (I just borrowed `setup.py` from xarray and forgot to update e-mail address section...) and I think this project is rather private not (yet) semi-official.

But yes, I hope to make this project semi-official in the future.

To do so, this project needs to be completed and to gain more users.

Anyway, please raise an issue at the repository and continue the discussion there.

[Ondřej Grover]

Dear Fuji-san,

thank you for explaining the situation with xr-scipy

I'm glad to see another fusioneer using xarray! We actually briefly met a month ago at the EPS in Prague where you were presenting GP regression on TS profiles. I was the one suggesting to use pymc3 with a Latent GP for the variable lengthscale.

For future reference, I made and issue

https://github.com/fujiisoup/xr-scipy/issues/4

where we could discuss merging of xarray-dsp and perhaps even xrsigrpoc and xscale which all try to do similar things with scipy. 

I believe it would be beneficial for the community if all these scipy-wrapping efforts were merged into one project using all the strong points from the different projects.

Anyway, I was also wondering that's the general plan or outlook or procedure for writing general-purpose extension libraries based on xarray.

So this is also a question to the wider community.

I noticed there are several naming conventions around:

- xr-scipy

- xarray-topo

- xrsigproc

And that is just the package naming convention, the import-level name is often different, usually it's something like xrscipy or xarray_dsp.

Which one is recommended/preferred?

What is the general recommendation when creating an accessor? Should it be publicly advertised on the xarray website to prevent name clashes between libraries?

And what is the general idea about making such libraries like xr-scipy semi-official? Or at least advertised on the xarray website? I though there was some list some time ago but cannot find it now.

Kind regards,

Ondrej Grover

--
You received this message because you are subscribed to the Google Groups "xarray" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xarray+un...@googlegroups.com.
To post to this group, send email to xar...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/64a4b200-8ee0-4ee3-9ee7-c1a2ddf8a8d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benoît Bovy]

Hi Ondřej,

FYI, there is no real consensus yet on which naming conventions to use for xarray related packages. There has been some discussion is this issue: https://github.com/pydata/xarray/issues/1447.

There is also a specific section on xarray's documentation for projects using xarray, see http://xarray.pydata.org/en/stable/faq.html#what-other-projects-leverage-xarray. Maybe this section could be more discoverable and would deserve its own top entry in TOC?

Cheers,
Benoît Bovy

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/CAOyjJOKQe%2BFtV%3DE9Y57mYEBe_kstba7zqqr-SKg_31UCWGQEBw%40mail.gmail.com.

[Ondřej Grover]

Dear Benoît Bovy,

thank you for the pointer towards the list in the FAQ. I'm glad the list has grown substantionally since I last checked.

Yes, it would be best to make it more discoverable.

There is some hint of such an intent at the bottom of

http://xarray.pydata.org/en/stable/internals.html#extending-xarray

But that is not so easy to find, so I'd say it's better to have some top-level TOC entry for "The xarray ecosystem".

I'm also interested in the opinions and wishes of the community regarding either the merging of similar packages (e.g. scipy wrappers) into a larger one or to rather have several small wrapper libraries for each specific functionality. If the small packages were well discoverable, didn't conflict with each other and had clear specifications/scope, that could also be a good approach, something like scikits.

Kind regards,

Ondrej G.

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/5900cad3-42e6-e0ce-ec2a-ec7c64f7f765%40gmail.com.