multiannual filter

[Omnain Kutos]

Hello!

We have been ERDDAP users and greatly appreciate the valuable functionalities it offers. Through our experience, we have identified a potential enhancement that we believe would benefit the ERDDAP user community, and we would like to request the possibility of the addition of a new feature. We propose the inclusion of an option to filter data by a range of months (per year): a multiannual filter.

The utility behind this is to enable users to extract data for specific months, allowing for more nuanced and comparable analyses across different years. For example, being able to filter data solely for the months of July through November each year, would facilitate year-on-year comparisons while excluding data from other months.

We understand that features development are finite, but we believe that this addition could improve the user experience and broaden the applicability of ERDDAP in diverse research scenarios. However, if there is already a way to achieve this functionality that we may have overlooked, we would greatly appreciate any guidance in this regard.

We look forward to hearing your thoughts on this and would be happy to provide any clarification as needed.

Thank you!

Omnain

[bobsimons2.00]

Can you please write out what the syntax for the filter would be, and what exactly the filter would do with the various options?

[Omnain Kutos]

Hi Bob,

I wanted to share an image that illustrates our specific data filtering needs. This image outlines the criteria for filtering data based on dates and years. Part of the URL would be structured as follows and it would return a dataset with only the data from March 27th to September 22nd, for the years 2014 to 2020.

calendar-date>=03-27&calendar-date<=09-22&year>=2014&year<=2020

I hope this this clarifies our requirements.

Thank you for your time to this matter !

Omnain

[Roy Mendelssohn - NOAA Federal]

Hi Omnain:

For my own information I was wondering if you see this for tables, grids or both, and whether you want to use this on your own ERDDAP (ie where a modification to the dataset might meet your needs) or on other ERDDAPs, that would require a change to ERDDAP. Also what if this capability were available in a script would that meet you needs (and if so what language do you use - R? Python? Matlab?).

Bob or Chris may have other thoughts and I could be wrong on this, but I am concerned that this will end up as more work than is obvious at first blush. For tables, for example, a function like "distinct()" would make sense for all datasets, though for some it might not do much. But not every dataset in ERDDAP has time, so having that function as an option on those dataset pages would make no sense. So besides writing a function to filter the data as you described, code would have to check such an option made sense for a dataset, and there would have to be differences in how the dataset page is generated based on that. So this would no longer just be a question of writing the code to do the filter, but also re-writing some of the guts of ERDDAP on how it generates the pages you see, and then it all has to be tested over a broad class of use cases. And that can be a lot of work

There is a long list of user requests for features, and we have to evaluate each for the work involved relative to the reward, and also look to see if there are alternate ways to do what is needed. Your example for instance, can be done by first obtaining the data for the time bounds you desire and then filtering out only the data you want in R or Matlab or Python - which s a pretty straightforward operation, or else looping over years for the desired day range in each year and combining the results. If either of these would meet your needs, I would be happy to work with you in developing a script to do this in the language of your choice.

Again, maybe Bob or Chris see a simple way to do this and that would be great.

-Roy

> On Dec 19, 2023, at 1:16 PM, Omnain Kutos <kut...@ogsl.ca> wrote:
>
> Hi Bob,
>
> I wanted to share an image that illustrates our specific data filtering needs. This image outlines the criteria for filtering data based on dates and years. Part of the URL would be structured as follows and it would return a dataset with only the data from March 27th to September 22nd, for the years 2014 to 2020.
>
> calendar-date>=03-27&calendar-date<=09-22&year>=2014&year<=2020
>
>
>
>
>

> I hope this this clarifies our requirements.
>
> Thank you for your time to this matter !
>
> Omnain
>
> Le jeudi 7 décembre 2023 à 16:35:07 UTC-5, bobsimons2.00 a écrit :
> Can you please write out what the syntax for the filter would be, and what exactly the filter would do with the various options?
>
> On Thursday, December 7, 2023 at 4:09:40 PM UTC-5 kut...@ogsl.ca wrote:
> Hello!
>
> We have been ERDDAP users and greatly appreciate the valuable functionalities it offers. Through our experience, we have identified a potential enhancement that we believe would benefit the ERDDAP user community, and we would like to request the possibility of the addition of a new feature. We propose the inclusion of an option to filter data by a range of months (per year): a multiannual filter.
>
> The utility behind this is to enable users to extract data for specific months, allowing for more nuanced and comparable analyses across different years. For example, being able to filter data solely for the months of July through November each year, would facilitate year-on-year comparisons while excluding data from other months.
>
> We understand that features development are finite, but we believe that this addition could improve the user experience and broaden the applicability of ERDDAP in diverse research scenarios. However, if there is already a way to achieve this functionality that we may have overlooked, we would greatly appreciate any guidance in this regard.
>
> We look forward to hearing your thoughts on this and would be happy to provide any clarification as needed.
>
> Thank you!
>
> Omnain
>

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/4abd226c-ec3c-4f91-979a-c4aa8fdf0c98n%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[bobsimons2.00]

I'll go along with some of Roy's comments.

The type of filtering you are requesting seems very specific to something you need to do. It seems like this filtering is just the first step in an analysis you are doing and that you'll be using some other program (Matlab? R?) to do further filtering and analysis. In that case, I think it makes more sense to just get all of the relevant data (including some you don't want) by requesting all of the data from 2014-03-27 through 2020-09-22 and then doing the filtering and further analysis in Matlab, R, or wherever.

But I could be wrong. Please feel free to redirect me and convince me that there is a general need for this feature.

[Omnain Kutos]

Hi Bob and Roy,

Absolutely, we genuinely value both of your suggestions. Thank you for sharing your perspectives. Your insight will be valuable in explaining to our partners the considerations and possibilities regarding the requested features.

Thank you!

Omnain

[Roy Mendelssohn - NOAA Federal]

I am working on some utilities that build on the R package 'rerddap" one of which does just what you are requesting. No ETA but I have it essentially working (there is a big difference between me running it for myself and code suitable to distribute to others). If you are interested I will let you know when it is available for testing, it will be on my Github page.

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/23e88142-e3ab-419e-b46b-6a0940c76193n%40googlegroups.com.

[Omnain Kutos]

Hi Roy,

I'm happy to learn that one of your utilities aligns closely with what our partners are looking for. We are definitely interested in being informed once it's ready for testing.

Thank you!

Omnain

[Roy Mendelssohn - NOAA Federal]

Hi Omnain:

If you are still interested in this and use R, please look at my package at:

https://github.com/rmendels/rerddapUtils

in particular the function 'griddap_season()'. I reiterate what is on that page, at the moment the package is for testing only, do to use it in production or any thing important. I believe it all works fine but it has not been tested enough to not warn people. But if you could give it a try and provide any feedbacks/problems, that would be appreciate. There is a vignette that I hope clarifies the usage.

Thanks,

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/fe3891e6-56af-48a2-8215-be9fc4cceab7n%40googlegroups.com.