is it possible to define to ignore nan vale when i do ds.groupby('time.week').mean()

[yar michl]

is it possible to define to ignore nan vale when i do  ds.groupby('time.week').mean()

i have a a xarray with with daily data of a year with a dims of ('time', 'y', 'x') in the daily data sometime i have nan vale

and whaen i do ds.groupby('time.season').mean() i  got nan in some In some places 

in pandas.DataFrame i understood it  will automatically exclude NaN numbers when i do mean

i didnt see a way to do igonre nan in gropby in xarray

should i  mask the nan vale ? and then do  ds.groupby('time.week').mean()

thek for the help

and thanks for the xarray module

[Stephan Hoyer]

Xarray excludes NaN values by default, just like pandas. If desired, you can disable this by calling .mean(skipna=False).

You will still see NaNs in your results if every value to be averaged is NaN.

--
You received this message because you are subscribed to the Google Groups "xarray" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xarray+unsubscribe@googlegroups.com.
To post to this group, send email to xar...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/4ae6c887-6b22-41df-b787-5168d9c21bc0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[yar michl]

Thanks

On Wednesday, October 12, 2016 at 2:36:06 AM UTC+3, Stephan Hoyer wrote:

Xarray excludes NaN values by default, just like pandas. If desired, you can disable this by calling .mean(skipna=False).

You will still see NaNs in your results if every value to be averaged is NaN.

On Tue, Oct 11, 2016 at 4:32 PM, yar michl <yaro...@gmail.com> wrote:

is it possible to define to ignore nan vale when i do  ds.groupby('time.week').mean()

i have a a xarray with with daily data of a year with a dims of ('time', 'y', 'x') in the daily data sometime i have nan vale

and whaen i do ds.groupby('time.season').mean() i  got nan in some In some places 

in pandas.DataFrame i understood it  will automatically exclude NaN numbers when i do mean

i didnt see a way to do igonre nan in gropby in xarray

should i  mask the nan vale ? and then do  ds.groupby('time.week').mean()

thek for the help

and thanks for the xarray module

--
You received this message because you are subscribed to the Google Groups "xarray" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xarray+un...@googlegroups.com.

[Edward]

Does this also apply to custom functions using .apply()?

The function I am applying is using a function from scipy that still computes even if the input array values are nan   (genextreme.fit())

I tried adding a line to my function to ignore if the array is nans, but this slowed things by a factor of 3, i.e. it is faster to compute the nans than to exclude them.

Currently I am doing 

stacked.groupby('allpoints').apply(func)

Would there be any good way to exclude the nans before the groupby proceedure?

On Wednesday, 12 October 2016 01:36:06 UTC+2, Stephan Hoyer wrote:

Xarray excludes NaN values by default, just like pandas. If desired, you can disable this by calling .mean(skipna=False).

You will still see NaNs in your results if every value to be averaged is NaN.

On Tue, Oct 11, 2016 at 4:32 PM, yar michl <yaro...@gmail.com> wrote:

is it possible to define to ignore nan vale when i do  ds.groupby('time.week').mean()

i have a a xarray with with daily data of a year with a dims of ('time', 'y', 'x') in the daily data sometime i have nan vale

and whaen i do ds.groupby('time.season').mean() i  got nan in some In some places 

in pandas.DataFrame i understood it  will automatically exclude NaN numbers when i do mean

i didnt see a way to do igonre nan in gropby in xarray

should i  mask the nan vale ? and then do  ds.groupby('time.week').mean()

thek for the help

and thanks for the xarray module

--
You received this message because you are subscribed to the Google Groups "xarray" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xarray+un...@googlegroups.com.

[Stephan Hoyer]

On Wed, Nov 9, 2016 at 5:51 AM, Edward <eam...@gmail.com> wrote:

Does this also apply to custom functions using .apply()?

The function I am applying is using a function from scipy that still computes even if the input array values are nan   (genextreme.fit())

I tried adding a line to my function to ignore if the array is nans, but this slowed things by a factor of 3, i.e. it is faster to compute the nans than to exclude them.

Currently I am doing 

stacked.groupby('allpoints').apply(func)

Would there be any good way to exclude the nans before the groupby proceedure?

If the variable you are grouping over ("allpoints") has NaNs, then strange things may happen -- I'm not sure if that case has been tested.

But if only your data has NaNs, it will be up to your custom function to handle them as you see fit. Xarray handles skipna=True by using NumPy functions like nanmean, not by adding on its own logic.

[Edward]

Thanks Stephan

yes, it is just the data that has NaNs.

So having just read your comments about caching, could that be used to efficiently cache the result of scipy.genextreme.fit([NaN, NaN... NaN]) ?

[Stephan Hoyer]

Xarray's built-in caching is pretty different -- it's for data lazily loaded from disk.

You should look into dask.array if you want caching for computation.

--

You received this message because you are subscribed to the Google Groups "xarray" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xarray+unsubscribe@googlegroups.com.

To post to this group, send email to xar...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/xarray/7c0f6c4f-5b2c-4a7e-9d20-318ea7a9d6d5%40googlegroups.com.