Forward-looking rolling window?

[Paul Hobson]

Is there anyway to get the window of a rolling function to look forward? My current work around is as follows:

import numpy as np

import pandas

drange = pandas.date_range('1/1/2013', '3/1/2013', freq='1H')

x = pandas.Series(np.random.normal(size=len(drange)), index=drange)

# daily rolling sum...

xx = pandas.rolling_sum(x, 24) # looks back

yy = pandas.Series(xx[23:].values, index=xx.index[:-23]) # offset to "look forward"

(Background: the application of this to continuously assess the quantitative precipitation forecasts from NOAA)

Thanks,

-paul

[Wes McKinney]

> --
> You received this message because you are subscribed to the Google Groups
> "PyData" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pydata+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Not at the moment. this would be a nice addition to the library, though

[Paul Hobson]

Cool. I'll try to take a look at the source code soon and see if I can pull a PR together.

[andy hayden]

As a belated answer, this came up on SO http://stackoverflow.com/questions/22820292/how-to-use-pandas-rolling-functions-on-a-forward-looking-basis.

You can look at rolling_sum of the reversed Series, and reverse that:

    pd.rolling_sum(x[::-1], 24)[::-1]

This may "just work" on all (?) rolling functions ...Would make good PR!

Issue here: https://github.com/pydata/pandas/issues/6772

[andy hayden]

As a belated answer, this came up on SO http://stackoverflow.com/questions/22820292/how-to-use-pandas-rolling-functions-on-a-forward-looking-basis.

You can look at rolling_sum of the reversed Series, and reverse that:

    pd.rolling_sum(x[::-1], 24)[::-1]

This may "just work" on all (?) rolling functions ...Would make good PR!

Issue here: https://github.com/pydata/pandas/issues/6772

On Sunday, 23 June 2013 15:32:38 UTC-7, Paul Hobson wrote:

[Skipper Seabold]

Some advertising.

What you want is also a kind of convolution filter. There's a PR [1]
in statsmodels to add these along with a recursive filter and some
seasonal decomposition stuff. It's mostly done. It just needs a once
over. It wraps pandas code correctly, though you'd have to shift the
index back, because it's assumed the filter is centered around t.
There's an option for a one-sided, backwards looking filter, but I
suppose we could add a forwards-looking one as well to make the shift
unnecessary. I never knew a use case for it. There are many ways to do
this using convolution, fft convolution, or scipy.signal.lfilter, FYI.
FFT might be tough to beat for long series.

[1] https://github.com/statsmodels/statsmodels/pull/1484

[~/statsmodels/statsmodels-cython/statsmodels/tsa/filters/]
[8]: drange = pd.date_range('1/1/2013', '3/1/2013', freq='1H')

[~/statsmodels/statsmodels-cython/statsmodels/tsa/filters/]
[9]: x = pd.Series(np.random.random(size=len(drange)), index=drange)

[~/statsmodels/statsmodels-cython/statsmodels/tsa/filters/]
[10]: pd.rolling_sum(x[::-1], 24)[::-1].head(5)
[10]:
2013-01-01 00:00:00 10.912721
2013-01-01 01:00:00 11.697446
2013-01-01 02:00:00 12.077230
2013-01-01 03:00:00 11.589507
2013-01-01 04:00:00 12.504221
Freq: H, dtype: float64


[~/statsmodels/statsmodels-cython/statsmodels/tsa/filters/]
[11]: filt = [0] * 23 + [1] * 24 # only care about upcoming 24


[~/statsmodels/statsmodels-cython/statsmodels/tsa/filters/]
[12]: sm.tsa.filters.convolution_filter(x, filt).shift(-23,
freq='H').dropna().head(5)
[12]:
2013-01-01 00:00:00 10.912721
2013-01-01 01:00:00 11.697446
2013-01-01 02:00:00 12.077230
2013-01-01 03:00:00 11.589507
2013-01-01 04:00:00 12.504221
Freq: H, dtype: float64

> For more options, visit https://groups.google.com/d/optout.

[Paul Hobson]

Haha wow. Resurrected! Seems so obvious now. Since that ship that prompted this original question 9 months ago has long sailed, I might as well ask, is it possible to include a previously returned value in a rolling_apply() calculation?

Here's an over-simplified example: I want my rolling function to sum up a row. If that result is even, keep that sum. Otherwise, use the previous result. It would look like this in the end:

              A    B    C  sum  rolled
date                                  
2010-01-31  172   47  117  336     336
2010-02-28  192   67  195  454     454
2010-03-31  103    9   21  133     454
2010-04-30   36   87   70  193     454
2010-05-31   88  140   58  286     286
2010-06-30  193   39   87  319     286

But you would get there with:

# setup

import numpy as np
import pandas
np.random.seed(0)

N = 6
data = np.random.random_integers(low=0, high=200, size=(N, 3))
#cols = pandas.MultiIndex.from_product([('cost', 'value'), (1, 2, 3)], names=['quantity', 'user_id'])
dates = pandas.DatetimeIndex(freq='1M', start='2010-01-01', periods=N, name='date')
df = pandas.DataFrame(data, index=dates, columns=list('ABC'))

# mystery magic?

df['rolled'] = pandas.rolling_apply(df, crazy_function, window=2)

Again, I know this is a silly example where this isn't necessary, but my actual application is pretty complex and would just muddy the issue.

Thanks,

-paul