[SciPy-User] Pickling timeseries Dates

[Robert Ferrell]

IIRC, timeseries Date objects cannot be pickled. I'm now trying to do
some parallel computation. Using IPython, pickling seems to be
integral parallel computing. I'm not sure about the python
multiprocessing module yet.

My question is, how do I do parallel computation with timeseries data?

thanks,
-robert
_______________________________________________
SciPy-User mailing list
SciPy...@scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

[Pierre GM]

On Dec 10, 2009, at 9:05 AM, Robert Ferrell wrote:
> IIRC, timeseries Date objects cannot be pickled.

Unfortunately, that's true. Gonna have to fix that. However, DateArrays and TimeSeries are pickable. You may want to put your Date in a single-element DateArray

> I'm now trying to do
> some parallel computation. Using IPython, pickling seems to be
> integral parallel computing. I'm not sure about the python
> multiprocessing module yet.
>
> My question is, how do I do parallel computation with timeseries data?

Well, you entered unchartered territories, sorry... Let me know how it goes

[Robert Ferrell]

On Dec 10, 2009, at 7:27 AM, Pierre GM wrote:

> On Dec 10, 2009, at 9:05 AM, Robert Ferrell wrote:
>> IIRC, timeseries Date objects cannot be pickled.
>
> Unfortunately, that's true. Gonna have to fix that. However,
> DateArrays and TimeSeries are pickable. You may want to put your
> Date in a single-element DateArray

Great hint. I had just assumed DateArrays and TimeSeries couldn't be
pickled because Dates can't be pickled. This makes it all much
easier. I've been str() single Date instances, which has been working
fine. single-element DateArray is probably easier.

>
>> I'm now trying to do
>> some parallel computation. Using IPython, pickling seems to be
>> integral parallel computing. I'm not sure about the python
>> multiprocessing module yet.
>>
>> My question is, how do I do parallel computation with timeseries
>> data?
>
>
> Well, you entered unchartered territories, sorry... Let me know how
> it goes

First I have to learn some basic Python multiprocessing stuff.

Thanks,
-robert

[Pierre GM]

On Dec 10, 2009, at 9:41 AM, Robert Ferrell wrote:
>
> On Dec 10, 2009, at 7:27 AM, Pierre GM wrote:
>
>> On Dec 10, 2009, at 9:05 AM, Robert Ferrell wrote:
>>> IIRC, timeseries Date objects cannot be pickled.
>>
>> Unfortunately, that's true. Gonna have to fix that. However,
>> DateArrays and TimeSeries are pickable. You may want to put your
>> Date in a single-element DateArray
>
> Great hint. I had just assumed DateArrays and TimeSeries couldn't be
> pickled because Dates can't be pickled. This makes it all much
> easier. I've been str() single Date instances, which has been working
> fine. single-element DateArray is probably easier.

Note that there's a problem with pickling DateArrays: you lose the frequency at unpickling time. I gonna fix that ASAP. Pickling TimeSeries works quite fine, though (I used them a lot).

[Robert Ferrell]

On Dec 10, 2009, at 8:07 AM, Pierre GM wrote:

> On Dec 10, 2009, at 9:41 AM, Robert Ferrell wrote:
>>
>> On Dec 10, 2009, at 7:27 AM, Pierre GM wrote:
>>
>>> On Dec 10, 2009, at 9:05 AM, Robert Ferrell wrote:
>>>> IIRC, timeseries Date objects cannot be pickled.
>>>
>>> Unfortunately, that's true. Gonna have to fix that. However,
>>> DateArrays and TimeSeries are pickable. You may want to put your
>>> Date in a single-element DateArray
>>
>> Great hint. I had just assumed DateArrays and TimeSeries couldn't be
>> pickled because Dates can't be pickled. This makes it all much
>> easier. I've been str() single Date instances, which has been
>> working
>> fine. single-element DateArray is probably easier.
>
>
> Note that there's a problem with pickling DateArrays: you lose the
> frequency at unpickling time. I gonna fix that ASAP. Pickling
> TimeSeries works quite fine, though (I used them a lot).

Thanks for the heads up. I can work around that pretty easily, but
I'll listen for the fix.

[Robert Ferrell]

On Dec 10, 2009, at 10:35 AM, Robert Ferrell wrote:

>
> On Dec 10, 2009, at 8:07 AM, Pierre GM wrote:
>
>> On Dec 10, 2009, at 9:41 AM, Robert Ferrell wrote:
>>>
>>> On Dec 10, 2009, at 7:27 AM, Pierre GM wrote:
>>>
>>>> On Dec 10, 2009, at 9:05 AM, Robert Ferrell wrote:
>>>>> IIRC, timeseries Date objects cannot be pickled.
>>>>
>>>> Unfortunately, that's true. Gonna have to fix that. However,
>>>> DateArrays and TimeSeries are pickable. You may want to put your
>>>> Date in a single-element DateArray
>>>
>>> Great hint. I had just assumed DateArrays and TimeSeries couldn't
>>> be
>>> pickled because Dates can't be pickled. This makes it all much
>>> easier. I've been str() single Date instances, which has been
>>> working
>>> fine. single-element DateArray is probably easier.
>>
>>
>> Note that there's a problem with pickling DateArrays: you lose the
>> frequency at unpickling time. I gonna fix that ASAP. Pickling
>> TimeSeries works quite fine, though (I used them a lot).
>
> Thanks for the heads up. I can work around that pretty easily, but
> I'll listen for the fix.
>
>

Fernando reminded me that any date scalar will trigger the pickle
problem. Some of my objects have Date instances stashed all over the
place. Converting all those to single-element DateArrays is not
feasible.

Why can't Date instances be pickled, and how hard would it be to
change that?

thanks,
-robert

[Pierre GM]

On Dec 10, 2009, at 10:12 PM, Robert Ferrell wrote:
>>
> Fernando reminded me that any date scalar will trigger the pickle
> problem. Some of my objects have Date instances stashed all over the
> place. Converting all those to single-element DateArrays is not
> feasible.

And converting them to something else, like a datetime or even just a tuple (value,freq)

>
> Why can't Date instances be pickled, and how hard would it be to
> change that?

because we never implemented it on the C side, and that'll depend on whether Matt Knox can do it or not.

[David Baddeley]

It also shouldn't be to hard to register a new pickle handler that takes care of the conversion to e.g. datetime for you. I'm not familiar with timeseries stuff, but wrote something similar to let me pickle slices, which I've included below - as you can see it's pretty trivial. After that you can carry on treating your date (slice) objects as though they were picklable.

import copy_reg

def pickleSlice(slice):
return unpickleSlice, (slice.start, slice.stop, slice.step)

def unpickleSlice(start, stop, step):
return slice(start, stop, step)

copy_reg.pickle(slice, pickleSlice, unpickleSlice)

best wishes,
David

--- On Fri, 11/12/09, Pierre GM <pgmde...@gmail.com> wrote:

[Robert Ferrell]

On Dec 10, 2009, at 8:37 PM, Pierre GM wrote:

> On Dec 10, 2009, at 10:12 PM, Robert Ferrell wrote:
>>>
>> Fernando reminded me that any date scalar will trigger the pickle
>> problem. Some of my objects have Date instances stashed all over the
>> place. Converting all those to single-element DateArrays is not
>> feasible.
>
> And converting them to something else, like a datetime or even just
> a tuple (value,freq)

I often grab a Date from a DateArray. Of course, I could try to use
datetime for every date scalar, and DateArray for arrays of dates.
But that's not nearly as convenient nor robust as using a single
representation whenever I need a date.

>
>
>>
>> Why can't Date instances be pickled, and how hard would it be to
>> change that?
>
>
> because we never implemented it on the C side, and that'll depend on
> whether Matt Knox can do it or not.

Understood.

[Robert Ferrell]

Thanks for the suggestion. I may try that. Seems pretty straight
forward.

-r

[Robert Ferrell]

On Dec 10, 2009, at 9:28 PM, David Baddeley wrote:

> It also shouldn't be to hard to register a new pickle handler that
> takes care of the conversion to e.g. datetime for you. I'm not
> familiar with timeseries stuff, but wrote something similar to let
> me pickle slices, which I've included below - as you can see it's
> pretty trivial. After that you can carry on treating your date
> (slice) objects as though they were picklable.
>
> import copy_reg
>
> def pickleSlice(slice):
> return unpickleSlice, (slice.start, slice.stop, slice.step)
>
> def unpickleSlice(start, stop, step):
> return slice(start, stop, step)
>
> copy_reg.pickle(slice, pickleSlice, unpickleSlice)
>
> best wishes,
> David
>

That was a great suggestion. Seems to work for me. If I import this
at the top of my package (or as soon as I can) will it be available
whenever I need it?

In case anybody else needs to pickle Dates, here's the code that seems
to work for me.

=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
========================================================================
# Attempt to pickle timeseries date instances.
# This converts a ts.Date instance to a string
# Frequency is lost, but I'm only using 'd', so that's okay for me.

import pickle
import copy_reg
import scikits.timeseries as ts

def pickleDT(dt):
"""Pickle __reduce__ method for a date instance dt by converting to a
string, then pickling."""
dA = str(dt)
ds = pickle.dumps(dA)
# This is what __reduce__ is supposed to return
return unpickleDT, (ds,)

def unpickleDT(dA):
"""Unpickle a date instance with frequency 'd'."""
ds = pickle.loads(dA)
dt = ts.Date('d', ds)
return dt

# Register these methods
copy_reg.pickle(ts.Date, pickleDT, unpickleDT)

if __name__ == '__main__':
sD = ts.Date('d', '2009-11-30')
ds = pickle.dumps(sD)
dt = pickle.loads(ds)
print 'Started with %s, unpickled %s.' % (sD, dt)
if dt == sD:
print 'Yeah, flawless victory!'
else:
print 'Boo, shameful failure.'