[SciPy-User] Writing an Interpolation Function to Disk?

[Sloan Lindsey]

Hi,
I've been using the new interpolation routines
(scipy.interpolate.CloughTocher2DInterpolator) quite happily in my
project but I'm wondering if there is a way to save the output from
the interpolation function to disk. I'm using rather large datasets
(200,000+ points) and it takes an appreciable time to recalculate the
interpolant every time that I run my program. I'd like it if I could
dump the baked interpolant to disk and then restore it on execution.
Of course I probably need to generate the interpolant per machine but
I can deal with that. Is there any cool way to do this?
Thanks,
Sloan Lindsey
Technical University of Vienna
_______________________________________________
SciPy-User mailing list
SciPy...@scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

[Pauli Virtanen]

On Fri, 25 Feb 2011 12:42:47 +0100, Sloan Lindsey wrote:
> I've been using the new interpolation routines
> (scipy.interpolate.CloughTocher2DInterpolator) quite happily in my
> project but I'm wondering if there is a way to save the output from the
> interpolation function to disk. I'm using rather large datasets
> (200,000+ points) and it takes an appreciable time to recalculate the
> interpolant every time that I run my program.

I assume most of the time is taken by constructing the Delaunay
triangulation (scipy.spatial.Delaunay(points)).

> I'd like it if I could
> dump the baked interpolant to disk and then restore it on execution. Of
> course I probably need to generate the interpolant per machine but I can
> deal with that. Is there any cool way to do this?

Ideally, this would work:

import pickle
f = open('file.pck', 'wb')
pickle.dump(interpolator, f)
f.close()

but it doesn't work, since there's a small unnecessary technical snatch
(scipy.spatial.interpnd has the wrong __name__). This will be fixed in
the next version of Scipy, so a small workaround is needed in the
meantime:

class PickleableInterpolator(CloughTocher2DInterpolator):
def __getstate__(self):
return (self.tri, self.values, self.grad, self.is_complex,
self.fill_value, self.values_shape)
def __setstate__(self, data):
self.tri, self.values, self.grad, self.is_complex, \
self.fill_value, self.values_shape = data
self.points = self.tri.points

This kind of mucking around might break in future Scipy versions, so I
suggest you put it in 'if scipy.__version__ == "0.9.0":'

Note that pickling objects like this is quite brittle --- the internal
details of what needs to be pickled may change, so do not expect the
pickle files to work across different Scipy versions.

--
Pauli Virtanen

[Pauli Virtanen]

On Fri, 25 Feb 2011 20:40:26 +0000, Pauli Virtanen wrote:
[clip]
> pickle.dump(interpolator, f)

For most stuff, one should do

pickle.dump(interpolator, f, protocol=2)

so that all Numpy arrays get dumped as binary data rather than first
converted to text in the pickle stream.

[Gael Varoquaux]

On Fri, Feb 25, 2011 at 08:42:02PM +0000, Pauli Virtanen wrote:
> On Fri, 25 Feb 2011 20:40:26 +0000, Pauli Virtanen wrote:
> [clip]
> > pickle.dump(interpolator, f)

> For most stuff, one should do

> pickle.dump(interpolator, f, protocol=2)

> so that all Numpy arrays get dumped as binary data rather than first
> converted to text in the pickle stream.

Or use joblib's pickler subclass:
http://packages.python.org/joblib/generated/joblib.dump.html
which can only be read using joblib's load function, but will pickle
numpy arrays as .npy, as thus be fast at save and load.

G