is cPickle really this slow?

[Dag]

I have a script which starts by parsing a large file and turning into
a data stucture (a dictonary of lists) which I then use in the script.
This operation however takes about 12-15 seconds. Since the file hardly
ever changes , and if it changes it's because I changed it, thought I'd
save some time by pickeling the datastructure and and simply getting that
at the start of each script. So I pickle my dictionary, which ends up
being a 3.2MB file and try to run my script again, this time getting
the pickled dictonary instead of building it at runtime. This time
however the script takes over 35 seconds to run, close to three times
as long. Am I doing something wrong is loading and parsing larger
pickled files really very slow.

Dag

[Daniel Dittmar]

Have you used the binary option to 'dump'?
dump(object, file[, bin]) : If the optional bin argument is true, the binary
pickle format is used; otherwise the (less efficient) text pickle format is
used (for backwards compatibility, this is the default).

Daniel

[Roy Smith]

Dag <d...@animagicnet.no> wrote:
> So I pickle my dictionary, which ends up
> being a 3.2MB file and try to run my script again, this time getting
> the pickled dictonary instead of building it at runtime. This time
> however the script takes over 35 seconds to run, close to three times
> as long.

Did you use the optional third argument which tells pickle to write the
file out in binary? My experience is that the difference in performance
between reading binary and ascii pickles will be substantial.

[Dag]

Wow. Yea that made all the difference. Runtime went from 35 to 3 seconds.

Dag