One facet of the system involves scanning and loading into memory a
couple of parallel directory trees containing OTO 10^4 files. The
trees don't change during development/testing and the scan takes 30-40
seconds, so to save time I cache the loaded tree structure to disk, in
Perl with module Storable, and in Python with pickle.
In Perl, the save operation produces a file of about 3MB, and both
save and restore take a second or two. In Python, pickle.dump()
produces a similar-size file but takes 20 seconds, and pickle.load()
takes 45 seconds, which is actually LONGER than the time required to
scan the directory trees.
Is there anything I can do to speed up pickle.load() to get
performance comparable to Perl's Storable?
Have you read this:
http://www.python.org/doc/2.6/library/pickle.html
?
Have you considered using cPickle instead of pickle?
Have you considered using *ickle.dump(..., protocol=-1) ?
I'm using Python 3 on Windows (Server 2003). According to the docs
"The pickle module has an transparent optimizer (_pickle) written
in C. It is used whenever available. Otherwise the pure Python
implementation is used."
How can I tell if _pickle is being used?
Answered my own question
>>> import _pickle
>>> dir (_pickle)
['PickleError', 'Pickler', 'PicklingError', 'Unpickler',
'UnpicklingError', '__doc__', '__name__', '__package__']
>>> dir(_pickle.Pickler)
['__class__', '__delattr__', '__doc__', '__eq__', '__format__',
'__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',
'__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__',
'bin', 'clear_memo', 'dump', 'fast', 'memo', 'persistent_id']
>>> dir(_pickle.Pickler)
['__class__', '__delattr__', '__doc__', '__eq__', '__format__',
'__ge__', '__getattribute__', '__gt__', '__hash__', '__init__',
'__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__',
'bin', 'clear_memo', 'dump', 'fast', 'memo', 'persistent_id']
_pickle seems to be there. Also, if I step into the load
call (pydev under Eclipse) it steps into pickle.load() but
won't step into the call to the Unpickler constructor. I
assume that means it's calling out to the C implementation.
The slow performance is most likely due to the poor performance of
Python 3's IO, which is caused by (among other things) bad buffering
strategy. It's a Python 3 growing pain, and is being rewritten.
Python 3.1 should be must faster but it's not been released yet.
As a workaround, mmap the file instead. For example (untested):
f = open('dirlisting.dat','rb')
try:
f.seek(0,2)
size = f.tell()
f.seek(0,0)
m = mmap.mmap(f.fileno(),size,access=mmap.ACCESS_READ)
try:
dir_listing = pickle.loads(m)
finally:
m.close()
finally:
f.close()
Pickling the output left as an exercise.
Carl Banks
My suggestion for the Original Poster is just to try using Python 2.x,
if possible :-)
Bye,
bearophile
3.1a1 is out and I believe it has the io improvements.
Massive ones, too. It'd be interesting to see your results on the alpha.
On 3.1a1 the unpickle step takes 2.4 seconds, an 1875% improvement.
Thanks.
Surely you mean a 94.7% improvement?
Jean-Paul
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
Want to know? Come to PyCon - soon! http://us.pycon.org/
If you double your velocity (100% increase) the time required goes
down by 50%. A 1000% increase in velocity results in a 90% decrease
in time... etc. I guess I equate "performance" to velocity.