slow Python 3.0 write performance?
Istvan Albert
For me (on Windows) it runs over 7 times slower with Python 3.0
import time
lo, hi, step = 10**5, 10**6, 10**5
# writes increasingly more lines to a file
for N in range(lo, hi, step):
fp = open('foodata.txt', 'wt')
start = time.time()
for i in range( N ):
fp.write( '%s\n' % i)
fp.close()
stop = time.time()
print ( "%s\t%s" % (N, stop-start) )
Mike Driscoll
Ran on Windows XP virtual machine:
3.0 output:
100000 0.889999866486
200000 1.79699993134
300000 2.875
400000 3.73399996758
500000 4.71899986267
600000 5.59400010109
700000 7.04699993134
800000 7.31299996376
900000 8.375
2.5.2 output:
100000 0.156000137329
200000 0.296999931335
300000 0.640000104904
400000 0.640000104904
500000 0.78200006485
600000 0.952999830246
700000 1.13999986649
800000 1.25
900000 1.42199993134
Slowness in this exercise is confirmed on Windows XP.
Mike
Bruno Desthuilliers
> Could someone run the code below on both Python 2.5 and 3.0
>
> For me (on Windows) it runs over 7 times slower with Python 3.0
Already covered, I think:
http://groups.google.com/group/comp.lang.python/browse_frm/thread/9046eee09137c657#
sk...@pobox.com
Istvan> Could someone run the code below on both Python 2.5 and 3.0 For
Istvan> me (on Windows) it runs over 7 times slower with Python 3.0
...
I/O was completely rewritten for Python 3.0. Stdio is no longer used. At
the moment I believe much of the io subsystem is still implemented in
Python. Note these comments in io.py:
# New I/O library conforming to PEP 3116.
# This is a prototype; hopefully eventually some of this will be
# reimplemented in C.
It should get faster over time. It will get faster over a shorter period of
time if people contribute patches.
--
Skip Montanaro - sk...@pobox.com - http://smontanaro.dyndns.org/
Istvan Albert
> It should get faster over time. It will get faster over a shorter period of
> time if people contribute patches.
I see, thanks for the clarification.
I will make the point though that this makes python 3.0 unsuited for
anyone who has to process data. One could live with slowdowns of say
20-50 percent, to get the goodies that 3.0 offers, but when a process
that takes 1 second suddenly starts to take 10, it is makes the
situation untenable.
best,
Istvan
Christian Heimes
> I see, thanks for the clarification.
>
> I will make the point though that this makes python 3.0 unsuited for
> anyone who has to process data. One could live with slowdowns of say
> 20-50 percent, to get the goodies that 3.0 offers, but when a process
> that takes 1 second suddenly starts to take 10, it is makes the
> situation untenable.
The speed issue slipped through the alpha and beta releases. Apparently
no user has tested Python 3.0 with large files so far. Some bugs just
can't be found by the developers.
I've fixed the read() slowness yesterday. You'll get the fix in the next
release of Python 3.0 in a couple of weeks.
Christian
MRAB
that might help! :-)
Christian Heimes
> Does pysco with with Python 3.0 (the homepage says 2.5)? If it does then
> that might help! :-)
No, it won't help.
bearoph...@lycos.com
> I've fixed the read() slowness yesterday. You'll get the fix in the next
> release of Python 3.0 in a couple of weeks.
Thank you.
Bye,
bearophile
Istvan Albert
> I've fixed the read() slowness yesterday. You'll get the fix in the next
> release of Python 3.0 in a couple of weeks.
Does this fix speed up the write() function as well?
A previous poster suggested that in this case the slowdown is caused
by the new io code being written in python rather than C.
Istvan
Christian Heimes
> A previous poster suggested that in this case the slowdown is caused
> by the new io code being written in python rather than C.
For text mode Python 3's write() method is slower than Python 2.x's
method because all text is encoded. The slowdown is mostly caused by
additional code for line ending detection and decoding/encoding. The new
io library does more than the old file object
So far the new io library hasn't been optimized yet, too. Please give it
some time and report speed issues at http://bugs.python.org/.
Christian
Terry Reedy
> Istvan Albert wrote:
>> A previous poster suggested that in this case the slowdown is caused
>> by the new io code being written in python rather than C.
>
> method because all text is encoded. The slowdown is mostly caused by
> additional code for line ending detection and decoding/encoding. The new
> io library does more than the old file object
>
> So far the new io library hasn't been optimized yet, too. Please give it
> some time and report speed issues at http://bugs.python.org/.
On WinXP, tests 3 times each
-------------------------------------------
3.0
---
import time
line = 'a'*99 + '\n'
text = line*500000
start=time.time()
open('wtest', 'w').write(text)
print(time.time()-start)
# 3.4 to 4.0
import time
line = 'a'*99 + '\n'
text = line*500000
start=time.time()
text = bytes(line,'ascii')*500000
print(time.time()-start)
# 1.1 to 1.25
start=time.time()
open('wtest', 'wb').write(text) # note wb
print(time.time()-start)
# 1.5 to 1.8
----------------------------------------------
2.5
---
import time
line = 'a'*99 + '\n'
text = line*500000
start=time.time()
open('wtest', 'w').write(text)
print(time.time()-start)
# 3.3 to 4.1
import time
line = 'a'*99 + '\n'
text = line*500000
start=time.time()
open('wtest', 'wb').write(text) # note wb
print(time.time()-start)
# 1.0 to 1.6
-------------------------------------------------
Conclusion:
1. on WinXP, writing with 3.0 has no slowdown versus 2.5
2. on WinXP, binary mode writing is about 3 times faster that text mode
writing with its line-ending expansion. If that is not needed, binary
mode and explicit bytes-text conversion is about twice as fast.
Terry Jan Reedy