Mimimizing memory usage with xlwt

[Rémy Roy]

Hello,

I'm running xlwt in a memory limited environment and I'm trying to minimize the memory it uses to write a single file. My current application uses ~160 MB of memory to write a ~2.3 MB xls file which I felt was quite a lot. I found some tips about reducing this using the flush_row_data method each X rows (I used 200) on the worksheet objects. I was already reusing the single custom style for my datetime values.

I decided to check out the big-35Mb.py example (https://raw.github.com/python-excel/xlwt/master/xlwt/examples/big-35Mb.py) to see how much memory it would use. I've used Valgrind Massif (see massif.out.11071 for the raw output and massif-vis.png for the visualized version) to check it out. That example used 481 MB of memory at his peak on my system.

Are those numbers in line with what you guys know about memory usage for xlwt?
Isn't there a way to reduce it further?

It seems like xlwt needs to have all its data in memory before it can dumps (using the .save method) it on disk or in a binary xls format. Wouldn't it be possible to have a forward-only streaming xls writer that wouldn't need that much memory, that wouldn't need to know in advance all its data it has to write or is the xls file format just not workable in a streaming way?

These tests were perform using Python 2.7.5 and xlwt 0.7.5.

Regards,

[John Machin]

On Saturday, October 19, 2013 7:25:33 AM UTC+11, Rémy Roy wrote:

 

 is the xls file format just not workable in a streaming way?

correct