Memory limit

[jmruiz]

Hi,

someone knows the limit in size or rows or anything for a Workbook to
execute a save correctly?

I'm working with a Workbook with 65rows*6pages and I get the following
error:

MemoryError:
File "e:\FUENTES\informeTx\informeTxTipo2.py", line 251, in <module>
creaXLS(ficheroSalida,lasFilas)
File "e:\FUENTES\informeTx\informeTxTipo2.py", line 178, in creaXLS
book.save(nombreXLS)
File "c:\python26\lib\site-packages\xlwt-0.7.2-py2.6-win32.egg\xlwt
\Workbook.py", line 634, in save
File "c:\python26\lib\site-packages\xlwt-0.7.2-py2.6-win32.egg\xlwt
\Workbook.py", line 628, in get_biff_data

Any idea?

Thanks!

[John Machin]

On 23/09/2010 7:10 PM, jmruiz wrote:
> Hi,
>
> someone knows the limit in size or rows or anything for a Workbook to
> execute a save correctly?

No. It depends on too many factors.

>
> I'm working with a Workbook with 65rows*6pages and I get the following
> error:
>
> MemoryError:

page == sheet?
Do you mean 65 rows or 65000 rows?
How many columns and how many rows in each sheet?
Is the data that you are writing in any way abnormal e.g. you are
writing many text cells each of which contains a screen dump or an email
message?

How much memory do you have on your computer? How much (a) real memory
(b) swap space is being used by other processes?
Can you increase the (a) real memory (b) swap space available to xlwt
when you are trying to create this file?
Are you on a *x platform with an unreasonable "ulimit"?

Have you considered using Worksheet.flush_row_data ?

"""If a large number of rows have been written to a Worksheet and memory
usage is
becoming a problem, the flush_row_data method may be called on the
Worksheet.
Once called, any rows flushed cannot be accessed or modified.
It is recommended that flush_row_data is called for every 1000 or so
rows of a normal
size that are written to an xlwt.Workbook. If the rows are huge, that
number should be
reduced.""" ... taken from page 21 of the tutorial available from
http://www.python-excel.org

[John Yeung]

On Tue, Sep 4, 2012 at 6:52 AM, George <georg....@googlemail.com> wrote:
> I am writing an Excel workbook
> (~ 430 MB size) with 30 worksheets and each up to 60,000 lines. I do a
> flush() every 1000 lines. As there is more data every day [...]

> Any other ideas except for splitting the output file in two?

My other idea would be to store the data in an actual database, rather
than in an Excel workbook. That is way more data than is healthy for
Excel, especially when using the .xls format.

John Y.

[Chris Withers]

On 04/09/2012 11:52, George wrote:
>
> # flushing last time
> currentWs.flush_row_data()

How often are you calling flush_row_data?

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[George]

flush_row_data is called ~ 700 times.

[Chris Withers]

On 18/09/2012 17:14, George wrote:
> flush_row_data is called ~ 700 times.

That's not a very useful answer.

How many rows are you writing between calls to flush_row_data?

[John Yeung]

On Thu, Sep 27, 2012 at 7:44 AM, Chris Withers <ch...@simplistix.co.uk> wrote:
>
> On 18/09/2012 17:14, George wrote:
>>
>> flush_row_data is called ~ 700 times.
>
>
> That's not a very useful answer.
>
> How many rows are you writing between calls to flush_row_data?

Maybe he answered it that way because earlier in the thread (in his
first post) he already mentioned that he's flushing every 1000 rows.

Although, I'm not sure it adds up. He said (as I quoted before in
*my* previous post):

On Tue, Sep 4, 2012 at 6:52 AM, George <georg....@googlemail.com> wrote:
> I am writing an Excel workbook
> (~ 430 MB size) with 30 worksheets and each up to 60,000 lines. I do a
> flush() every 1000 lines. As there is more data every day [...]

If there are 30 worksheets, each with 60,000 rows, and a flush every
1000 rows, then shouldn't there be more like 1800 flushes rather than
700?

John Y.

[George]

I just stumbled over my old thread from 2012 because the problem happened again now and I solved it each time by switching to a larger/better Windows PC...

(So here is the current status: I'm flushing every 1000 rows, so I don't remember how I came up with 700 flushes, should be more ~ 18.000 back then)

Here are my observations, maybe it helps other users.

On
Win XP 32 bit (4 GB ram) - xlwt crashes when the Excel output file > 250 MB.

Win 7 64 bit (16 GB ram) but with 32 bit Python - xlwt crashes when the Excel output file > 350 MB.

Win 7 64 bit (4 GB ram) but with 64 bit Python - OK even with Excel output file > 400 MB.

From the size limitations on the different systems it seems to me that the OS memory handling / limitation plays into this because 64 bit Python is working well so far...

George

[John Yeung]

On Mon, Dec 22, 2014 at 8:00 AM, George <georg....@googlemail.com> wrote:
> Here are my observations, maybe it helps other users.

Thank you for sharing; I'm sure it does help some other users.

I still think the *best* answer if xlwt is crashing due to the size of
the generated Excel file is: Don't use Excel to store that much data
in one file.

Really, if the files are THAT big, then they are going to be
problematic for anyone using Excel anyway. That is, the recipient
will need a commensurately "big" PC just to open the damn thing, and
usually it's going to take a long, long time to open, even if it
doesn't crash their PC.

So files that big are only really useful for storing data *to be read
by programs*, not by humans (via the Excel GUI). And if it's going to
be read by programs anyway, then why bother with Excel format?

Sorry for the rant; it's not directed at anyone in particular. I know
sometimes the choice is not ours to make. But huge Excel files are
just an incredibly poor choice, and if we do have the choice, we
should choose something else. Almost anything else would be better.

John Y.

[Adrian Klaver]

+1. Excel != database
>
> John Y.
>


--
Adrian Klaver
adrian...@aklaver.com