memory issues with lots of files

28 views
Skip to first unread message

chacham

unread,
Sep 22, 2011, 6:18:58 PM9/22/11
to Spreadsheet::ParseExcel
i am processing multiple text files (testing with a few thousand right
now). the method is basically:

1) open an excel template
2) read format from second row (first row is header, only two rows,
one sheet, 15-30 columns)
3) put data from data file (text) into template. (all different sizes,
a few k to 128k)
4) loop

this is using a lot of memory on the (2GB 2008, running activestate
perl 64) server, it steadily grows to 1.7GB, hangs out there for a
while, then starts giving out of memory errors. it processes ~1000
files before generating errors.

i figured it was the parser instantiation:

$template = Spreadsheet::ParseExcel::SaveParser->new->Parse($_[0]);

IOW, never bothered capturing the parser object itself, and instead
went straight to the file.

So, (and i'm not too experience in perl, so i have what to learn) i
figured that would be created thousands of parser objects for no
reason, so instead i created just one, and reused it:

once: my $parser = Spreadsheet::ParseExcel::SaveParser->new();

reused: $template = $template = $parser->Parse($_[0]);

that didn't seem to help, so i tried once more by undef, making the
open sub:

undef $parser;
$parser = Spreadsheet::ParseExcel::SaveParser->new();

$template = $parser->Parse($_[0]);

that didn't help either.

i am making an assumption that it is the parser using all the memory,
i am guessing i need to help it get destroyed. i have no idea if this
is correct, or even if it is, what to do. please point me in the right
direction.

when perl terminates, the memory is released immediately, and i have
quite a few excel files.
Reply all
Reply to author
Forward
0 new messages