Big Excel (xlsx) file parsing (docjure, incanter...)

[Stanislav Sobolev]

Hello guys. I have excel file with huge columns in there.
When i used some plugin(docjure, or anything else) it shows me CompilerException java.lang.OutOfMemoryError: Java heap space, compiling
How can i handle that big file without converting it to csv? 
Primary problem in xlsx structure, it doesnt have lines or something, unlike csv.

[Vijay Kiran]

Hi,

I don't think docjure supports streaming options yet  - but you can try using Apache POI streaming API - http://poi.apache.org/spreadsheet/index.html - that should help in reading large files.

HTH,

@vijaykiran

[Stanislav Sobolev]

Allright, but if use SXSSF how can i stream excel file to that? 
https://github.com/ktsujister/clj-tsv2xls/blob/master/src/tsv2xls/core.clj Like in this example
can anybody provide me code example of using that streaming please?
with-open [out-stream (io/output-stream outfile) and further?


суббота, 7 сентября 2013 г., 12:30:43 UTC+6 пользователь Vijay Kiran написал:

[scott tudd]

orthogonal, perhaps helpful:

I wrote a clojure (wrapper) library that "streams" data in-and-out of Excel quite easily (and other applications, mostly finance, that use DDE).

especially useful if you need to monitor or updates changes to cells.

https://github.com/tuddman/clj-dde

feedback welcome.

[Ragnar Dahlén]

Have you tried increasing the heap size of your JVM using the -Xmx and -Xms options? 

We're loading some spreadsheets that are 80M in size and have to run with a ~1.5G heap to handle this...