360MB Xml on Neko: Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

[der Raab]

Hi everybody! I need to parse quite huge XML files (around 360MB) ran into this terminal error: Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

It took me already quite some time to actually make that run at all, since it's not possible to load the whole file at once (256MB size limit for strings). So I use a self written FileInput bytes buffer, looking for opening tags and parse every opening tag as a single Xml using Xml.parse(). Then I append the nodes recursively...

The algorithm seems to work correctly since I can run through smaller files and get the correct structure from the Xml but the first test loading and parsing on of the real files threw this error (see above) after parsing more than 3.630.000 opening Xml tags.

So I think I might simply making a general mistake an should simply use a different technique or how do you process such huge XML files?

Kind regards...

[der Raab]

I also used 

var maximumHeapSizeBytes : Int = 1024 * 1024 * 1024 * 10; // 10 GB
Sys.putEnv( "GC_MAXIMUM_HEAP_SIZE", "" + maximumHeapSizeBytes );

to increase the the heap size - bit still the same...

Is there a way to output memory usage in the terminal? :/

[Simon Krajewski]

Our XML parser isn't really suited for large XMLs because it requires
loading everything into memory. I haven't tried it myself but haxelib
has http://lib.haxe.org/p/girsu which might be a better approach for
this case.

Simon

[der Raab]

Thanks for the hint to that project! I'll try that out! Neko seems to hit it's limits round about 8 GB memory and 5,9 GB compressed memory... But I simply have no real clue what that even means... :)

[der Raab]

For now it helped to invoke Gc.run(true);
After every iteration. Took 30 Minutes to load and parse the full XML in memory but since I'm doing something similar like grisu I think I might switch.

On Tuesday, September 27, 2016 at 12:49:18 AM UTC+2, der Raab wrote: