GC overhead limit exceeded

[Michael Ruepp]

Hi,

I try programatically to create term/docvectors on a quite large corpus. Now I get the error below when creating doc vector store.

Any clue anyone? Will it be enough to increase vm memory? -Xmx and Xms?

Thanks

There are 4051896 terms (and 4396040 docs).

Training term vectors for field contents

Processed 0 terms ... Processed 1000 terms ... Processed 2000 terms ... Processed 3000 terms ... Processed 4000 terms ... Processed 5000 terms ... Processed 6000 terms ... Processed 7000 terms ... Processed 8000 terms ... Processed 9000 terms ... Processed 10000 terms ... Processed 20000 terms ... Processed 30000 terms ... Processed 40000 terms ... Processed 50000 terms ... Processed 60000 terms ... Processed 70000 terms ... Processed 80000 terms ... Processed 90000 terms ... Processed 100000 terms ... Processed 110000 terms ... Processed 120000 terms ... Processed 130000 terms ... Processed 140000 terms ... Processed 150000 terms ... Processed 160000 terms ... Processed 170000 terms ... Processed 180000 terms ... Processed 190000 terms ... Processed 200000 terms ... Processed 210000 terms ... Processed 220000 terms ... Processed 230000 terms ... Processed 240000 terms ... Processed 250000 terms ... Processed 260000 terms ... Processed 270000 terms ... Processed 280000 terms ... Processed 290000 terms ... Processed 300000 terms ... Processed 310000 terms ... Processed 320000 terms ... Processed 330000 terms ... Processed 340000 terms ... Processed 350000 terms ... Processed 360000 terms ... Processed 370000 terms ... Processed 380000 terms ... Processed 390000 terms ... Processed 400000 terms ... Processed 410000 terms ... Processed 420000 terms ... Processed 430000 terms ... Processed 440000 terms ... Processed 450000 terms ... Processed 460000 terms ... Processed 470000 terms ... Processed 480000 terms ... Processed 490000 terms ... Processed 500000 terms ... Processed 510000 terms ... Processed 520000 terms ... Processed 530000 terms ... Processed 540000 terms ... Processed 550000 terms ... Processed 560000 terms ... Processed 570000 terms ... Processed 580000 terms ... Processed 590000 terms ... Processed 600000 terms ... Processed 610000 terms ... Processed 620000 terms ... Processed 630000 terms ... Processed 640000 terms ... Processed 650000 terms ... Processed 660000 terms ... Processed 670000 terms ... Processed 680000 terms ... Processed 690000 terms ... Processed 700000 terms ... Processed 710000 terms ... Processed 720000 terms ... Processed 730000 terms ... Processed 740000 terms ... Processed 750000 terms ... Processed 760000 terms ... Processed 770000 terms ... Processed 780000 terms ... Processed 790000 terms ... Processed 800000 terms ... Processed 810000 terms ... Processed 820000 terms ... Processed 830000 terms ... Processed 840000 terms ... Processed 850000 terms ... Processed 860000 terms ... Processed 870000 terms ... Processed 880000 terms ... Processed 890000 terms ... Processed 900000 terms ... Processed 910000 terms ... Processed 920000 terms ... Processed 930000 terms ... Processed 940000 terms ... Processed 950000 terms ... Processed 960000 terms ... Processed 970000 terms ... Processed 980000 terms ... Processed 990000 terms ... Processed 1000000 terms ... Processed 1010000 terms ... Processed 1020000 terms ... Processed 1030000 terms ... Processed 1040000 terms ... Processed 1050000 terms ... Processed 1060000 terms ... Processed 1070000 terms ... Processed 1080000 terms ... Processed 1090000 terms ... Processed 1100000 terms ... Processed 1110000 terms ... Processed 1120000 terms ... Processed 1130000 terms ... Processed 1140000 terms ... Processed 1150000 terms ... Processed 1160000 terms ... Processed 1170000 terms ... Processed 1180000 terms ... Processed 1190000 terms ... Processed 1200000 terms ... Processed 1210000 terms ... Processed 1220000 terms ... Processed 1230000 terms ... Processed 1240000 terms ... Processed 1250000 terms ... Processed 1260000 terms ... Processed 1270000 terms ... Processed 1280000 terms ... Processed 1290000 terms ... Processed 1300000 terms ... Processed 1310000 terms ... Processed 1320000 terms ... Processed 1330000 terms ... Processed 1340000 terms ... Processed 1350000 terms ... Processed 1360000 terms ... Processed 1370000 terms ... Processed 1380000 terms ... Processed 1390000 terms ... Processed 1400000 terms ... Processed 1410000 terms ... Processed 1420000 terms ... Processed 1430000 terms ... Processed 1440000 terms ... Processed 1450000 terms ... Processed 1460000 terms ... Processed 1470000 terms ... Processed 1480000 terms ... Processed 1490000 terms ... Processed 1500000 terms ... Processed 1510000 terms ... Processed 1520000 terms ... Processed 1530000 terms ... Processed 1540000 terms ... Processed 1550000 terms ... Processed 1560000 terms ... Processed 1570000 terms ... Processed 1580000 terms ... Processed 1590000 terms ... Processed 1600000 terms ... Processed 1610000 terms ... Processed 1620000 terms ... Processed 1630000 terms ... Processed 1640000 terms ... Processed 1650000 terms ... Processed 1660000 terms ... Processed 1670000 terms ... Processed 1680000 terms ... Processed 1690000 terms ... Processed 1700000 terms ... Processed 1710000 terms ... Processed 1720000 terms ... Processed 1730000 terms ... Processed 1740000 terms ... Processed 1750000 terms ... Processed 1760000 terms ... Processed 1770000 terms ... Processed 1780000 terms ... Processed 1790000 terms ... Processed 1800000 terms ... Processed 1810000 terms ... Processed 1820000 terms ... Processed 1830000 terms ... Processed 1840000 terms ... Processed 1850000 terms ... Processed 1860000 terms ... Processed 1870000 terms ... Processed 1880000 terms ... Processed 1890000 terms ... Processed 1900000 terms ... Processed 1910000 terms ... Processed 1920000 terms ... Processed 1930000 terms ... Processed 1940000 terms ... Processed 1950000 terms ... Processed 1960000 terms ... Processed 1970000 terms ... Processed 1980000 terms ... Processed 1990000 terms ... Processed 2000000 terms ... Processed 2010000 terms ... Processed 2020000 terms ... Processed 2030000 terms ... Processed 2040000 terms ... Processed 2050000 terms ... Processed 2060000 terms ... Processed 2070000 terms ... Processed 2080000 terms ... Processed 2090000 terms ... Processed 2100000 terms ... Processed 2110000 terms ... Processed 2120000 terms ... Processed 2130000 terms ... Processed 2140000 terms ... Processed 2150000 terms ... Processed 2160000 terms ... Processed 2170000 terms ... Processed 2180000 terms ... Processed 2190000 terms ... Processed 2200000 terms ... Processed 2210000 terms ... Processed 2220000 terms ... Processed 2230000 terms ... Processed 2240000 terms ... Processed 2250000 terms ... Processed 2260000 terms ... Processed 2270000 terms ... Processed 2280000 terms ... Processed 2290000 terms ... Processed 2300000 terms ... Processed 2310000 terms ... Processed 2320000 terms ... Processed 2330000 terms ... Processed 2340000 terms ... Processed 2350000 terms ... Processed 2360000 terms ... Processed 2370000 terms ... Processed 2380000 terms ... Processed 2390000 terms ... Processed 2400000 terms ... Processed 2410000 terms ... Processed 2420000 terms ... Processed 2430000 terms ... Processed 2440000 terms ... Processed 2450000 terms ... Processed 2460000 terms ... Processed 2470000 terms ... Processed 2480000 terms ... Processed 2490000 terms ... Processed 2500000 terms ... Processed 2510000 terms ... Processed 2520000 terms ... Processed 2530000 terms ... Processed 2540000 terms ... Processed 2550000 terms ... Processed 2560000 terms ... Processed 2570000 terms ... Processed 2580000 terms ... Processed 2590000 terms ... Processed 2600000 terms ... Processed 2610000 terms ... Processed 2620000 terms ... Processed 2630000 terms ... Processed 2640000 terms ... Processed 2650000 terms ... Processed 2660000 terms ... Processed 2670000 terms ... Processed 2680000 terms ... Processed 2690000 terms ... Processed 2700000 terms ... Processed 2710000 terms ... Processed 2720000 terms ... Processed 2730000 terms ... Processed 2740000 terms ... Processed 2750000 terms ... Processed 2760000 terms ... Processed 2770000 terms ... Processed 2780000 terms ... Processed 2790000 terms ... Processed 2800000 terms ... Processed 2810000 terms ... Processed 2820000 terms ... Processed 2830000 terms ... Processed 2840000 terms ... Processed 2850000 terms ... Processed 2860000 terms ... Processed 2870000 terms ... Processed 2880000 terms ... Processed 2890000 terms ... Processed 2900000 terms ... Processed 2910000 terms ... Processed 2920000 terms ... Processed 2930000 terms ... Processed 2940000 terms ... Processed 2950000 terms ... Processed 2960000 terms ... Processed 2970000 terms ... Processed 2980000 terms ... Processed 2990000 terms ... Processed 3000000 terms ... Processed 3010000 terms ... Processed 3020000 terms ... Processed 3030000 terms ... Processed 3040000 terms ... Processed 3050000 terms ... Processed 3060000 terms ... Processed 3070000 terms ... Processed 3080000 terms ... Processed 3090000 terms ... Processed 3100000 terms ... Processed 3110000 terms ... Processed 3120000 terms ... Processed 3130000 terms ... Processed 3140000 terms ... Processed 3150000 terms ... Processed 3160000 terms ... Processed 3170000 terms ... Processed 3180000 terms ... Processed 3190000 terms ... Processed 3200000 terms ... Processed 3210000 terms ... Processed 3220000 terms ... Processed 3230000 terms ... Processed 3240000 terms ... Processed 3250000 terms ... Processed 3260000 terms ... Processed 3270000 terms ... Processed 3280000 terms ... Processed 3290000 terms ... Processed 3300000 terms ... Processed 3310000 terms ... Processed 3320000 terms ... Processed 3330000 terms ... Processed 3340000 terms ... Processed 3350000 terms ... Processed 3360000 terms ... Processed 3370000 terms ... Processed 3380000 terms ... Processed 3390000 terms ... Processed 3400000 terms ... Processed 3410000 terms ... Processed 3420000 terms ... Processed 3430000 terms ... Processed 3440000 terms ... Processed 3450000 terms ... Processed 3460000 terms ... Processed 3470000 terms ... Processed 3480000 terms ... Processed 3490000 terms ... Processed 3500000 terms ... Processed 3510000 terms ... Processed 3520000 terms ... Processed 3530000 terms ... Processed 3540000 terms ... Processed 3550000 terms ... Processed 3560000 terms ... Processed 3570000 terms ... Processed 3580000 terms ... Processed 3590000 terms ... Processed 3600000 terms ... Processed 3610000 terms ... Processed 3620000 terms ... Processed 3630000 terms ... Processed 3640000 terms ... Processed 3650000 terms ... Processed 3660000 terms ... Processed 3670000 terms ... Processed 3680000 terms ... Processed 3690000 terms ... Processed 3700000 terms ... Processed 3710000 terms ... Processed 3720000 terms ... Processed 3730000 terms ... Processed 3740000 terms ... Processed 3750000 terms ... Processed 3760000 terms ... Processed 3770000 terms ... Processed 3780000 terms ... Processed 3790000 terms ... Processed 3800000 terms ... Processed 3810000 terms ... Processed 3820000 terms ... Processed 3830000 terms ... Processed 3840000 terms ... Processed 3850000 terms ... Processed 3860000 terms ... Processed 3870000 terms ... Processed 3880000 terms ... Processed 3890000 terms ... Processed 3900000 terms ... Processed 3910000 terms ... Processed 3920000 terms ... Processed 3930000 terms ... Processed 3940000 terms ... Processed 3950000 terms ... Processed 3960000 terms ... Processed 3970000 terms ... Processed 3980000 terms ... Processed 3990000 terms ... Processed 4000000 terms ... Processed 4010000 terms ... Processed 4020000 terms ... Processed 4030000 terms ... Processed 4040000 terms ... Processed 4050000 terms ... 

Created 4051896 term vectors.

Initializing new document vector store ... 

Building document vectors ... Processed 0 terms ... Processed 1000 terms ... Processed 2000 terms ... Processed 3000 terms ... Processed 4000 terms ... Processed 5000 terms ... Processed 6000 terms ... Processed 7000 terms ... Processed 8000 terms ... Processed 9000 terms ... Processed 10000 terms ... Exception in thread "TimerQueue" java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.concurrent.locks.ReentrantLock$Sync.newCondition(ReentrantLock.java:168)

at java.util.concurrent.locks.ReentrantLock.newCondition(ReentrantLock.java:500)

at javax.swing.TimerQueue.run(TimerQueue.java:192)

at java.lang.Thread.run(Thread.java:745)

[Dominic Widdows]

Here are a couple of things you can try:

- Set "-docindexing incremental" in your command line parameters. Or "-docindexing none" if you don't need document vectors at all.

- Since you have about as many document as terms, and your process got through the term indexing part, you should be able to get though with the current indexing strategy if you can set -Xmx to about 2.5 times your current allocation, that should work.

Best wishes,

Dominic

--
You received this message because you are subscribed to the Google Groups "Semantic Vectors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semanticvecto...@googlegroups.com.
To post to this group, send email to semanti...@googlegroups.com.
Visit this group at http://groups.google.com/group/semanticvectors.
For more options, visit https://groups.google.com/d/optout.