Ran out of memory due to huge data set Indexing at server start time

[ankur bhargava]

I am new to CQEngine and tried to implement in my existing project  which a large Auto Portal based in India. I am having huge data set currently in Database table for user review which stores some text against each Car Model and each model can have multiple reviews , as of now this table having around 40000 rows and a column in this table is having type TEXT. 

And till now from Live Production DB maximum length of data in this particular column is 58233 characters. Corresponding this table I am having POJO class and this class is having a String type attribute in it. 

I want to have huge data set to get indexed. Initially I tried to create ConcurrentIndexedCollection in heap memory  at the server(Tomcat 7) start time and it ran out of memory. Then I tried to persist index in file system. Still I am running out of memory at the server start. 

So please suggest approach to handle this. As of now I am giving try in my local environment with the production dump. 

I am just amazed by the fact of Sql like query on Collection. And if it would work then will implement each module of my project. 

[Niall]

Hi Ankur,

If you are using DiskPersistence, and only using the Disk indexes (this means you should not use any indexes except the disk ones), then effectively CQEngine should not use any heap memory or RAM at all.

Are you perhaps trying to load the entire collection into a List at startup, and then adding that List to the IndexedCollection? If so the List could be the problem, so you could load objects into the IndexedCollection one at a time, or else try to load them in batches.

What you are trying to do should work - let me know how you get on!

Best regards,

Niall