I'm investigating Heliosearch 0.09 as an alternative to Solr (4.5.1) + traditional replication. Our current problem with our Solr is the heap size needed to contain the caches required to make a large index work.
-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:MaxGCPauseMillis=200 -XX:+UseLargePages -XX:+AggressiveOpts -Xms4g -Xmx6g (though huge pages aren't enabled on this system at present).
Compared to our current Solr solution and even my previous attempts with Heliosearch we are seeing far fewer stop-the-world pauses (single figures per day) which is fantastic!
However, the pauses we do see seem to be linked to some strange behaviour which I was wondering whether anyone here might be able to shed some light on. Our document cache has a size of 4096 which is working well under normal circumstances but in the minutes immediately preceding we see a sudden increase in inserts (and evictions) from the document cache with no linked rise in requests coming in to the cluster (we're talking ~60k between commits to ~4m).
Is anyone able to think of a reason for all replicas of all nine shards to suddenly start churning through documents without an increase in the number of requests? Our queries are mostly dismax requests with boosting along with queries for documents with specific IDs. Faceting is used reasonably frequently but these document cache surges don't always cause GC pauses.
Please let me know if any more information is needed. Thanks.