Sudden rush of document cache inserts/evictions

20 views
Skip to first unread message

Neil Prosser

unread,
Jan 28, 2015, 9:19:04 AM1/28/15
to helio...@googlegroups.com
I'm investigating Heliosearch 0.09 as an alternative to Solr (4.5.1) + traditional replication. Our current problem with our Solr is the heap size needed to contain the caches required to make a large index work.

I'm using the Shawn Heisey's suggested G1 settings (https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_First.29_Collector) and JDK 8u40 (it's an early-access build) and we're seeing great results. The pertinent JVM options are:

-XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:MaxGCPauseMillis=200 -XX:+UseLargePages -XX:+AggressiveOpts -Xms4g -Xmx6g (though huge pages aren't enabled on this system at present).

Compared to our current Solr solution and even my previous attempts with Heliosearch we are seeing far fewer stop-the-world pauses (single figures per day) which is fantastic!

However, the pauses we do see seem to be linked to some strange behaviour which I was wondering whether anyone here might be able to shed some light on. Our document cache has a size of 4096 which is working well under normal circumstances but in the minutes immediately preceding we see a sudden increase in inserts (and evictions) from the document cache with no linked rise in requests coming in to the cluster (we're talking ~60k between commits to ~4m).

Is anyone able to think of a reason for all replicas of all nine shards to suddenly start churning through documents without an increase in the number of requests? Our queries are mostly dismax requests with boosting along with queries for documents with specific IDs. Faceting is used reasonably frequently but these document cache surges don't always cause GC pauses.

Please let me know if any more information is needed. Thanks.

Neil Prosser

unread,
Jan 29, 2015, 3:20:40 AM1/29/15
to helio...@googlegroups.com
Sorry everyone. I was befuddled by this for a few days and of course as soon as I'd posted here I got a lead. It turns out that this issue is down to someone very kindly deep-paging their search results (4M+ items into a boosted query). It all makes perfect sense now and we'll be turning off the ability for people to attempt to page that far in to the documents!
Reply all
Reply to author
Forward
0 new messages