Anyone have any idea of how many queries/minute Tomcat6/SOLR4 can handle on nice hardware?

[minorgod]

Greetings. I know asking questions about request limits usually elicits a bunch of "it depends" answers, but I'm not looking for an authoritative answer here, just any anecdotal info anyone might have based on their real-world experiences.  I've been running stress tests on a new Tomcat6/Solr4 server and I can't seem to cause it to slow down or crash up to the limits of what solrmeter can throw at it. The hardware is a modern quad core server processor running CentOS5 (I think the processor is an E5645 6-Core 2.4 GHz) with 32GB of RAM and decent storage subsystem.  This hardware is currently acting as a slave mysql db server with most of its 32GB of ram allocated to MySQL, but the MySQL load is very low to non-existent aside from keeping itself in sync with a very busy master server. So, I set up Tomcat6/Solr4 with all default settings, set up my index and a bunch of test queries, then started trying to kill it with solrmeter. I'm connecting to it over a VPN, and I seem to be hitting the limits of my VPN and or solrmeter's memory limits at between 15,000-20,000 queries/minute so I can't test beyond that without adding some more test machines on other networks. With that said, my average query time stays below 1ms with a 50% cache hit rate up to the limits I'm able to test (about 15,000 queries per minute). In other words, Tomcat/Solr don't seem to be having any probs keeping up with the load, it's just my network connection and/or solrmeter that are crapping out. So, I'm wondering if anyone else has any vague ideas of how many requests per minute I can theoreticaly throw at Tomcat6/SOLR4 before one of them will start to show problems, given that I haven't seen any probs yet at 15,000 queries/minute. I wonder if it could keep up with 30,000 requests/minute? When my app goes into production I anticipate a steady load of about 20,000-25,000 requests/minute. Anyone want to comment?

[Eric Pugh]

SolrMeter is one application running on one box, and I don't think it is crazy optimized for load.   I think that at the thousands of queries a minutes point in time, you really need to look at other tools that are better at generating load.   Even if you could generate 15,000 queries a second from SolrMeter according to the UI, I would be dubious that it was actually doing that and reporting it accurately.

If you are at the 15,000 qpm, then I'd say just plan on being able to scale horizontally over a couple of boxes, and you'll be fine.   

Honestly, I am dubious that you've been able to actually generate 15,000 qpm, can you confirm that Solr is saying the same thing?

On Jan 24, 2013, at 4:38 PM, minorgod wrote:

Greetings. I know asking questions about request limits usually elicits a bunch of "it depends" answers, but I'm not looking for an authoritative answer here, just any anecdotal info anyone might have based on their real-world experiences.  I've been running stress tests on a new Tomcat6/Solr4 server and I can't seem to cause it to slow down or crash up to the limits of what solrmeter can throw at it. The hardware is a modern quad core server processor running CentOS5 (I think the processor is an E5645 6-Core 2.4 GHz) with 32GB of RAM and decent storage subsystem.  This hardware is currently acting as a slave mysql db server with most of its 32GB of ram allocated to MySQL, but the MySQL load is very low to non-existent aside from keeping itself in sync with a very busy master server. So, I set up Tomcat6/Solr4 with all default settings, set up my index and a bunch of test queries, then started trying to kill it with solrmeter. I'm connecting to it over a VPN, and I seem to be hitting the limits of my VPN and or solrmeter's memory limits at between 15,000-20,000 queries/minute so I can't test beyond that without adding some more test machines on other networks. With that said, my average query time stays below 1ms with a 50% cache hit rate up to the limits I'm able to test (about 15,000 queries per minute). In other words, Tomcat/Solr don't seem to be having any probs keeping up with the load, it's just my network connection and/or solrmeter that are crapping out. So, I'm wondering if anyone else has any vague ideas of how many requests per minute I can theoreticaly throw at Tomcat6/SOLR4 before one of them will start to show problems, given that I haven't seen any probs yet at 15,000 queries/minute. I wonder if it could keep up with 30,000 requests/minute? When my app goes into production I anticipate a steady load of about 20,000-25,000 requests/minute. Anyone want to comment?

--
 
 

-----------------------------------------------------

Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com

Co-Author: Apache Solr 3 Enterprise Search Server available from http://www.packtpub.com/apache-solr-3-enterprise-search-server/book

This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.