Best practices for load balancing SolrNet with SolrCloud

[Martin Bayly]

Hi

Just getting started with both SolrCloud and SolrNet so sorry if this is a dumb question or too vague or just it depends....

I've read through the SolrCloud reference manual once but it probably hasn't all sunk in fully yet so the answers may all be in there somewhere.

But does anyone have any thoughts on best practices for using SolrNet with SolrCloud.

In particular I'm looking for insights into what are the best approaches for ensuring that additions to the index and queries are effectively load balanced.

My expectation is we'll have a single Collection with multiple shards and replicas distributed over a number of nodes.

Our web application will be scaled horizontally so we'll have multiple web nodes querying SolrCloud.

We may or may not have multiple indexer nodes.

From what I understand SolrCloud will handle sending index update operations to the appropriate shard regardless of which node I connect to from the client?

And SolrCloud will also handle distributing requests to appropriate replicas regardless of which node I connect to from the client? 

Under Distributed Requests (https://cwiki.apache.org/confluence/display/solr/Distributed+Requests) it discussed being able to pass a list of servers (shards) as part of the query for load balancing purposes.

But that seems to require the client to have lots of information about which nodes are up and active?

I remember seeing a reference to a Java client that has smarts built in to talk to the Zookeeper ensemble so that it can know which nodes to direct requests to.  But I don't think that is available with SolrNet yet.

So is the only solution with SolrNet to add some other form of network load balancer in front of SolrCloud? Or is that often unnecessary and we can rely on SolrCloud's internal load balancing.

I guess the biggest issue would be that if we are connecting to a single known node, and that node goes down we are hooped.

Cheers

Martin

[Mauricio Scheffer]

Hi Martin,

The question of integrating SolrNet with SolrCloud has come up before: https://groups.google.com/d/msg/solrnet/-IJ-V_uRDJ0/qkmsoXUmneYJ . 

The current status is still the same. Eric Hauser's .NET ZooKeeper client ( https://github.com/ewhauser/zookeeper/tree/trunk/src/dotnet ) seems to be well-maintained and there's a NuGet package as well: http://www.nuget.org/packages/ZooKeeperNet/ .

So the way to go seems to be to port CloudSolrServer ( https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrServer.java ), probably as a SolrNet SolrConnection decorator, using ZooKeeperNet to talk to ZooKeeper.

Cheers

--

Mauricio

--
You received this message because you are subscribed to the Google Groups "SolrNet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrnet+u...@googlegroups.com.
To post to this group, send email to sol...@googlegroups.com.
Visit this group at http://groups.google.com/group/solrnet.
For more options, visit https://groups.google.com/groups/opt_out.

[Martin Bayly]

Ah, ok, I did see that post previously - that's probably where I saw the reference to the smart Java client - but it didn't sink in that it was actually answering my question - probably because this is all still so new to me.

Will review that post again and check out CloudSolrServer - thanks for the links.

Cheers

Martin

[[mRg]]

It's good that the .NET Zookeeper client is still being maintained, it looks like a decent project to take a dependency on. The CloudSolrServer is something I am interested in so I may give the port of it a go in the coming weeks, if anyone else is interested.

I also noticed this : http://lucene.apache.org/solr/4_2_1/solr-solrj/org/apache/solr/client/solrj/impl/LBHttpSolrServer.html which is a similar useful version of SolrServer to have in a cloud setup (as it notes though, not useful in a master/slave scenario) so this might be useful to port as well.

Opinions ?

Stephen

[Mauricio Scheffer]

Yep, I've been considering implementing something like LBHttpSolrServer for a few years now ( https://github.com/mausch/SolrNet/issues/96 ), but I haven't needed it personally yet.

It seems that the equivalent of LBHttpSolrServer doesn't need any additional dependencies so it could go into the core of SolrNet.

While the ZooKeeper-aware port could go into a separate project/package (within the same repository, of course), so that the dependencies are cleanly separated.

Cheers

--

Mauricio