About cluster status while adding a new node

[Sébastien Han]

Hello,

I would like more precision about the 

When a node is about join the cluster, it sends a request throught the gcomm url to a current member of the cluster. The chosen node status will changed from JOINED to DONOR and apparently a FLUSH TABLES WITH READ LOCK is performed by the SST. I'm currently using the RSYNC method which seems to be the fastest one.

My question is: 

-Does the full cluster in a READ-ONLY status or only the DONOR node? I ask this question because your wiki says that **only** the donor is locked and the Percona XtraDB docs says that the **entire** cluster is locked...

-What happen to the incoming requests? I use HAProxy in from of every nodes. Are they manage like writeset caching during state transfer? With those options: gcs.recv_q_hard_limit, gcs.recv_q_soft_limit, gcs.max_throttle.

Thank you in advance :)

Cheers!

[Henrik Ingo]

On Wed, Aug 8, 2012 at 12:23 AM, Sébastien Han <han.se...@gmail.com> wrote:
> Hello,
>
> I would like more precision about the
> When a node is about join the cluster, it sends a request throught the gcomm
> url to a current member of the cluster. The chosen node status will changed
> from JOINED to DONOR and apparently a FLUSH TABLES WITH READ LOCK is
> performed by the SST. I'm currently using the RSYNC method which seems to be
> the fastest one.
>
> My question is:
>
> -Does the full cluster in a READ-ONLY status or only the DONOR node? I ask
> this question because your wiki says that **only** the donor is locked and
> the Percona XtraDB docs says that the **entire** cluster is locked...

Of course only the donor is locked (and this is only in the case of
rsync method, for instance with the xtrabackup sst method the donor is
not locked).

> -What happen to the incoming requests? I use HAProxy in from of every nodes.
> Are they manage like writeset caching during state transfer? With those
> options: gcs.recv_q_hard_limit, gcs.recv_q_soft_limit, gcs.max_throttle.

Currently an incoming query to the donor, if you are using rsync sst,
will block and wait for its turn (or until it exceeds a lock timeout
and fails).
In the next release a new configuration option
wsrep_sst_donor_rejects_queries will be introduced. When set to TRUE,
the donor will immediately reject an incoming query with "Unknown
error". The benefit of this is that HAProxy can then immediately
discover such failure and re-route to another node.

See https://bugs.launchpad.net/codership-mysql/+bug/1002714 for more info.

henrik
--
henri...@avoinelama.fi
+358-40-8211286 skype: henrik.ingo irc: hingo
www.openlife.cc

My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559

[Sébastien Han]

Thank you for the clarification :)

I'll have a look to xtrabackup thought.

Cheers!

[Laurent MINOST]

Hi Henrik,

I would like to get more informations about what you stated in your previous answer please :

 In the next release a new configuration option  
 wsrep_sst_donor_rejects_queries will be introduced. When set to TRUE, 
 the donor will immediately reject an incoming query with "Unknown 
 error". The benefit of this is that HAProxy can then immediately 
 discover such failure and re-route to another node. 

Could you please explain how you configure HAProxy to re-route the query to another node please ? IMO when a query has reached a node after passing HAproxy, if it received a Unknown error, then there is no configuration options from HAProxy view to make it 'retry' the query on another node of the cluster ?? Maybe I'm wrong and not knowing of such behavior or maybe you were saying this by thinking of such retries at the application level ?

Thanks by advance for the clarification.

Regards,

Laurent

[Davide Bozzelli]

On Fri, Aug 10, 2012 at 12:25 PM, Laurent MINOST <lol...@gmail.com> wrote:

Hi Henrik,

I would like to get more informations about what you stated in your previous answer please :

 In the next release a new configuration option  
 wsrep_sst_donor_rejects_queries will be introduced. When set to TRUE, 
 the donor will immediately reject an incoming query with "Unknown 
 error". The benefit of this is that HAProxy can then immediately 
 discover such failure and re-route to another node. 

Could you please explain how you configure HAProxy to re-route the query to another node please ? IMO when a query has reached a node after passing HAproxy, if it received a Unknown error, then there is no configuration options from HAProxy view to make it 'retry' the query on another node of the cluster ?? Maybe I'm wrong and not knowing of such behavior or maybe you were saying this by thinking of such retries at the application level ?

http://www.mysqlperformanceblog.com/2012/06/20/percona-xtradb-cluster-reference-architecture-with-haproxy/

Basically each node has a xinetd service which expose the status of the node.

Haproxy simply checks this status and, based on the return value, decide where to route.

Have fun.

--
Got problems with Windows? - ReBooT
Got problems with Linux? - Be RooT

[Henrik Ingo]

You are correct. A TCP/IP proxy like HAProxy is of course ignorant of
any MySQL errors. I was assuming you use some kind of check-script,
like the one Percona has developed:
http://www.percona.com/doc/percona-xtradb-cluster/haproxy.html

Personally, I prefer to use load-balancing directly from the JDBC and
PHP_ND drivers. Here the load-balancer will catch the error (either a
TCP/IP error or MySQL Unknown error) immediately as they happen and
then redirect to another node, without the need for any polling
script.

henrik