High availability of MaxScale on Kubernetes

[Cristi Cioriia]

Hi guys,

I have a question about deploying a highly available Maxscale in a Kubernetes environment.

In the Maxscale documentation (https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-20-limitations-and-known-issues-within-mariadb-maxscale/) it says that: "A server can only be monitored by one monitor. If multiple monitors monitor the same server, the state of the server is non-deterministic." Does this limitation apply to the Galera monitor too and does that mean that there shouldn't be two Maxscale servers running at the same time and monitoring the same servers?

Greetings,
Cristi

[Markus Mäkelä]

Hi,

The limitation on multiple monitors monitoring the same servers only applies to the same MaxScale instance. It is perfectly safe to have separate MaxScale instances monitor the same cluster.

Markus

--
You received this message because you are subscribed to the Google Groups "MaxScale" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxscale+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Markus Mäkelä, Software Engineer
MariaDB Corporation
t: +358 40 7740484 | Skype: markus.j.makela 

[Cristi Cioriia]

Thanks for your quick reply Markus, I'd have one more question regarding this: does the combination between the readwritesplit router module and the Galera monitor module configured with set_priority option and priority for servers would ensure that the write requests end up being served by the same Galera node even if one of the Maxscale nodes dies and the other takes over?

Our use case is as follows: we have one Maxscale instance that forwards the write queries to the same Galera node, e.g. galera-1 and we achieved that by using the readwritesplit router module. What we'd like to achieve now is that if we have a setup with two Maxscale instances running at the same time and the first instance dies, then the second one would take over and continue to send the write requests to the same galera-1 node. I noticed that the Galera monitor (https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-14/maxscale-galera-monitor/) allows choosing the server to be used as master with the set_priority and priority options and thought that configuring both the maxscale instances with set_priority=true and the same priority for the servers on both Maxscale instances would help us achieve our goals. Is this correct?

Thanks and regards,
Cristi

[Markus Mäkelä]

Hi,

Yes, using the priority system is one option but that is usually used when geographic location of the server dictates the order in which MaxScale should write to. The normal Galera monitoring algorithm already guarantees that multiple MaxScale instances pick the same server as the master. The algorithm chooses the node with the lowest value in wsrep_local_index as the master which guarantees that all MaxScale instances will use the same server for writes.

Markus

[Cristi Cioriia]

Hi,

Ah, ok, that's even better. So that means that even if both Maxscale servers are up and running and receive requests alternatively they will choose the same Galera cluster as master? So if request 1 goes to Maxscale 1 and is routed to galera-1 and then request-2 goes to Maxscale-2 it will end up still be executed on galera-1? Is the wsrep_local_index value going to be seen the same in both Maxscale instances so that they both route requests to galera-1? Can't it happen that they see different values at slightly different moments in time and therefore route to different servers?

Cristi

Greetings,
Cristi

[Markus Mäkelä]

Hi,

The Galera cluster itself has to acknowledge the failure of a server before the value of wsrep_local_index can change. By default, MaxScale uses the lowest value but it is also possible to configure it to use the node with only the value 0. This guarantees that the cluster must reorganize itself before MaxScale changes the node where it sends writes. You can enable this option in MaxScale 2.1 by adding root_node_as_master=true to the monitor definition.

Whether sending writes to a different node in a Galera cluster is a problem depends on your application. If it can handle a possible deadlock error on commit or if it uses autocommit, then it should not cause problems.

Markus

[Cristi Cioriia]

Thanks, Markus, this was very helpful.

[Cristi Cioriia]

Hi,

One more question though: I noticed that the root_node_as_master appeared in version 2.1 and we use version 2.0.4. The issue: https://jira.mariadb.org/browse/MXS-1335 says that this option should "hould only be enabled if multiple MaxScale instances use the same set of Galera nodes". Does this mean that we can't use Maxscale 2.0.4? What would be risks we expose to if we use Maxscale 2.0.4?

Greetings,
Cristi

[Markus Mäkelä]

Hi,

The only risk in using 2.0 is the fact that there's a possibility of sending writes to two different nodes. If two MaxScale instances are running at the same time and both of them are receiving traffic then when one of them loses the connection to the root node, it will use another node for writes.

If you only use one MaxScale instance at a time, there is no problem even with 2.0. As I mentioned previously, whether write conflicts are a problem depends largely on the client application.

Markus