Automate configuration of maxscale server

[Erwin Daul]

Hello everyone, hope you all had a good year, despite the Corona thingy,

 

We’re thinking about adding a maxscale server (possibly multiple in the future) in our infrastructure.

We have a lot of mariadb servers (all configured in pairs : master and slave).

We have automated the configuration of mariadb servers with Ansible, and we would like to do the same with MaxScale.

 

Thing is… there’s no reload option for maxscale (I’m using maxscale 2.4.14)? If I add a new pair of mariadb servers, and configure them (along with Monitor, Services, and Listeners) in maxscale.cnf or etc/maxscale.cnf.d/,

I’m forced to do a restart? Isn’t there an alternative?

 

I guess I can use commands like “maxctrl create XXXXX”, “maxctrl alter XXXXX”, “maxctrl destroy XXXXX” to change the configuration without shutting down maxscale, but this is hardly automatable.

 

Also, there’s no way to alter a listener? I’m forced to destroy and then create one?

 

 

Regards,

 

--

Erwin

[Markus Mäkelä]

Hi,

Currently runtime changes to MaxScale are done via maxctrl, similar to how runtime changes are done in MariaDB via the SQL interface. If you could expand on your use-case a little bit, perhaps we can find a solution for it with the tools that are currently available or come up with a new feature that would solve it in the future. If you want the reload functionality to be added as a feature, you can file a new feature request on the MariaDB Jira under the MaxScale project.

Alteration of listeners is going to be in MaxScale 2.6 as implementing it required some fundamental changes in the listener configuration implementation. You should be able to configure most everything without having to destroy and recreate the listener. The current limitations for what cannot be changed are found here (an end-user facing document should be written for this)

Markus

--
You received this message because you are subscribed to the Google Groups "MaxScale" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxscale+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/CANDQcQ2BSXxbzZYUgi5mfVwX%2B28gZ0Qewve7uHxqsLaKnMby8g%40mail.gmail.com.

-- 
Markus Mäkelä, Senior Software Engineer
MariaDB Corporation

[Erwin Daul]

Hello Markus,

Thank you for answering.

As we frequently deploy new pairs of MariaDB servers, I think the ideal solution would be to use a template for the maxscale configuration of each new pair.

So each template will for example be filled with 2 servers, 1 monitor, 2 services (1 readwritsplit, 1 readconnroute), 2 listeners (1 Read-Write-Service, 1 Read-Only-Service).
The modified template will then be transferred to the maxscale configuration directory (in /etc/maxscale.cnf.d/ if I'm right).
Then, I would just need a way to tell maxscale "ok, please re-read your configuration files and add/modify any new server/monitor/service/listener/etc you see."

MaxScale restarts seems fast, but since we sometimes avec long-running transactions, I prefer to reload rather than restart to avoid breaking them.

I guess I'm used to do service reloads for that (like with nginx), but sending a command to tell MaxScale to re-read it's configuration files is also ok.

The whole idea is to tell MaxScale "Okay, I want you configured this way, apply whatever you see new or changed, but don't touch anything else".

That way, when we deploy new pairs of MariaDB servers, they will directly be available through MaxScale, with 0 human intervention.

What do you think of this? Is there already a way to do that (and is not in the docs)?

I'll proceed to do a feature request.

--

Erwin

[Markus Mäkelä]

Hi,

Thanks for explaining your use case, this clarifies what you're attempting to do with MaxScale. Currently there's no way to have MaxScale re-read the configuration files which means the only way to achieve what you want is to use either MaxCtrl or the REST API directly.

Markus

To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/6a6ddf26-c59e-4c0d-b5f1-9ee788817b83n%40googlegroups.com.

[Erwin Daul]

Alright, thanks for your input, I guess the maxctrl command is enough for now, I'm just wondering how mass reconfiguration of servers/services will be when we have more servers...

Could you tell me if you could consider a feature like reloading in a future version? Maybe in the next few months?
Or is it something not planned at all currently?

Thanks,

--

Erwin

[Markus Mäkelä]

Hi,

I think people haven't actively requested in the recent years now that the runtime modification of MaxScale is doable with MaxCtrl.

As for whether it'll be considered for a new feature, I can't really say. We need to evaluate whether this is a feature that's truly needed and that it meshes with existing features. Having multiple ways of doing the same thing is not efficient nor is it maintainable.

New major features won't be added to any GA releases of MaxScale and this would count as one. I wouldn't expect to see a new MaxScale major release in the next few months.

Markus

To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/e7bca091-ad5f-4692-b9e2-b9a376ede15an%40googlegroups.com.

[Erwin Daul]

I can understand.

I've tried creating everything with maxctrl. Everything seems fine, until I create the listeners.
Here's the command I'm using :

$ maxctrl create listener mariadb-10-5-02_Read-Write-Service mariadb-10-5-02_Read-Write-Listener 4306 --interface=0.0.0.0 --authenticator=MariaDBAuth --authenticator-options=lower_case_table_names=true

multiple things :

1) I can't seem to find any command to show the current configuration of the listeners, I can only create and destroy them?

This forces me to check /var/lib/maxscale/maxscale.cnf.d/mariadb-10-5-02_Read-Write-Listener.cnf, here's the content of the file :

[mariadb-10-5-02_Read-Write-Listener]

type=listener

address=0.0.0.0

authenticator=MariaDBAuth

authenticator_options=

port=4306

protocol=mariadbclient

service=mariadb-10-5-02_Read-Write-Service

ssl=false

ssl_cert_verify_depth=9

ssl_verify_peer_certificate=false

ssl_version=MAX

type=listener

2) Notice the "type=listener" appears twice? This doesn't seem to bother maxscale however

3) "authenticator_options" is empty, I tried multiple things to make "lower_case_table_names=true" appear, but nothings ever happens, what am I doing wrong?

I can fill the void, so that the line becomes :

authenticator_options=lower_case_table_names=true

And then restart the service. This works, but then again... I'm restarting the service, and I need to avoid that, I'm obligated to use this options or else some of our tools don't work...

I use MaxScale in version 2.4.14

[Seva Kobylin]

Another case for this feature is ability to use file-based systems like ansible. It’s much easier to use templating with reload then commands.

31 дек. 2020 г., в 15:15, Markus Mäkelä <markus...@mariadb.com> написал(а):



To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/74b56274-fb3c-db62-1cdb-6541b9aaa062%40mariadb.com.

[Markus Mäkelä]

Hi,

2) Notice the "type=listener" appears twice? This doesn't seem to bother maxscale however

That looks like it might be a double addition of the type parameter at the time the listener is serialized. Based on the fact that it works, it's most likely ignored by the configuration parsing code.

I'll look into fixing this even if it works correctly now. It looks like a thing that's bound to cause problems in the future.

3) "authenticator_options" is empty, I tried multiple things to make "lower_case_table_names=true" appear, but nothings ever happens, what am I doing wrong?

I can fill the void, so that the line becomes :

authenticator_options=lower_case_table_names=true

And then restart the service. This works, but then again... I'm restarting the service, and I need to avoid that, I'm obligated to use this options or else some of our tools don't work...

This does seem like it's a bug and I took a look at the 2.4 maxctrl code and it does indeed seem to be a bug in it. I've filed MXS-3360 for this and it should be fixed in the next releases of MaxScale.

Thanks for reporting these problems!

Markus

[Dylan Northrup]

I would like to +1 this use-case.  We use Ansible for our application configuration.  Being able to non-disruptively make configuration changes by updating config files and triggering a reload has been our go-to method.  Having a known config to converge to on all systems, in practice, tends to be more stable than "apply changes in order on all systems and they'll eventually wind up at the same spot", which is what relying solely on MaxCtrl to make changes gets you.  You either need to take downtime for your service, or you will eventually end up with configuration drift.

I appreciate the sentiment that 'Having multiple ways of doing the same thing is not efficient nor is it maintainable'.  I'd be happy with being able to point MaxCtrl (or some other utility that uses MaxCtrl's code paths as a communication method with the cluster) at a config file and have it tell me what commands need to be run/changes need to be made to converge the cluster to that state.  Then, being able to take that output, run it through MaxCtrl (iterating if need be) to end up with a converged configuration.

The ultimate goal for me is to have a way to verify that what is specified in my source control as a configuration is what is running on the system (and update the system as necessary to make that a reality, if necessary).

To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/9E2B227C-1FAE-4441-BB2B-AF1310E39ED5%40gmail.com.

--

Dylan Northrup

Tock Site Reliability Engineer

dy...@tockhq.com

[Wagner Bianchi]

Not sure If I understood what you mean completely as if you have an Ansible playbook/role for setting up the configurations of your maxscale fleet of instances, you could pretty sure keep them all with the same configurations if that is what you meant to say. Configurations dynamically applied with the maxctrl utility are persisted on files located at the --persistdir, by default, "/var/lib/maxscale/maxscale.cnf.d". In the case MaxScale restarts, those files are read as part of the restart, and you have your configurations with the same state as before.

Regarding configurations drifts, in case something on tour Ansible playbooks are not that IDEMPOTENT for some reason, take a look at the link below:

https://mariadb.com/kb/en/mariadb-maxscale-25-maxctrl/#cluster 

If you have any questions, please, let us know.

--

Wagner Bianchi, MariaDB RDBA Team
Email: bia...@mariadb.com

To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/CA%2B%2Boe7TGrQRTywKGpUVL-mOoMeQofOz22h8dz_UjVNJF4qBobQ%40mail.gmail.com.

[Erwin Daul]

I think it's the reload/restart part which is important with Ansible.
Sure, we can write a playbook to create/modify configuration files, but then, there's no way other than a restart to apply this new/changed configuration... Which break create a tiny bit of downtime and break currently running transactions (that's the  most annoying part, in our case, some transactions can take a few minutes, sometimes even more).
So if there were a way to tell Maxscale to re-read it's configuration and change/apply whatever it sees new, that would be a blast.

 
--
Erwin

[Wagner Bianchi]

I went a little backwards on your first message, and I got this:

Thing is… there’s no reload option for maxscale (I’m using maxscale 2.4.14)? If I add a new pair of mariadb servers, and configure them (along with Monitor, Services, and Listeners) in maxscale.cnf or etc/maxscale.cnf.d/, I’m forced to do a restart? Isn’t there an alternative?

To create a new cluster dynamically, you have the following as an example. The below can be easily automated as you can pass some extra-vars when calling your playbook for a new replication cluster creation. If you test it out and tail the maxscale log, you will see that your cluster is ready to accept connections after running the below.

#: create the MariaDBMon monitor
maxctrl create monitor mdbmonitor mariadbmon --monitor-user=maxmon --monitor-password=35F6238086BDEB4CF5C3396B10312E84 replication_user=mariadb replication_password=35F6238086BDEB4CF5C3396B10312E84

#: task: configuring the monitor for the replication cluster
maxctrl alter monitor mdbmonitor monitor_interval 2000 
maxctrl alter monitor mdbmonitor failcount 3
maxctrl alter monitor mdbmonitor auto_failover true
maxctrl alter monitor mdbmonitor auto_rejoin true
maxctrl alter monitor mdbmonitor enforce_read_only_slaves true

#: task: create a listener
maxctrl create listener rwsplit-service mdb-listener 3306

#: task: create servers
maxctrl create server n01 45.55.55.213 3306
maxctrl create server n02 45.55.55.215 3306

#: task: link servers with the service
maxctrl link service rwsplit-service n01
maxctrl link service rwsplit-service n02

#: task: link servers with the monitor
maxctrl link monitor mdbmonitor n01
maxctrl link monitor mdbmonitor n02

After running the above by hand or from within your Ansible playbook, you have a cluster (MaxScale+MariaDB Servers) ready to accept connections:

[root@mxs01 ~]# maxctrl list servers --tsv
n02 10.132.0.4 3306 0 Slave, Running 0-1-12
n01 10.132.0.3 3306 0 Master, Running 0-1-12

* the 12 trxs is related to creating users and giving them the right GRANTs.

The above relies on a maxscale basic configuratuion file (/etc/maxscale.cnf) as below:

[maxscale]

threads                     = auto

log_augmentation            = 1

ms_timestamp                = 1

syslog                      = 1

writeq_high_water           = 32M

writeq_low_water            = 512K

[rwsplit-service]

type                        = service

router                      = readwritesplit

user                        = maxusr

password                    = <add your encrypted maxusr password here>

max_sescmd_history          = 500

prune_sescmd_history        = true

master_reconnection         = true

master_failure_mode         = fail_on_write

You can notice that I'm using separate users for the service, the monitor and another one for replication, in case you're going to set up the MariaDBMon - if it is the GaleraMOn you don't need a replication user. The tricky part is that you need to have a playbook to setup MaxScale when I recommend you also to run the maxkeys and maxpasswd within individual plays so you can encrypt or hash users passwords going in configuration files on disk (see below plays for a RedHat based OS):

...

    #: maxscale: the software package

    #: maxscale-experimental: extra features like filters

    #: MariaDB-client: to test ping backends from the maxscale server

    - name: Setting up the MariaDB MaxScale packages (maxscale,maxscale-experiemental)
      yum:
        name: ['maxscale','maxscale-experimental','MariaDB-client']
        state: latest
        security: yes
      tags:
        - maxscale_setup
   
    #: run the maxkeys to create the /var/lib/maxscale/.secrets

    #: if you have more than one maxscale for HA, you need to have the same .secrets all around to run maxctrl cluster sync
    - name: Creating the .secrets key (be carefull when using maxctrl sync cluster for different .secrets)
      shell: maxkeys
      register: cmd_maxkeys
    - debug: msg="{{ cmd_maxkeys.stderr_lines }}"

    #: chown maxscale:maxscale for the /var/lib/maxscale/.secrets (maxscale 2.3, 2.4++ you don't need it)

    #: before executing the below play, you can use stat to check if the file exists for any reason
    - name: Setting up the correct file ownership for the /var/lib/maxscale.secrets
      file:
        path: /var/lib/maxscale/.secrets
        owner: maxscale
        group: maxscale
        mode: 0400

    #: run maxpasswd for users and add to a local variable

    #: if you have users with different passwords, you need to rework the below play

    - name: Creating the encrypted version of the service user password
      shell: maxpasswd "{{ maxscale_service_user }}"
      register: cmd_maxpasswd
    - debug: msg="Service user encrypted password, {{ cmd_maxpasswd.stdout }}"
      

    #: within the maxscale.j2 I'm using the password = {{ cmd_maxpasswd.stdout }} to interpolate the values
    - name: Adding a basic configuration file for MaxScale so we can start the service
      template:
        src: maxscale.j2
        dest: /etc/maxscale.cnf
        owner: root
        group: root
        mode: '0644'

...

Also, there’s no way to alter a listener? I’m forced to destroy and then create one?

You can't alter a listener; you need to drip and recreate if you want to add SSL for a non-SSL listener or change the port it listens for connections, for example. You can use the MaxSCale REST API, or you can use MaxCtrl.

[root@mxs01 ~]# maxctrl destroy listener rwsplit-service mdb-listener
OK

[root@mxs01 ~]# maxctrl create listener rwsplit-service mdb-listener 3307
OK

I hope it helps. If you have any other questions, let's carry on the chat.

--

Wagner Bianchi, MariaDB RDBA Team
Email: bia...@mariadb.com

To view this discussion on the web, visit https://groups.google.com/d/msgid/maxscale/acb00b65-fe56-493f-a564-50d8066fb186n%40googlegroups.com.