How does orchestrator work with a large distributed topology?

[fb...@yelp.com]

Hi there, 

I am trying to deploy orchestrator to monitor the databases implemented at work. Our setup has several master-replicas topologies running across db hosts. I am a little confused about how orchestrator should be deployed for us. I am following the shared backend model.

Some questions I have specifically:

1) How does orchestrator discover topologies? Does each service simply probe the mysql instance found on the same host on which it is running, and then traverse whatever topology from there?

2) If only the leader node will be discovering topologies, why is the orchestrator service run on all service boxes? 

3) How is the leader node decided?

4) Since there are multiple master-replicas topologies that we would like orchestrator to visualize, does that mean that there will be one leader node for each of these?

Thanks for taking the time to read my questions!

[Manjot Singh]

Hi,

I would start with simply installing one orchestrator instance to understand how it works.

You would set up the topology username and password in /etc/orchestrator.json.conf. Once done, log into the orchestrator web interface and click discover. Put in the FQDN or IP of the master of one of your topologies. It should automatically discover all associated servers. 

Do the same again for a master in each topology.

Once you have done that and are comfortable with discovery, then look into Orchestrator’s HA options.

Thanks,

Manjot Singh
Data & Infrastructure Architect, Percona
Phone 1-919-249-7364,  Skype: percona.msingh
Sacramento, CA USA  GMT-8
http://www.percona.com/
http://www.mysqlperformanceblog.com/

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/4a9bee2d-4e4d-4a95-a5ca-b72bbda64152%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shlomi Noach]

Thanks, Manjot!

@fbox, I'd like to add:

I think you are confusing what the orchestrator "service" boxes are. There are independent from your MySQL boxes. Call them "utility" boxes. You want to have orchestrator installed on, say, three such boxes, just for sake of HA.

(Though I completely agree with Manjot that for starters you can suffice with a single orchestrator box).

Anyway those boxes would monitor your entire MySQL topologies. You can have hundreds or thousands of MySQL servrs in as many clusters you like, and you will still have those three orchestrator boxes monitoring them all.

1) How does orchestrator discover topologies? Does each service simply probe the mysql instance found on the same host on which it is running, and then traverse whatever topology from there?

You will need to educate orchestrator _once_ about your servers. At the very least, tell orchestrator about one servers (say, master) from each topology. orchestrator will then keep those servers in mind, and further drill down/up the topology to discover the entire topology and map the graph.

One common trick: using your puppet/chef/whatever, setup a cronjob on every MySQL server, one per day, to announce itself to orchestrator. Either:

orchestrator -c discover -i $(hostname)

the above assumes you have the orchestrator binary deployed on all MySQL servers (you don't strictly have to), or:

curl http://my.orchestrator.com/api/discover/<fqdn>/3306

the above assumes you have HTTP access to the orchestrator service (which you should).

Starting orchestrator 3.0 (BTW, next week will be a GA release), you can also

orchestrator-client -c discover -i $(hostname)

the above is a script that will run the HTTP API call for you.

2) If only the leader node will be discovering topologies, why is the orchestrator service run on all service boxes? 

Hopefully my initial description answers that. orchestrator should only run on a few boxes; not on MySQL boxes.

3) How is the leader node decided?

the orchestrator nodes all use the same shared backend (you say you use the shared backend setup). They use this logic to elect a leader: http://code.openark.org/blog/mysql/leader-election-using-mysql

4) Since there are multiple master-replicas topologies that we would like orchestrator to visualize, does that mean that there will be one leader node for each of these?

No. A single orchestrator leader node will probe and visualize ALL your servers. The same single leader will run failure detection & failovers for ALL your clusters.

This is proven to work on X,000 servers at Booking.com.

If you do happen to believe the single orchestrator leader is incapable of handling ALL your servers, then it's up to you to "shard": run two or more completely different orchestrator setups (each with its own backend DB), and only educate each orchestrator setup with a subset of your clusters. I don't believe you will be in need of this.

Shlomi

On Fri, Sep 8, 2017 at 7:57 PM, Manjot Singh <manjot...@percona.com> wrote:

Hi,

I would start with simply installing one orchestrator instance to understand how it works.

You would set up the topology username and password in /etc/orchestrator.json.conf. Once done, log into the orchestrator web interface and click discover. Put in the FQDN or IP of the master of one of your topologies. It should automatically discover all associated servers. 

Do the same again for a master in each topology.

Once you have done that and are comfortable with discovery, then look into Orchestrator’s HA options.

Thanks,

Manjot Singh
Data & Infrastructure Architect, Percona
Phone 1-919-249-7364,  Skype: percona.msingh
Sacramento, CA USA  GMT-8
http://www.percona.com/
http://www.mysqlperformanceblog.com/

On Fri, Sep 8, 2017 at 9:48 AM, <fb...@yelp.com> wrote:

Hi there, 

I am trying to deploy orchestrator to monitor the databases implemented at work. Our setup has several master-replicas topologies running across db hosts. I am a little confused about how orchestrator should be deployed for us. I am following the shared backend model.

Some questions I have specifically:

1) How does orchestrator discover topologies? Does each service simply probe the mysql instance found on the same host on which it is running, and then traverse whatever topology from there?

2) If only the leader node will be discovering topologies, why is the orchestrator service run on all service boxes? 

3) How is the leader node decided?

4) Since there are multiple master-replicas topologies that we would like orchestrator to visualize, does that mean that there will be one leader node for each of these?

Thanks for taking the time to read my questions!

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.

To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/4a9bee2d-4e4d-4a95-a5ca-b72bbda64152%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/CALB--KVLKJ1%3DhdVHDwZkK0Q%3Dagfd326OhJM%2BQ28GGHug8TcYFA%40mail.gmail.com.

[fb...@yelp.com]

Thanks Manjot and Shlomi! This clears everything up really well for me.

[Shlomi Noach]

@fbox, if you can identify what parts of the documentation are lacking or confusing, I'd be grateful.

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/054c080f-0a44-4d27-bbab-393d40b65690%40googlegroups.com.

[fb...@yelp.com]

I think I got mixed up reading this page: https://github.com/github/orchestrator/blob/master/docs/deployment-shared-backend.md

I could put up a PR with some suggestions of how I think it could be clearer, if that's cool?

On Friday, September 8, 2017 at 9:48:23 AM UTC-7, fb...@yelp.com wrote:

[Shlomi Noach]

yes please!

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/845fc315-cc90-4922-a3aa-52cc06e522da%40googlegroups.com.