Uhm... you have the monitor on one of the data nodes? That would be a very bad idea ;)

[Tomas]

Hi experts,

I read some old topic and find put monitor on one of the db host will
be a very bad idea,

Can someone tell me why ?


but i do not put monitor on my prefer DB and I never want to a
switchover except a fail,

can that config ok ?

[Arjen Lentz]

Hi Tomas

So if that server fails completely, there'll be no switch. End of story.

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/
OurDelta: enhanced builds for MySQL @ http://ourdelta.org

[Tomas]

Yes, that will cause the trouble,

but i can put a check script running on another master,
If it find the master server with MMM Monitor is down,

It start another monitor(with proper config) running on it self, Is
that workable ?

Thanks a lot.

Yours Tomas

On Sep 30, 9:54 pm, Arjen Lentz <ar...@openquery.com> wrote:
> Hi Tomas
>
> On 01/10/2009, at 2:31 PM, Tomas wrote:
>
> > I read some old topic and find put monitor on one of the db host will
> > be a very bad idea,
>
> > Can someone tell me why ?
>
> > but i do not put monitor on my prefer DB and I never want to a
> > switchover except a fail,
>
> > can that config ok ?
>
> So if that server fails completely, there'll be no switch. End of story.
>
> Cheers,
> Arjen.
> --
> Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
> Exceptional Services for MySQL at a fixed budget.
>

> Follow our blog athttp://openquery.com/blog/

[Arjen Lentz]

Hi Tomas

On 02/10/2009, at 6:34 AM, Tomas wrote:
> Yes, that will cause the trouble,
>
> but i can put a check script running on another master,
> If it find the master server with MMM Monitor is down,
>
> It start another monitor(with proper config) running on it self, Is
> that workable ?

There are specific tools for that kind of stuff (heartbeat,
keepalived). Combining them with MMM might not be the best idea,
there's bound to be some race conditions. What if the MMM on the first
box is unreachable but the box itself works...
You seem to be not using the MMM capabilities at all then, anyway...
I'm seriously wondering what you're trying to do.

If you have a screw, use a screwdriver. If you have a nail, use a
hammer.
Insisting to use a screwdriver when dealing with a nail will make for
a bent nail, a messed up screwdriver, and sore fingers.
Your choice.


Cheers,
Arjen.

> On Sep 30, 9:54 pm, Arjen Lentz <ar...@openquery.com> wrote:
>> Hi Tomas
>>
>> On 01/10/2009, at 2:31 PM, Tomas wrote:
>>
>>> I read some old topic and find put monitor on one of the db host
>>> will
>>> be a very bad idea,
>>
>>> Can someone tell me why ?
>>
>>> but i do not put monitor on my prefer DB and I never want to a
>>> switchover except a fail,
>>
>>> can that config ok ?
>>
>> So if that server fails completely, there'll be no switch. End of
>> story.
>>
>> Cheers,
>> Arjen.
>> --
>> Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
>> Exceptional Services for MySQL at a fixed budget.
>>
>> Follow our blog athttp://openquery.com/blog/
>> OurDelta: enhanced builds for MySQL @http://ourdelta.org
> >

--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Tomas]

Dear Arjen,

Yes, you get the point, What we have is only two servers which
will be setup as Dual master mysql database environment, no more
servers in production environment any more(add a server means cost and
is denied ).

We still need provide HA to from DB side, so we are using MMM with
limited resource.
And i googled in the internet, there seems no good HA solution for
only 2 mysql servers setup as Dual master, yet.

Thanks a lot for your advice. I also very concern about the stability
of MMM.

Yours Tomas

[Istvan Podor]

Hai Tomas,

It's a kind of sucks, but I can understand. I really recommend XEN or
some other virtual machine solution if you want MMM at all cost. If not,
I would recommend drbd+heartbeat for a 2 machine configuration. But if
you need both machine in production, I understand MMM but don't forget
it's not the best for balancing like moving the writer IP to the active
master and the read to the passive all the time.

Maybe HA proxy would be useful. But MMM in this scenerio is not the best
imho.

regards,
Istvan

Tomas írta:

[Baron Schwartz]

Tomas,

> Yes, you get the point, What we have is only two servers which
> will be setup as Dual master mysql database environment, no more
> servers in production environment any more(add a server means cost and
> is denied ).

Then if you want to use MMM, the best you can do is set up the monitor
on the passive master. If the passive master dies, there is nothing
to fail over to anyway, so you lose nothing.

But, if you switch active-passive, then you'd better log in quickly
and switch the monitor to the other node, because now your monitor and
active server are the same, and you won't have failover if it dies.
There are a lot of reasons why this is a bad situation, and why MMM is
not the right choice here, but you can at least get a one-way failover
if you want.

The real problem is, you're trying to get HA on a budget. Good|cheap:
pick one. Sometimes the cost is simply what it is -- you either pay
the money or you don't have HA. Whoever is denying you the money but
demanding the HA needs to be told that they are denying themselves HA.

[Tomas]

Thanks my expert, Baron and IstVan and Arjen,

Yes, you are definitely right, and My plan is put the monitor on
passive server
and provide both read/write function by the active server, my
application do not have read bottleneck yet.

the MMM working here is just a transparent failover plan in case
active server is done.
and I will use MMM angel to monitoring agent and monitor on passive
server frequently.

while switchover happens and the original passive server(now active
one) will be a Single Point of failure :(

I have a question also, during test, even I kill mmm_monitor and
mmm_agent on servers,
the floating IP (my writer IP) still can work, IS that designed to do
so?

How long these floating IP will be available ? because while my
passive server has some trouble, but active server is fine,
I want application still can access normally via the floating Writer
IP address :)

Anyway, Thanks a lot to all these great experts !!!

Yours Tomas

[Istvan Podor]

Hey Tomas,

I have a question also, during test, even I kill mmm_monitor and
mmm_agent on  servers,
the floating IP (my writer IP) still can work,  IS that designed to do
so?

Yes correct. When the monitoring node goes offline, every single agent keeps the last known state until its get another order.

How long these floating IP will be available ? because while my
passive server has some trouble, but active server is fine,
I want application still can access normally via the floating Writer
IP address :)

Answer above :)

Baron's idea to keep the monitoring on the passive master is a great one. But tell me please

if you don't have any slaves and if you wan't to use only one of the masters at one time, why do you need MMM?

Don't misunderstand me, I do love MMM. Its about 8hours and I'm gonna talk about it on a conference, but I think "always choose the best solution for your problem" or with other words Keep it as simple as possible but not more.  DRBD would do this job for you as its planned to do exactly this job. MMM is a little bit more experimental and designed to manage slaves. Are you sure with your decision?

Regards, 

Istvan

[Tomas]

Thank you Istvan,

if use DRDB, I may need install heartbeat or HAproxy to work for
failover, right ? So I need think about more
for failure handling, like DRDB error, HA proxy failure and Heartbeat
failure also, those possibility of failure , i think , will also
increase the unavailability of DB.

if use HAproxy, maybe I need another server to host the monitor, or
the judge role.(correct me if I am wrong, I do not good at HAproxy
yet.)
here is the link for that :
http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/

The most urgent requirement for our DB is keep it alive, while error,
recover As fast as possible.

As MMM can handle it without those requirement, and in the future I
can make use of the passive master(like mysql upgrade,big table
operations, active db server maintanance),
Maybe I will some trouble to move it from Ubuntu to redhat 5
Entherprise. I still think MMM will provide DB HA for us .

google some docs and DRDB is not perfect , someone want move it to MMM
also :)

Thanks a lot for your advice, i will look into more about DRDB and
what we really need.

Great to discuss with experts, learn a lot.

Best regards,

Yours Tomas

[Tomas]

And beside that, I also have slave in another subnet replicate from
those Dual server through SSH tunnel.

I have another question here, from my understanding, this slave 'd
better point to passive mysql db, not any of the read virtual IP
address, am i right ?

Thanks a lot

Yours Tomas

On Oct 2, 2:01 pm, Istvan Podor <istvan.po...@gmail.com> wrote:

[Istvan Podor]

Hey Tomas,

You would just need drbd+heartbeat to work,but no ha proxy. Remote
slave would work too. But on that conference my subject will be "How
to keep staying online, at (almost) all cost" or something like thisso
I understand what you are saying. And I'm one who used drbd and now
use MMM instead :)

Ok then, good to know that you know your possibilities, so if you need
any help... :)

Istvan

[Arjen Lentz]

Hi Tomas

On 03/10/2009, at 6:45 AM, Tomas wrote:
> I have a question also, during test, even I kill mmm_monitor and
> mmm_agent on servers,
> the floating IP (my writer IP) still can work, IS that designed to
> do so?

Depends on how you kill 'em.
In MMM 1.2.x, if somehow MMM shuts down gracefully, the virtual IP
addresses are removed.
Which is nasty.

In MMM 2 they stay.
There are a few other conditions where MMM 1.2.x does the wrong thing,
from Open Query's experience.
For new deployments of MMM we use 2.x now, but we still have a few
clients on MMM 1.2.x.

> How long these floating IP will be available ?

The IP will stay active where it is, unless you were to reboot the
server that has it.
That would make the IP disappear since it's not part of the auto
configuration on startup.

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Arjen Lentz]

Hi Tomas

On 03/10/2009, at 8:59 AM, Tomas wrote:
> And beside that, I also have slave in another subnet replicate from
> those Dual server through SSH tunnel.

I'd recommend using just an SSL slave connection using MySQL's
internal SSL capabilities.
It's a bit of extra work to set up, but less can go wrong - if the
connection drops MySQL will be able to reconnect; if you use SSH, it
can drop and MySQL would not be able to reconnect or do anything about
it.

> I have another question here, from my understanding, this slave 'd
> better point to passive mysql db, not any of the read virtual IP
> address, am i right ?

Usually you'd point slaves at the active master (and MMM can manage
that), since that means they'd be the most up to date when something
happens.
When MMM manages that, it points slaves at the internal IP address of
whatever master it connects to (rather than the virtual IP), so it can
control exactly when a switchover occurs. You can't just flip a slave
to another master with virtualIP as the binlog filename/position would
be different!

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Arjen Lentz]

Hi Tomas,

On 03/10/2009, at 3:29 AM, Baron Schwartz wrote:
> The real problem is, you're trying to get HA on a budget. Good|cheap:
> pick one. Sometimes the cost is simply what it is -- you either pay
> the money or you don't have HA. Whoever is denying you the money but
> demanding the HA needs to be told that they are denying themselves HA.

I think these are wise words.
MMM solutions are pretty low budget anyway, but you can't skimp
further on it without compromising the HA capability.

If you do MMM with only 2 machines, then a) there are conditions it
can't deal with and b) every event will involve manual work, making
mistakes and other problems more likely. In my opinion, and I'm sure
Baron will agree with this also, it's bad economy. An extra machine
does not cost much.
In a VM environment, it's a small extra VM on a different physical box.
In a physical hosting environment, it can be any other box.
So if you have say a webserver or a vm/box for reporting, anything
like that, you would run the MMM monitor there.

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Arjen Lentz]

Hi Tomas

On 03/10/2009, at 8:51 AM, Tomas wrote:
> if use DRDB, I may need install heartbeat or HAproxy to work for
> failover, right ? So I need think about more
> for failure handling, like DRDB error, HA proxy failure and Heartbeat
> failure also, those possibility of failure , i think , will also
> increase the unavailability of DB.

DRBD+Heartbeat is another way of doing things, but it's not the same
as MMM. It covers different situations and your backup server is not
live - so you can't do schema changes (add index, etc) or other
operations on it. It can be a valid choice, depending on your needs.

HAproxy has nothing to do with DRBD/Heartbeat.

> if use HAproxy, maybe I need another server to host the monitor, or
> the judge role.(correct me if I am wrong, I do not good at HAproxy
> yet.)
> here is the link for that :
> http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/

That blog is not quite complete.

Open Query uses HAproxy on its own infrastructure, since we have
masters in separate datacentres and thus we can't have a floating IP.
There are a couple of interesting issues with that which we're still
exploring.
HAproxy too is a different solution from MMM, depending on how you
set it up it will not be exactly the same as MMM.

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Tomas]

Dear Arjen,

Thanks a lot for your advice.
In your comment, you point out if use only 2 servers,

b) every event will involve manual work, making
mistakes and other problems more likely.

I am not sure about that, If I use MMM, application will be
transparent of DB status,
and the failover will be done automatically to the passive DB while
the active DB is not workable,
Why i still need manually involved? (Even i will lost some
transactions, I think the while application will be running without
manually involvement)

Do you mean the other slave server need to be manually point to
passive DB?

because other Slave db is in the other geographic location, I think
they have to be manually changed while error happens.

Thanks a lot
Yours Tomas

On Oct 4, 5:08 pm, Arjen Lentz <ar...@openquery.com> wrote:
> Hi Tomas
>
> On 03/10/2009, at 8:51 AM, Tomas wrote:
>
> > if use DRDB, I may need install heartbeat or HAproxy to work for
> > failover, right ?  So I need think about more
> > for failure handling, like DRDB error, HA proxy failure and Heartbeat
> > failure also, those possibility of failure , i think , will also
> > increase the unavailability of DB.
>
> DRBD+Heartbeat is another way of doing things, but it's not the same  
> as MMM. It covers different situations and your backup server is not  
> live - so you can't do schema changes (add index, etc) or other  
> operations on it. It can be a valid choice, depending on your needs.
>
> HAproxy has nothing to do with DRBD/Heartbeat.
>
> > if use HAproxy, maybe I need another server to host the monitor, or
> > the judge role.(correct me if I am wrong, I do not good at HAproxy  
> > yet.)
> > here is the link for that :

> >http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-fa...

>
> That blog is not quite complete.
>
> Open Query uses HAproxy on its own infrastructure, since we have  
> masters in separate datacentres and thus we can't have a floating IP.
> There are a couple of interesting issues with that which we're still  
> exploring.
> HAproxy  too is a different solution from MMM, depending on how you  
> set it up it will not be exactly the same as MMM.
>
> Cheers,
> Arjen.
> --
> Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
> Exceptional Services for MySQL at a fixed budget.
>

> Follow our blog athttp://openquery.com/blog/

[Arjen Lentz]

Hi Tomas

On 06/10/2009, at 9:41 AM, Tomas wrote:
> Thanks a lot for your advice.
> In your comment, you point out if use only 2 servers,
> b) every event will involve manual work, making
> mistakes and other problems more likely.
>
> I am not sure about that, If I use MMM, application will be
> transparent of DB status,
> and the failover will be done automatically to the passive DB while
> the active DB is not workable,
> Why i still need manually involved?

You want to be running MMM on the passive master.
So when the active master changes, you need to stop MMM on that server
and change it to the other.

And mind you, MMM can change masters even when the active master is
not completely dead, this can happen due to timing and other issues.
Normally that's fine and harmless, but in your case it'll involve a
risk every time, namely that if MMM keeps running on the active master
machine, if the machine actually fails there won't be failover
happening.

So, more babysitting required. I still don't think it's a good idea,
but it's your infra.

Cheers,
Arjen.
--
Arjen Lentz, Exec.Director @ Open Query (http://openquery.com)
Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/

[Tomas]

> You want to be running MMM on the passive master.
> So when the active master changes, you need to stop MMM on that server
> and change it to the other.

Dear Arjen,
Normally , I want MMM monitor running on the passive master.
but if a switch-over happens, MMM monitor will running on active
master, And the switchover always means
1, someone manually do a switch-over, which is controlable.
2. MMM function, which may can configure its behaviour(that is my
concern also)
or
3, the original master has some issue and MMM auto switch to passive
master, at this scenario, I think I do not need change MMM monitor
until the server host "both original passive master(now the active
master) and MMM monitor" failed and the "original active master(which
is passive master now)" is resumed back.

So, I think my plan is try to keep MMM monitor running on passive
master and if a failover/switchover happens, it can work on active
master(original passive master ) for a period of time.
while the normal Active master is resumed, i will switch write IP
back.

In case of Passive master 's host failure, I plan to setup config of
another MMM monitor on active master host also, But i am not sure i
need it ,
That only be useful while my active master server resumed back after a
failure or switch-over and simultaneously find totally passive master
host unreachable.

Yes, you are right, a lot of babysitting need, and make MMM
complicated :( but still, It is HA solution which I found can help us
with limited resources :( .

I think of DRDB also, but We need another server available for other
usage, and switch-over time also need consider. Never test to evaluate
that yet ...

Thanks a lot for your advice.

Yours Tomas

> Follow our blog athttp://openquery.com/blog/

[madpah]

Hi Guys,

Firstly, I've only really just delved into MMM and it's capabilities,
and have just deployed it successfully (in tandem) to two productions
clusters linked by circular replication (it's a fairly mental setup).

Thanks to all involved in MMM - it's made life much easier and it
seems to work fantastically!

Secondly, I've read the above thread (research...) and what strikes me
is that couldn't the Monitoring Node be considered a single point of
failure anyhow, regardless of whether or not it is on a separate box
to all database servers being managed?

I appreciate that the agents on the database servers do nothing
further until they are told to.

What are your thoughts on this? Have I missed something obvious?

Thanks,

Paul

[Walter Heck]

Hi Paul,

it is indeed a SPOF, but not a critical one. You should have some
external monitoring on the monitor, and then yoiu should be fine.

kind regards,

Walter