Check if SST is in progress?

[Bogdan Dobrelya]

When mysqld starts and initiates either IST or full SST as a joiner, let's say using xtrabackup-v2 with xbstream, it creates a pidfile only after it's ready for connections and responds to SQL queries.

This creates an undefined state for external monitoring cannot *reliably* guess if there is an SST in progress and mysqld is alive OR it's dead/not responding and there is no SST in fact.

Because the pidfile is yet to be created. And select/show based checks are yet to be accepted. A Schrödinger's cat it is!

I'm wondering how one may be sure the mysqld process is running healthy SST OR it is rather dead?

Parsing logs for patterns? Looks like a bad idea.

Checking for xbstream/innobackupex/wsrep_sst* processes running around? I'm not sure, although looks better...

Checking for established connections to the 4444, 4568, 4567? I'm afraid this cannot be trusted.

Any ideas?

[James Wang]

I vaguely remember that if SST is in progressing, socat would be running and also rsync or xtrabackup depending on your SST setting.

Sorry if my memory failed me and please correct me

[Bogdan Dobrelya]

So do I, ended up checking socat/xbstream/innobackupex processes. Would
be nice to make this more native and user friendly (a reliable external
monitoring is a critical UX)

>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "codership" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/codership-team/-g6JhtP0C9o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> codership-tea...@googlegroups.com
> <mailto:codership-tea...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

[James Wang]

Did you mean IST?  SST happens for example when a new node join a cluster (so you are fully aware of it).

To check IST - you may need tcpdump.

May "show status like 'wsrep%'; " help you

I am no expertise here.  Better a codership team member (Alexey, do you read this please, Sir?) to confirm this.

So do I, ended up checking socat/xbstream/innobackupex processes. Would
be nice to make this more native and user friendly (a reliable external
monitoring is a critical UX)
--

[Philip Stoev]

Hello,

For rsync and xtrabackup SST, you can look for a file named sst_in_progress
in the mysql data directory. It is created by the SST scripts when SST
starts and cleaned up at end.

During mysqldump SST, SHOW STATUS should work and wsrep_local_state_comment
should show "Joiner".

Philip Stoev

--
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.

[Bogdan Dobrelya]

On 05/05/2016 03:39 PM, Philip Stoev wrote:
> Hello,
>
> For rsync and xtrabackup SST, you can look for a file named
> sst_in_progress in the mysql data directory. It is created by the SST
> scripts when SST starts and cleaned up at end.

That was an initial solution. Although it had shown being not reliable.
When mysqld crashes with unfinished SST, that file remains to be wiped
off by the nearest SST. And exactly this timeframe is not clear for
external monitoring to reliably detect failure.

>
> During mysqldump SST, SHOW STATUS should work and
> wsrep_local_state_comment should show "Joiner".

I didn't check for mysqldump, but for xtrabackup-v2, this doesn't work
for donor and joiner nodes - they cannot accept connections to select or
show things ;(

>
> Philip Stoev
>
> -----Original Message----- From: Bogdan Dobrelya
> Sent: Thursday, May 05, 2016 11:32
> To: codership
> Subject: [codership-team] Check if SST is in progress?
>
>
> When mysqld starts and initiates either IST or full SST as a joiner,
> let's say using xtrabackup-v2 with xbstream, it creates a pidfile only
> after it's ready for connections and responds to SQL queries.
>
> This creates an undefined state for external monitoring cannot
> *reliably* guess if there is an SST in progress and mysqld is alive OR
> it's dead/not responding and there is no SST in fact.
> Because the pidfile is yet to be created. And select/show based checks
> are yet to be accepted. A Schrödinger's cat it is!
>
> I'm wondering how one may be sure the mysqld process is running healthy
> SST OR it is rather dead?
>
> Parsing logs for patterns? Looks like a bad idea.
> Checking for xbstream/innobackupex/wsrep_sst* processes running around?
> I'm not sure, although looks better...
> Checking for established connections to the 4444, 4568, 4567? I'm afraid
> this cannot be trusted.
>
> Any ideas?
>
>
>
>


--