different datadir on different nodes

19 views

Skip to first unread message

Eyal Hayun

unread,

Mar 26, 2018, 5:35:32 AM3/26/18

to codership

Hi,

I have managed to load 3 similar node cluster (closed LAN location) with the same my.conf file (see below)

but when i set up new node on the same LAN but with different datadir location it failed to start, giving timeout error when running: 'systemctl restart mysql'

The init start of the service show that it managed to connect to all other, looking in the datadir i can see it start pulling data from other nodes and gets fill but it doesn't finish and exiting.

here is /var/log/mysql/error.log

2018-03-26T08:51:31.280654Z 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.16693S), skipping check

2018-03-26T08:51:31.972303Z 0 [Note] WSREP: Node 4c8f514e state prim

2018-03-26T08:51:32.363896Z 0 [Note] WSREP: view(view_id(PRIM,4c8f514e,1512) memb {

4c8f514e,0

93723306,0

b0f7f58d,0

ca91f78b,0

dfd1bbf7,0

} joined {

} left {

} partitioned {

})

2018-03-26T08:51:32.363932Z 0 [Note] WSREP: save pc into disk

2018-03-26T08:51:32.364115Z 0 [Note] WSREP: (dfd1bbf7, 'tcp://0.0.0.0:4567') connection to peer dfd1bbf7 with addr tcp:/4XXXX:4567 timed out, no messages seen in PT3S

2018-03-26T08:51:32.614574Z 0 [Note] WSREP: gcomm: connected

2018-03-26T08:51:32.614627Z 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636

2018-03-26T08:51:32.614769Z 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)

2018-03-26T08:51:32.614783Z 0 [Note] WSREP: Opened channel 'ultranet_cluster'

2018-03-26T08:51:32.614918Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 5

2018-03-26T08:51:32.614965Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

2018-03-26T08:51:32.614935Z 0 [Note] WSREP: Waiting for SST to complete.

2018-03-26T08:51:32.615055Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 09d08537-30d3-11e8-9207-3a3393535fd9

2018-03-26T08:51:32.795216Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 09d08537-30d3-11e8-9207-3a3393535fd9 from 3 (happy)

2018-03-26T08:51:32.797561Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 09d08537-30d3-11e8-9207-3a3393535fd9 from 0 (doc)

2018-03-26T08:51:32.798089Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 09d08537-30d3-11e8-9207-3a3393535fd9 from 1 (dopey)

2018-03-26T08:51:32.798127Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 09d08537-30d3-11e8-9207-3a3393535fd9 from 2 (snow-white)

2018-03-26T08:51:33.002638Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 09d08537-30d3-11e8-9207-3a3393535fd9 from 4 (eagle)

2018-03-26T08:51:33.002681Z 0 [Note] WSREP: Quorum results:

version = 4,

component = PRIMARY,

conf_id = 1469,

members = 4/5 (joined/total),

act_id = 160154,

last_appl. = -1,

protocols = 0/8/3 (gcs/repl/appl),

group UUID = 636f192d-2753-11e8-8337-bf67cf06a4cd

2018-03-26T08:51:33.002697Z 0 [Note] WSREP: Flow-control interval: [36, 36]

2018-03-26T08:51:33.002721Z 0 [Note] WSREP: Trying to continue unpaused monitor

2018-03-26T08:51:33.002732Z 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 160154)

2018-03-26T08:51:33.002896Z 1 [Note] WSREP: State transfer required:

Group state: 636f192d-2753-11e8-8337-bf67cf06a4cd:160154

Local state: 00000000-0000-0000-0000-000000000000:-1

2018-03-26T08:51:33.002945Z 1 [Note] WSREP: New cluster view: global state: 636f192d-2753-11e8-8337-bf67cf06a4cd:160154, view# 1470: Primary, number of nodes: 5, my index: 4, protocol version 3

2018-03-26T08:51:33.002956Z 1 [Warning] WSREP: Gap in state sequence. Need state transfer.

2018-03-26T08:51:33.003108Z 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '4XXX' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '3111' '' '

2018-03-26T08:51:33.114614Z 0 [Note] WSREP: (dfd1bbf7, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-03-26T08:51:33.353249Z 1 [Note] WSREP: Prepared SST request: rsync|4XXX:4444/rsync_sst

2018-03-26T08:51:33.353266Z 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-03-26T08:51:33.353282Z 1 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-03-26T08:51:33.353287Z 1 [Note] WSREP: Assign initial position for certification: 160154, protocol version: 3

2018-03-26T08:51:33.353308Z 0 [Note] WSREP: Service thread queue flushed.

2018-03-26T08:51:33.353368Z 1 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (636f192d-2753-11e8-8337-bf67cf06a4cd): 1 (Operation not permitted)

at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.

2018-03-26T08:51:33.546756Z 0 [Note] WSREP: Member 4.0 (eagle) requested state transfer from '*any*'. Selected 0.0 (doc)(SYNCED) as donor.

2018-03-26T08:51:33.546767Z 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 160154)

2018-03-26T08:51:33.546797Z 1 [Note] WSREP: Requesting state transfer: success, donor: 0

2018-03-26T08:51:33.546810Z 1 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 636f192d-2753-11e8-8337-bf67cf06a4cd:160154

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: Stopped MySQL Wsrep Server.

-- Subject: Unit mysql.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has finished shutting down.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: Starting MySQL Wsrep Server...

-- Subject: Unit mysql.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has begun starting up.

Mar 26 04:01:26 rpusl-eagleUBU mysql[5972]: WSREP: Failed to recover position:

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: mysql.service: Control process exited, code=exited status=1

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: Failed to start MySQL Wsrep Server.

-- Subject: Unit mysql.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has failed.

-- The result is failed.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: mysql.service: Unit entered failed state.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: mysql.service: Failed with result 'exit-code'.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: Stopped MySQL Wsrep Server.

-- Subject: Unit mysql.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has finished shutting down.

Mar 26 04:01:26 rpusl-eagleUBU systemd[1]: Starting MySQL Wsrep Server...

-- Subject: Unit mysql.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has begun starting up.

Mar 26 04:01:27 rpusl-eagleUBU mysql[6046]: WSREP: Failed to recover position:

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: mysql.service: Control process exited, code=exited status=1

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: Failed to start MySQL Wsrep Server.

-- Subject: Unit mysql.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has failed.

-- The result is failed.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: mysql.service: Unit entered failed state.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: mysql.service: Failed with result 'exit-code'.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: Stopped MySQL Wsrep Server.

-- Subject: Unit mysql.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has finished shutting down.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: mysql.service: Start request repeated too quickly.

Mar 26 04:01:27 rpusl-eagleUBU systemd[1]: Failed to start MySQL Wsrep Server.

-- Subject: Unit mysql.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mysql.service has failed.

-- The result is failed.

here is my.cnf i used

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

[mysqld]

datadir=/var/lib/mysql/

socket=/var/lib/mysql/mysql.sock

user=mysql

binlog_format=ROW

bind-address=0.0.0.0

default_storage_engine=innodb

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=4294967296

tmp_table_size=536870912

max_heap_table_size=536870912

wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_cluster_name="ultranet_cluster"

wsrep_cluster_address="gcomm://1XXX,2XXX,3XXX"

wsrep_node_name="node1"

wsrep_node_address="1XXX"

wsrep_sst_method=rsync

[mysql_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

[client]

socket=/var/lib/mysql/mysql.sock

Any idea? I also tried changing to the same datadir on the distance node but same still the node doesn't start?

Brian :

unread,

Mar 27, 2018, 4:33:07 PM3/27/18

to Eyal Hayun, codership

Hi Eyal

There are a couple of bugs that this maybe

https://bugs.launchpad.net/codership-mysql/+bug/1251342
https://jira.mariadb.org/browse/MDEV-10754

Certainly I couldn't ever get non standard data locations to work
using rsync sst method. xtrabackup2 will work though.

Brian

> --
> You received this message because you are subscribed to the Google Groups
> "codership" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to codership-tea...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward

0 new messages