Docker node fails to come up

[Venkat R]

I'm following the steps on http://galeracluster.com/2015/05/getting-started-galera-with-docker-part-2-2/ to bring up a 3 node galera cluster on DigitalOcean servers (When I try to replicate the same on 3 virtual machines, they work fine.). Every time I try to bring up a node, it fails with only the following shown in docker logs:

------------------

2015-07-23 01:55:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

2015-07-23 01:55:03 1 [Note] WSREP: Setting wsrep_ready to 0

2015-07-23 01:55:03 1 [Note] WSREP: Read nil XID from storage engines, skipping position init

2015-07-23 01:55:03 1 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

2015-07-23 01:55:03 1 [Note] WSREP: wsrep_load(): Galera 3.9(rb7b5dd0) by Codership Oy <in...@codership.com> loaded successfully.

2015-07-23 01:55:03 1 [Note] WSREP: CRC-32C: using hardware acceleration.

2015-07-23 01:55:03 1 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1

2015-07-23 01:55:03 1 [Note] WSREP: Passing config to GCS: base_host = 45.55.226.219; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recove

------------------

I've tried passing --wsrep_debug=ON to check debug logs but that does not improve the logs. I've checked that ufw is disabled on the host and docker is set to complain mode in apparmor.

What more should I check? Why is the node failing to come up? Is there anything I can do further to debug?

-Venkat

[Philip Stoev]

Hello,

The log you have pasted is truncated, so it is not possible to determine
what happened. The first step would be to run docker in foreground mode,
without the -d option, which hopefully will allow you to see the log in its
entirety.

Philip Stoev

--
You received this message because you are subscribed to the Google Groups
"codership" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to codership-tea...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Venkat R]

Philip,

   I'm sorry but even in the foreground mode, the logs are exactly the same before the container exits. I am at my wits end.

-Venkat

[Philip Stoev]

Hello,

The only issue on the Galera side that is known to cause such an abrupt
process exit is related to using systemd to start Galera, which is probably
not the case here.

Therefore I think it is important to obtain the entirety of the log, as
there is probably more of it and an actual error message further down. You
can try the following:

* open an interactive bash session into the container and start mysqld
manually from within. This way you will control the output of the process
directly and should hopefully be able to see all of it.
* start Galera under the gdb debugger so that you can see if it receives a
legitimate signal to exit so soon in the initialization sequence, or if
there is an actual crash.

[Venkatasubramanian Ramachandran]

Philip,

   I did as you suggested - I opened a terminal to the container and ran mysqld --gdb. I find the same logs except for one extra word at the end of the logs which says "Killed"

------------------

root@adc7e957047a:/# mysqld --gdb --debug

2015-07-25 06:53:25 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

2015-07-25 06:53:25 33 [Note] WSREP: Read nil XID from storage engines, skipping position init

2015-07-25 06:53:25 33 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

2015-07-25 06:53:25 33 [Note] WSREP: wsrep_load(): Galera 3.9(rb7b5dd0) by Codership Oy <in...@codership.com> loaded successfully.

2015-07-25 06:53:25 33 [Note] WSREP: CRC-32C: using hardware acceleration.

2015-07-25 06:53:25 33 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1

2015-07-25 06:53:26 33 [Note] WSREP: Passing config to GCS: base_host = 172.17.0.2; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery

Killed

------------------

I tried mysql --debug as well but found no traces generated in /tmp/mysqld.trace. 

What else can I do to debug further?

Thanks,

-Venkat

--

-------------------------------------
Venkat Ramachandran
+91 9449072323

[Venkatasubramanian Ramachandran]

Philip,

  Could you guide me as to what I should do further to generate more debug logs which will be of help?

Thanks,

-Venkat