PXC/galera crash

[Ilias Bertsimas]

Hello,

It seems we've got a weird crash with Percona Xtradb Cluster 5.5.31-23.7.5-438.squeeze and from the limited backtrace it may point to galera. Only 1 of the 3 nodes crashed.

We also got another weird ring buffer issue after a network split with duplicate rows and self-leaving nodes that I need to gather more information and submit, maybe they are related.

18:48:48 UTC - mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

......

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7ef255]

/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6bbb74]

/lib/libpthread.so.0(+0xeff0)[0x7f4af3b42ff0]

/usr/lib64/libgalera_smm.so(_ZN6gcache10RingBuffer14get_new_bufferEl+0x167)[0x7f4af0f303e7]

/usr/lib64/libgalera_smm.so(_ZN6gcache10RingBuffer6mallocEl+0x39)[0x7f4af0f305e9]

/usr/lib64/libgalera_smm.so(_ZN6gcache6GCache6mallocEl+0x97)[0x7f4af0f31ed7]

/usr/lib64/libgalera_smm.so(gcs_defrag_handle_frag+0x92)[0x7f4af0feb232]

/usr/lib64/libgalera_smm.so(gcs_core_recv+0x4a1)[0x7f4af0ff0b11]

/usr/lib64/libgalera_smm.so(+0x15e5d0)[0x7f4af0ff75d0]

/lib/libpthread.so.0(+0x68ca)[0x7f4af3b3a8ca]

/lib/libc.so.6(clone+0x6d)[0x7f4af27e5b6d]

Kind Regards,

Ilias.

[Alex Yurchenko]

Hi Ilias,

Did it happen right after configuration change?

[Ilias Bertsimas]

Hi Alex,

The last configuration change on the cluster was the following:

130905 19:19:03 [Note] WSREP: forgetting cc0f6846-163c-11e3-bdc4-5eac83527c2d (tcp://x.x.x.x:4567)

130905 19:19:03 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 4

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: sent state msg: 4332fb10-164f-11e3-b17d-23a6df35788b

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: got state msg: 4332fb10-164f-11e3-b17d-23a6df35788b from 0 (node3)

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: got state msg: 4332fb10-164f-11e3-b17d-23a6df35788b from 2 (garb)

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: got state msg: 4332fb10-164f-11e3-b17d-23a6df35788b from 3 (node2)

130905 19:19:03 [Note] WSREP: STATE EXCHANGE: got state msg: 4332fb10-164f-11e3-b17d-23a6df35788b from 1 (node1)

130905 19:19:03 [Note] WSREP: Quorum results:

        version    = 2,

        component  = PRIMARY,

        conf_id    = 212,

        members    = 4/4 (joined/total),

        act_id     = 5104719739,

        last_appl. = 5104718896,

        protocols  = 0/4/2 (gcs/repl/appl),

        group UUID = 402367df-5fd0-11e2-0800-58b321ec9eec

130905 19:19:03 [Note] WSREP: Flow-control interval: [1038090, 1048576]

130905 19:19:03 [Note] WSREP: New cluster view: global state: 402367df-5fd0-11e2-0800-58b321ec9eec:5104719739, view# 213: Primary, number of nodes: 4, my index: 1, protocol version 2

130905 19:19:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

130905 19:19:03 [Note] WSREP: Assign initial position for certification: 5104719739, protocol version: 2

130905 19:19:09 [Note] WSREP:  cleaning up cc0f6846-163c-11e3-bdc4-5eac83527c2d (tcp://x.x.x.x:4567)

But the crash was at least an hour after that.

Kind Regards,

Ilias.

[Alexey Yurchenko]

Hi Ilias. I guess this is starting to get "official attention": https://bugs.launchpad.net/galera/+bug/1152565

[Ilias Bertsimas]

Okay thanks Alex,

I will keep an eye on it!