Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
codership
Home
+ new post
About this group
Join this group
Unknown command after apparently successful join while writes are underway
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   5 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Jase  
View profile  
 More options Oct 24 2012, 1:14 am
From: Jase <jasejon...@gmail.com>
Date: Tue, 23 Oct 2012 22:14:27 -0700 (PDT)
Local: Wed, Oct 24 2012 1:14 am
Subject: Unknown command after apparently successful join while writes are underway
Print | Individual message | Show original | Report this message | Find messages by this author

Hi,

I am attempting to join the 4th node to a working cluster that is currently
receiving writes. 3 nodes are working fine. When I join the 4th node, the
join appears to go fine, there are no problems that I can identify in the
logs, and the wsrep_on variable is set to ON on the joiner.

If I stop writing to any node in the cluster, this node can be restarted
and will not perform another sst but does work properly.

Here are the logs on the joiner:

121024 12:32:50 [Warning] You need to use --log-bin to make --binlog-format
work.
121024 12:32:50 [Note] WSREP: wsrep_load(): loading provider library 'none'
121024 12:32:52 [Note] WSREP: Service disconnected.
121024 12:32:53 [Note] WSREP: Some threads may fail to exit.
121024 12:32:53 [Warning] You need to use --log-bin to make --binlog-format
work.
121024 12:32:53 [Note] WSREP: wsrep_load(): loading provider library 'none'
121024 12:32:53 [Note] WSREP: Service disconnected.
121024 12:32:54 [Note] WSREP: Some threads may fail to exit.
121024 12:33:49 [Note] WSREP: wsrep_load(): loading provider library
'/usr/lib/galera/libgalera_smm.so'
121024 12:33:49 [Note] WSREP: wsrep_load(): Galera 22.1.1(r95) by Codership
Oy loaded succesfully.
121024 12:33:49 [Note] WSREP: Preallocating 134219040/134219040 bytes in
'/var/lib/mysql//galera.cache'...
121024 12:33:49 [Note] WSREP: Passing config to GCS: gcache.dir =
/var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0;
gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M;
gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit =
16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle
= 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit
= 0.25; replicator.commit_order = 3
121024 12:33:49 [Note] WSREP: wsrep_sst_grab()
121024 12:33:49 [Note] WSREP: Start replication
121024 12:33:49 [Warning] WSREP: state file not found:
/var/lib/mysql//grastate.dat
121024 12:33:49 [Note] WSREP: Assign initial position for certification:
-1, protocol version: 1
121024 12:33:49 [Note] WSREP: Setting initial position to
00000000-0000-0000-0000-000000000000:-1
121024 12:33:49 [Note] WSREP: protonet asio version 0
121024 12:33:49 [Note] WSREP: backend: asio
121024 12:33:49 [Note] WSREP: GMCast version 0
121024 12:33:49 [Note] WSREP: (dc3f073b-1d7a-11e2-0800-6c1bbe3cc082,
'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
121024 12:33:49 [Note] WSREP: (dc3f073b-1d7a-11e2-0800-6c1bbe3cc082,
'tcp://0.0.0.0:4567') multicast: , ttl: 1
121024 12:33:49 [Note] WSREP: EVS version 0
121024 12:33:49 [Note] WSREP: PC version 0
121024 12:33:49 [Note] WSREP: gcomm: connecting to group
'my_wsrep_cluster', peer ''
121024 12:33:49 [Note] WSREP: GMCast::handle_stable_view:
view(view_id(PRIM,dc3f073b-1d7a-11e2-0800-6c1bbe3cc082,1) memb {
        dc3f073b-1d7a-11e2-0800-6c1bbe3cc082,

} joined {
} left {
} partitioned {
})

121024 12:33:49 [Note] WSREP: gcomm: connected
121024 12:33:49 [Note] WSREP: Changing maximum packet size to 64500,
resulting msg size: 32636
121024 12:33:49 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
121024 12:33:49 [Note] WSREP: Opened channel 'my_wsrep_cluster'
121024 12:33:49 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0,
memb_num = 1
121024 12:33:49 [Note] WSREP: Waiting for SST to complete.
121024 12:33:49 [Note] WSREP: Starting new group from scratch:
dc3fe281-1d7a-11e2-0800-67dd803567ca
121024 12:33:49 [Note] WSREP: STATE_EXCHANGE: sent state UUID:
dc401389-1d7a-11e2-0800-cf32a98ae4bd
121024 12:33:49 [Note] WSREP: STATE EXCHANGE: sent state msg:
dc401389-1d7a-11e2-0800-cf32a98ae4bd
121024 12:33:49 [Note] WSREP: STATE EXCHANGE: got state msg:
dc401389-1d7a-11e2-0800-cf32a98ae4bd from 0 (Test63)
121024 12:33:49 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (joined/total),
        act_id     = 0,
        last_appl. = -1,
        protocols  = 0/1/1 (gcs/repl/appl),
        group UUID = dc3fe281-1d7a-11e2-0800-67dd803567ca
121024 12:33:49 [Note] WSREP: Flow-control interval: [8, 16]
121024 12:33:49 [Note] WSREP: Restored state OPEN -> JOINED (0)
121024 12:33:49 [Note] WSREP: Member 0 (Test63) synced with group.
121024 12:33:49 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
121024 12:33:49 [Note] WSREP: New cluster view: global state:
dc3fe281-1d7a-11e2-0800-67dd803567ca:0, view# 1: Primary, number of nodes:
1, my index: 0, protocol version 1
121024 12:33:49 [Note] WSREP: SST complete, seqno: 0
121024 12:33:49 InnoDB: The InnoDB memory heap is disabled
121024 12:33:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121024 12:33:49 InnoDB: Compressed tables use zlib 1.2.3.3
121024 12:33:49 InnoDB: Initializing buffer pool, size = 128.0M
121024 12:33:49 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
121024 12:33:49  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
121024 12:33:49  InnoDB: Log file ./ib_logfile0 did not exist: new to be
created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
121024 12:33:49  InnoDB: Log file ./ib_logfile1 did not exist: new to be
created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
121024 12:33:50  InnoDB: Waiting for the background threads to start
121024 12:33:51 InnoDB: 1.1.8 started; log sequence number 0
121024 12:33:51 [Note] Event Scheduler: Loaded 0 events
121024 12:33:51 [Note] WSREP: wsrep_notify_cmd is not defined, skipping
notification.
121024 12:33:51 [Note] WSREP: Assign initial position for certification: 0,
protocol version: 1
121024 12:33:51 [Note] WSREP: Synchronized with group, ready for connections
121024 12:33:51 [Note] WSREP: wsrep_notify_cmd is not defined, skipping
notification.
121024 12:33:51 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.17'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306
 wsrep_22.3.r3645
121024 12:34:36 [Note] /usr/sbin/mysqld: Normal shutdown
121024 12:34:36 [Note] WSREP: Stop replication
121024 12:34:36 [Note] WSREP: Closing send monitor...
121024 12:34:36 [Note] WSREP: Closed send monitor.
121024 12:34:36 [Note] WSREP: gcomm: terminating thread
121024 12:34:36 [Note] WSREP: gcomm: joining thread
121024 12:34:36 [Note] WSREP: gcomm: closing backend
121024 12:34:36 [Note] WSREP: GMCast::handle_stable_view: view((empty))
121024 12:34:36 [Note] WSREP: Received self-leave message.
121024 12:34:36 [Note] WSREP: gcomm: closed
121024 12:34:36 [Note] WSREP: Flow-control interval: [0, 0]
121024 12:34:36 [Note] WSREP: Received SELF-LEAVE. Closing connection.
121024 12:34:36 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 0)
121024 12:34:36 [Note] WSREP: RECV thread exiting 0: Success
121024 12:34:36 [Note] WSREP: New cluster view: global state:
dc3fe281-1d7a-11e2-0800-67dd803567ca:0, view# -1: non-Primary, number of
nodes: 0, my index: -1, protocol version 1
121024 12:34:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping
notification.
121024 12:34:36 [Note] WSREP: applier thread exiting (code:0)
121024 12:34:36 [Note] WSREP: recv_thread() joined.
121024 12:34:36 [Note] WSREP: Closing slave action queue.
121024 12:34:38 [Note] WSREP: rollbacker thread exiting
121024 12:34:38 [Note] Event Scheduler: Purging the queue. 0 events
121024 12:34:38 [Note] WSREP: dtor state: CLOSED
121024 12:34:38 [Note] WSREP: apply mon: entered 0
121024 12:34:38 [Note] WSREP: apply mon: entered 0
121024 12:34:38 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
121024 12:34:38 [Note] WSREP: cert index usage at exit 0
121024 12:34:38 [Note] WSREP: cert trx map usage at exit 0
121024 12:34:38 [Note] WSREP: deps set usage at exit 0
121024 12:34:38 [Note] WSREP: avg deps dist 0
121024 12:34:38 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
121024 12:34:38 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 0)
121024 12:34:38 [Note] WSREP: Flushing memory map to disk...
121024 12:34:38  InnoDB: Starting shutdown...
121024 12:34:38  InnoDB: Shutdown completed; log sequence number 1595675
121024 12:34:38 [Note] /usr/sbin/mysqld: Shutdown complete
121024 12:35:13 [Note] WSREP: wsrep_load(): loading provider library
'/usr/lib/galera/libgalera_smm.so'
121024 12:35:13 [Note] WSREP: wsrep_load(): Galera 22.1.1(r95) by Codership
Oy loaded succesfully.
121024 12:35:13 [Note] WSREP: Reusing existing
'/var/lib/mysql//galera.cache'.
121024 12:35:13 [Note] WSREP: Passing config to GCS: gcache.dir =
/var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0;
gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M;
gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit =
16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle
= 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit
= 0.25; replicator.commit_order = 3
121024 12:35:13 [Note] WSREP: wsrep_sst_grab()
121024 12:35:13 [Note] WSREP: Start replication
121024 12:35:13 [Note] WSREP: Found saved state:
dc3fe281-1d7a-11e2-0800-67dd803567ca:0
121024 12:35:13 [Note] WSREP: Assign initial position for certification: 0,
protocol version: 1
121024 12:35:13 [Note] WSREP: Setting initial position to
dc3fe281-1d7a-11e2-0800-67dd803567ca:0
121024 12:35:13 [Note] WSREP: protonet asio version 0
121024 12:35:13 [Note] WSREP: backend: ...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Henrik Ingo  
View profile  
 More options Oct 24 2012, 3:05 am
From: Henrik Ingo <henrik.i...@avoinelama.fi>
Date: Wed, 24 Oct 2012 10:05:35 +0300
Local: Wed, Oct 24 2012 3:05 am
Subject: Re: [codership-team] Unknown command after apparently successful join while writes are underway
Print | Individual message | Show original | Report this message | Find messages by this author
Hi Jase

From the part of logs that you have sent, everything looks fine so
far. Do the logs end here or is there more? One would expect it to
proceed to a state JOINED -> SYNCED. Until it is

Also, SHOW STATUS LIKE "wsrep%" is often helpful, for instance I
usually check wsrep_cluster_size, which in your case should be 4 on
all nodes. Similarly, instead of wsrep_on variable what you really are
interested in are wsrep_local_state_comment (should be SYNCED) and
wsrep_ready (should be ON).

henrik

...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jase  
View profile  
 More options Oct 24 2012, 8:11 pm
From: Jase <jasejon...@gmail.com>
Date: Wed, 24 Oct 2012 17:11:42 -0700 (PDT)
Local: Wed, Oct 24 2012 8:11 pm
Subject: Re: [codership-team] Unknown command after apparently successful join while writes are underway
Print | Individual message | Show original | Report this message | Find messages by this author

Thanks for your comments Henrik.

Firstly, the logs I posted were the complete logs, so it appears that it
does not proceed fully to the SYNCED state.

Secondly, it seems that my show variables like "wsrep%"; command is not
listing the wsrep_local_state_comment variable. Do you think this might be
because the version I am using is galera-22.1.1-amd64.deb and
mysql-server-wsrep-5.5.17-22.3-amd64.deb? Or could the missing variable be
because of some kind of problem? It seems to me that even if there were a
problem this variable should at least be listed, so it is probably more
likely to be version related.

I also think that given that the join is obviously not proceeding fully, I
would be best to try upgrading the whole cluster to the latest version and
see if the problem still happens. If it does, I'll post again with the show
variables like "wsrep%" run on all nodes, and provide the logs from donor
and joiner. Due to other pressures I probably won't do this now since the
cluster is working, but I will try to do it in the next month or so.

Do you think I should post any further information when I do this (assuming
the same problem happens with the latest version)?

Jase

d=9522559 <http://www.linkedin.com/profile/view?id=9522559>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alex Yurchenko  
View profile  
 More options Oct 25 2012, 2:17 am
From: Alex Yurchenko <alexey.yurche...@codership.com>
Date: Thu, 25 Oct 2012 09:17:23 +0300
Local: Thurs, Oct 25 2012 2:17 am
Subject: Re: [codership-team] Unknown command after apparently successful join while writes are underway
Print | Individual message | Show original | Report this message | Find messages by this author
On 2012-10-25 03:11, Jase wrote:

> Thanks for your comments Henrik.

> Firstly, the logs I posted were the complete logs, so it appears that
> it
> does not proceed fully to the SYNCED state.

It does, it just takes time. The way you describe it, it does.

> Secondly, it seems that my show variables like "wsrep%"; command is
> not
> listing the wsrep_local_state_comment variable. Do you think this
> might be

well, you should probably have posted the full output here, then we
could see.

> because the version I am using is galera-22.1.1-amd64.deb and
> mysql-server-wsrep-5.5.17-22.3-amd64.deb? Or could the missing
> variable be

1.1 is old indeed. We'll be releasing 2.2 today/tomorrow. But that
should not matter in this case.

certainly

--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jase  
View profile  
 More options Oct 31 2012, 6:38 am
From: Jase <jasejon...@gmail.com>
Date: Wed, 31 Oct 2012 03:38:52 -0700 (PDT)
Local: Wed, Oct 31 2012 6:38 am
Subject: Re: Unknown command after apparently successful join while writes are underway
Print | Individual message | Show original | Report this message | Find messages by this author

Hi Alex and Henrik,

Yes Alex, you are right, it does become synced, it just takes some time. It
seems there is no problem. Thanks,

Jase

...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google