Multiple nodes on a single host
Patrick Crews
I'm trying to run multiple Galera nodes on a single host for testing
purposes and was wondering if there is any specific variable voodoo
that is required.
I am able to start a single node, running on port 9306, listen port =
4567, but when I attempt to connect additional nodes to this server, I
am met with the following errors:
[Warning] Ignoring user change to 'root' because the user was set to
'mysql' earlier on the command line
111109 17:14:24 [Warning] One can only use the --user switch if
running as root
111109 17:14:24 InnoDB: The InnoDB memory heap is disabled
111109 17:14:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111109 17:14:24 InnoDB: Compressed tables use zlib 1.2.3
111109 17:14:24 InnoDB: Using Linux native AIO
111109 17:14:24 InnoDB: Initializing buffer pool, size = 8.0M
111109 17:14:24 InnoDB: Completed initialization of buffer pool
111109 17:14:24 InnoDB: highest supported file format is Barracuda.
111109 17:14:24 InnoDB: Waiting for the background threads to start
111109 17:14:25 InnoDB: 1.1.8 started; log sequence number 1595675
111109 17:14:25 [Note] Event Scheduler: Loaded 0 events
111109 17:14:25 [Note] WSREP: wsrep_load(): loading provider library '/
usr/lib/galera/libgalera_smm.so'
111109 17:14:25 [Note] WSREP: wsrep_load(): Galera 21.1.0(r85) by
Codership Oy <in...@codership.com> loaded succesfully.
111109 17:14:25 [Note] WSREP: Reusing existing 'dbqp/workdir/bot0/s1/
var/master-data//galera.cache'.
111109 17:14:25 [Note] WSREP: Passing config to GCS: gcache.dir = dbqp/
workdir/bot0/s1/var/master-data/; gcache.keep_pages_size = 0;
gcache.mem_size = 0; gcache.name = dbqp/workdir/bot0/s1/var/master-
data//galera.cache; gcache.page_size = 128M; gcache.size = 128M;
gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16;
gcs.fc_master_slave = NO; gcs.max_packet_size = 64500;
gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807;
gcs.recv_q_soft_limit = 0.25; replicator.commit_order = 3
111109 17:14:25 [Note] WSREP: Start replication
111109 17:14:25 [Warning] WSREP: state file not found: dbqp/workdir/
bot0/s1/var/master-data//grastate.dat
111109 17:14:25 [Note] WSREP: Assign initial position for
certification: -1, protocol version: 1
111109 17:14:25 [Note] WSREP: Setting initial position to
00000000-0000-0000-0000-000000000000:-1
111109 17:14:25 [Note] WSREP: protonet asio version 0
111109 17:14:25 [Note] WSREP: backend: asio
111109 17:14:25 [Note] WSREP: GMCast version 0
111109 17:14:25 [Note] WSREP: (2ed858c2-0b20-11e1-0800-bc27b5b61c53,
'tcp://0.0.0.0:9306') listening at tcp://0.0.0.0:9306
111109 17:14:25 [Note] WSREP: (2ed858c2-0b20-11e1-0800-bc27b5b61c53,
'tcp://0.0.0.0:9306') multicast: , ttl: 1
111109 17:14:25 [Note] WSREP: EVS version 0
111109 17:14:25 [Note] WSREP: PC version 0
111109 17:14:25 [Note] WSREP: gcomm: connecting to group
'my_wsrep_cluster', peer '<hostname>:9306'
111109 17:14:25 [ERROR] WSREP: Address already in use
111109 17:14:25 [ERROR] WSREP: wsrep::connect() failed: 6
111109 17:14:25 [ERROR] Aborting
111109 17:14:25 [Note] WSREP: Service disconnected.
111109 17:14:26 [Note] WSREP: Some threads may fail to exit.
111109 17:14:26 InnoDB: Starting shutdown...
111109 17:14:26 InnoDB: Shutdown completed; log sequence number
1595675
111109 17:14:26 [Note] work/galera/galera55/bin/mysqld: Shutdown
complete
Sever startup command
/galera55/bin/mysqld \
# also tried with variants of ?gmcast.listen_addr w/ no joy
# tried variants of 127.0.0.1 as well.
--wsrep_cluster_address='gcomm://<hostname>:9306' \
--wsrep_provider=/usr/lib/galera/libgalera_smm.so \
--open-files-limit=1024 \
--local-infile \
--character-set-server=latin1 \
--connect-timeout=60 \
--log-bin-trust-function-creators=1 \
--key_buffer_size=1M \
--sort_buffer=256K \
--max_heap_table_size=1M \
--loose-innodb_data_file_path=ibdata1:10M:autoextend \
--loose-innodb_buffer_pool_size=8M \
--loose-innodb_write_io_threads=2 \
--loose-innodb_read_io_threads=2 \
--loose-innodb_log_buffer_size=1M \
--loose-innodb_log_file_size=5M \
--loose-innodb_additional_mem_pool_size=1M \
--loose-innodb_log_files_in_group=2 \
--slave-net-timeout=120 \
--log-bin= dbqp/workdir/bot0/s1/var/log/mysqld-bin \
--loose-enable-performance-schema \
--loose-performance-schema-max-mutex-instances=10000 \
--loose-performance-schema-max-rwlock-instances=10000 \
--loose-performance-schema-max-table-instances=500 \
--loose-performance-schema-max-table-handles=1000 \
--binlog-direct-non-transactional-updates \
--loose-enable-performance-schema \
--general_log=1 \
--general_log_file= dbqp/workdir/bot0/s1/var/log/mysqld.log \
--slow_query_log=1 \
--slow_query_log_file= dbqp/workdir/bot0/s1/var/log/mysqld-slow.log \
--basedir= /galera55 \
--datadir= dbqp/workdir/bot0/s1/var/master-data \
--tmpdir= dbqp/workdir/bot0/s1/var/tmp \
--character-sets-dir= /galera55/share/charsets \
--lc-messages-dir= /galera55/share \
--port=9307 \
--socket= dbqp/workdir/bot0/s1/var/s1.sock \
--pid-file= dbqp/workdir/bot0/s1/var/run/s1.pid \
--default-storage-engine=innodb \
--server-id=2 \
--secure-file-priv=' dbqp/workdir/bot0/s1/var' \
--user=root &
Patrick Crews
I am now able to connect to the first node from a second node on the
same host by setting:
--wsrep_cluster_address='gcomm://127.0.0.1:4567?
gmcast.listen_addr=tcp://127.0.0.1:5678'
where 4567 = the first node's listen port. One *must* specify this
and the new node's gmcast.listen_addr value
I'm now running into issues with state transfer and needing to set
wsrep_sst_receive_address manually for new nodes.
Could someone provide an example of a manual value, say for a server
listening on port 9307? My experiments have not proven fruitful.
Thanks,
Patrick
seppo....@codership.com
Good progress, you figured out the gcomm listen port configuration correctly.
For SST to work, you need to set wsrep_sst_receive_address to the
address where mysqld is listening for connections, and catenate port
after ':' tag. Your configuration could be .e.g:
wsrep_sst_receive_address=192.168.1.10:9307
Pay attention also to the wsrep_sst_auth setting, the account you
specify here must have been granted enough privileges so that donor
can connect in through whatever you specified
'wsrep_sst_receive_address' and load mysql dump.
-seppo
P.S. you seem to enable performance schema there. There was some
trouble with PFS with regard to mysqldump SST. I'm not sure if your
build will have issues with it, but you will see in donor error log if
PFS turns out a kill joy.
Quoting Patrick Crews <patric...@percona.com>:
> --
> You received this message because you are subscribed to the Google
> Groups "codership" group.
> To post to this group, send email to codersh...@googlegroups.com.
> To unsubscribe from this group, send email to
> codership-tea...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/codership-team?hl=en.
Alex Yurchenko
In general there are 4 resources that need to be different when you
want to run several MySQL/Galera nodes on one host:
1) data directory
2) mysql client port and/or address
3) galera replication listen port and/or address
4) receive address for state snapshot transfer
and later incremental state transfer receive address will be added to
the bunch. (I know, it is kinda a lot, but we don't see how it can be
meaningfully reduced yet)
The first two is the usual mysql stuff.
You figured out the third. It is also possible to pass it via
wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:5678"
as most other galera options. This may save you some extra typing.
The fourth one is wsrep_sst_receive_address. This is the address at
which the node will be listening for and receiving the state. Note that
in galera cluster _joining_ nodes are waiting for connections from
donors. It goes contrary to tradition and seems to confuse people time
and again, but there are good reasons it was made like that.
If you use mysqldump SST it should be the same as this mysql client
connection address plus you need to set wsrep_sst_auth variable to hold
user:password pair. The user should be privileged enough to read system
tables from donor and create system tables on this node. For simplicity
that could be just the root user. Note that it also means that you need
to properly set up the privileges on the new node before attempting to
join the cluster.
If you use rsync or xtrabackup SST, wsrep_sst_auth is not necessary
unless your SST script makes use of it. wsrep_sst_address can be
anything local (it may even be the same on all nodes provided you'll be
starting them one at a time).
Best regards,
Alex
--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011
Henrik Ingo
<alexey.y...@codership.com> wrote:
> and later incremental state transfer receive address will be added to the
> bunch. (I know, it is kinda a lot, but we don't see how it can be
> meaningfully reduced yet)
Just as an idea for the future: Galera could bind its own stuff to
listen on some random (available) ports, and make it visible via SHOW
STATUS. Then a joining node would first connect to the mysql port
(3306) and find out the galera specific ports, then connect to those
ports. This would make the 3306 port the only one the user needs to
care about.
Otoh it would make firewall maintenance a mess, so many user will
prefer to configure the port manually or at least have a default port
that is constant.
henrik
--
henri...@avoinelama.fi
+358-40-8211286 skype: henrik.ingo irc: hingo
www.openlife.cc
My LinkedIn profile: http://www.linkedin.com/profile/view?id=9522559