Kernel panic with message "echo 0 > /proc/sys/kernel/hung_task_timeout

amol

unread,

Nov 20, 2012, 11:01:59 PM11/20/12

to percona-d...@googlegroups.com

I had an incident where i was trying to import some data into my cluster db node 1 and the host became unresponsive with this messages in the syslog

[370920.271023] INFO: task mysql:32111 blocked for more than 120 seconds.

[370920.271029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[370920.271110] INFO: task mysql:32115 blocked for more than 120 seconds.

[370920.271116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[370920.271195] INFO: task mysql:32119 blocked for more than 120 seconds.

[370920.271201] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[370920.271282] INFO: task clustercheck:32121 blocked for more than 120 seconds.

[370920.271288] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

any idea what is going on?

this is going to be my production environment, hence i am a little skeptic now..

Alex Yurchenko

unread,

Nov 21, 2012, 6:00:26 AM11/21/12

to percona-d...@googlegroups.com

Too little information. Please describe your load, hardware, cluster
configuration, and provide my.cnf, status variables and log files (log
files from at least two nodes).

--
Alexey Yurchenko,
Codership Oy, www.codership.com
Skype: alexey.yurchenko, Phone: +358-400-516-011

amol

unread,

Nov 21, 2012, 10:58:24 AM11/21/12

to percona-d...@googlegroups.com

ok here are the other details, please let me know if more detaisl are needed

the only operation running on the system that time was importing data into a schema, so there was no such evident load on the system

I have attached the mysql log files from node 1 and node 2 around the time this happened which is around 121120 22:17:58

$ uname -a

Linux db-clusternode1 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

############ my.cnf ############################################################

#

# The MySQL database server configuration file.

#

# You can copy this to one of:

# - "/etc/mysql/my.cnf" to set global options,

# - "~/.my.cnf" to set user-specific options.

#

# One can use all long options that the program supports.

# Run program with --help to get a list of available options and with

# --print-defaults to see which it would actually understand and use.

#

# For explanations see

# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients

# It has been reported that passwords should be enclosed with ticks/quotes

# escpecially if they contain "#" chars...

# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs

# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.

[mysqld_safe]

socket = /var/run/mysqld/mysqld.sock

nice = 0

wsrep_urls = gcomm://10.179.249.32:4567,gcomm://10.178.232.141:4567,gcomm://10.178.232.146:4567,gcomm://

[mysqld]

#

# * Basic Settings

#

server_id=1

binlog_format=ROW

wsrep_provider=/usr/lib64/libgalera_smm.so

wsrep_slave_threads=2

wsrep_cluster_name=cluster

wsrep_sst_method=xtrabackup

wsrep_node_name=db-clusternode1

innodb_locks_unsafe_for_binlog=1

innodb_autoinc_lock_mode=2

log_slave_updates

wsrep_replicate_myisam=1

wsrep_sst_receive_address=10.179.249.32

wsrep_provider_options = "gmcast.listen_addr=tcp://0.0.0.0:4567; ist.recv_addr=10.179.249.32:4568; "

user = mysql

socket = /var/run/mysqld/mysqld.sock

port = 3306

basedir = /usr

datadir = /var/lib/mysql

tmpdir = /tmp

skip-external-locking

#

# Instead of skip-networking the default is now to listen only on

# localhost which is more compatible and is not less secure.

#bind-address = 127.0.0.1

#

# * Fine Tuning

#

key_buffer_size = 32M

myisam_recover = FORCE,BACKUP

max_connect_errors = 1000000

max_allowed_packet = 16M

thread_stack = 256K

max_sp_recursion_depth = 10

# This replaces the startup script and checks MyISAM tables if needed

# the first time they are touched

myisam-recover = BACKUP

#max_connections = 100

#table_cache = 64

#thread_concurrency = 10

tmp_table_size = 32M

max_heap_table_size = 32M

max_connections = 500

thread_cache_size = 50

open_files_limit = 65535

table_definition_cache = 4096

table_open_cache = 10240

#

# * Query Cache Configuration

#

query_cache_limit = 1M

query_cache_size = 16M

#

# * Logging and Replication

#

# Both location gets rotated by the cronjob.

# Be aware that this log type is a performance killer.

# As of 5.1 you can enable the log at runtime!

#general_log_file = /var/log/mysql/mysql.log

#general_log = 1

# LOGGING #

log_error = /var/log/mysql/error.log

slow_query_log = 1

slow_query_log_file = /var/log/mysql/mysql-slow.log

# Here you can see queries with especially long duration

#log_slow_queries = /var/log/mysql/mysql-slow.log

#long_query_time = 2

#log-queries-not-using-indexes

#

# The following can be used as easy to replay backup logs or for replication.

# note: if you are setting up a replication slave, see README.Debian about

# other settings you may need to change.

#server-id = 1

log_bin = /var/log/mysql/mysql-bin.log

expire_logs_days = 10

max_binlog_size = 100M

#binlog_do_db = include_database_name

binlog_ignore_db = performance_schema

binlog_ignore_db = information_schema

binlog_ignore_db = mysql

#

# * InnoDB

#

innodb_flush_method = O_DIRECT

innodb_log_files_in_group = 2

innodb_log_file_size = 128M

innodb_flush_log_at_trx_commit = 1

innodb_file_per_table = 1

innodb_buffer_pool_size = 2G

# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.

# Read the manual for more InnoDB related options. There are many!

#

# * Security Features

#

# Read the manual, too, if you want chroot!

# chroot = /var/lib/mysql/

#

# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".

#

# ssl-ca=/etc/mysql/cacert.pem

# ssl-cert=/etc/mysql/server-cert.pem

# ssl-key=/etc/mysql/server-key.pem

[mysqldump]

quick

quote-names

max_allowed_packet = 16M

[mysql]

#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]

key_buffer = 32M

#

# * IMPORTANT: Additional settings that can override those from this file!

# The files must end with '.cnf', otherwise they'll be ignored.

#

!includedir /etc/mysql/conf.d/

#################### end of my.cnf ##################################

node1_mysql_log.txt

node2_mysql_log.txt

amol

unread,

Nov 21, 2012, 11:04:18 AM11/21/12

to percona-d...@googlegroups.com

secondly i also noticed that now node1 does not start with this error

121121 1:56:50 InnoDB: Completed initialization of buffer pool

InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes

InnoDB: than specified in the .cnf file 0 134217728 bytes!

121121 1:56:50 [ERROR] Plugin 'InnoDB' init function returned error.

121121 1:56:50 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

121121 1:56:50 [ERROR] Unknown/unsupported storage engine: InnoDB

121121 1:56:50 [ERROR] Aborting

I have the entry in the my.cnf for logfile size to be 128 MB but why is it still creating a logfile of 5MB?

is it happening when this process is running in the system ?

13693 mysql 20 0 343m 126m 1904 D 13 3.1 2:09.21 xtrabackup_55 --defaults-file=/etc/mysql/conf.d/mysqld_safe_syslog.cnf --defaults-group=mysqld --prepare --target-dir=/var/lib/mysql

this is what the prepare.log looks like

$ sudo more /var/lib/mysql/innobackup.prepare.log

This software is published under

the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

IMPORTANT: Please check that the apply-log run completes successfully.

At the end of a successful apply-log run innobackupex

prints "completed OK!".

121121 01:05:00 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/mysql/conf.d/mysqld_safe_syslog.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysql

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)

xtrabackup: cd to /var/lib/mysql

xtrabackup: This target seems to be not prepared yet.

xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(8900038974)

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 1

xtrabackup: innodb_log_file_size = 2097152

121121 1:05:00 InnoDB: Using Linux native AIO

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 1

xtrabackup: innodb_log_file_size = 2097152

121121 1:05:00 InnoDB: Using Linux native AIO

xtrabackup: Starting InnoDB instance for recovery.

xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

121121 1:05:00 InnoDB: The InnoDB memory heap is disabled

121121 1:05:00 InnoDB: Mutexes and rw_locks use GCC atomic builtins

121121 1:05:00 InnoDB: Compressed tables use zlib 1.2.3

121121 1:05:00 InnoDB: Using Linux native AIO

121121 1:05:00 InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead

121121 1:05:00 InnoDB: Initializing buffer pool, size = 100.0M

121121 1:05:00 InnoDB: Completed initialization of buffer pool

121121 1:05:00 InnoDB: highest supported file format is Barracuda.

InnoDB: Log scan progressed past the checkpoint lsn 8900038974

121121 1:05:00 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Doing recovery: scanned up to log sequence number 8900039370 (0 %)

InnoDB: Last MySQL binlog file position 0 49592756, file name /var/log/mysql/mysql-bin.000028

121121 1:12:01 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 8900039370

[notice (again)]

If you use binary log and don't use any hack of group commit,

the binary log position seems to be:

InnoDB: Last MySQL binlog file position 0 49592756, file name /var/log/mysql/mysql-bin.000028

xtrabackup: starting shutdown with innodb_fast_shutdown = 1

121121 1:39:18 InnoDB: Starting shutdown...

121121 1:39:22 InnoDB: Shutdown completed; log sequence number 8900039370

121121 01:39:22 innobackupex: Restarting xtrabackup with command: xtrabackup_55 --defaults-file="/etc/mysql/conf.d/mysqld_safe_syslog.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysql

for creating ib_logfile*

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)

xtrabackup: cd to /var/lib/mysql

xtrabackup: This target seems to be already prepared.

xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'.

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 5242880

121121 1:39:22 InnoDB: Using Linux native AIO

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 5242880

121121 1:39:22 InnoDB: Using Linux native AIO

xtrabackup: Starting InnoDB instance for recovery.

xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

121121 1:39:22 InnoDB: The InnoDB memory heap is disabled

121121 1:39:22 InnoDB: Mutexes and rw_locks use GCC atomic builtins

121121 1:39:22 InnoDB: Compressed tables use zlib 1.2.3

121121 1:39:22 InnoDB: Using Linux native AIO

121121 1:39:22 InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead

121121 1:39:22 InnoDB: Initializing buffer pool, size = 100.0M

121121 1:39:22 InnoDB: Completed initialization of buffer pool

121121 1:39:22 InnoDB: Log file ./ib_logfile0 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile0 size to 5 MB

InnoDB: Database physically writes the file full: wait...

121121 1:39:22 InnoDB: Log file ./ib_logfile1 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile1 size to 5 MB

InnoDB: Database physically writes the file full: wait...

121121 1:39:22 InnoDB: highest supported file format is Barracuda.

InnoDB: The log sequence number in ibdata files does not match

InnoDB: the log sequence number in the ib_logfiles!

121121 1:39:22 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Last MySQL binlog file position 0 49592756, file name /var/log/mysql/mysql-bin.000028

121121 1:55:09 InnoDB: Waiting for the background threads to start

121121 1:55:10 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 8900039692

[notice (again)]

If you use binary log and don't use any hack of group commit,

the binary log position seems to be:

InnoDB: Last MySQL binlog file position 0 49592756, file name /var/log/mysql/mysql-bin.000028

xtrabackup: starting shutdown with innodb_fast_shutdown = 1

121121 1:56:44 InnoDB: Starting shutdown...

121121 1:56:49 InnoDB: Shutdown completed; log sequence number 8900039692

121121 01:56:49 innobackupex: completed OK!

Alex Yurchenko

unread,

Nov 21, 2012, 12:29:36 PM11/21/12

to percona-d...@googlegroups.com

You still failed to provide details about your hardware, RAM ans swap
sizes and how you were importing data in your schema (was it just data
or table definitions too, was it InnoDB tables, was it LOAD DATA or
INSERT, how big was it, did it go as a single transaction?) - all of
this is critical to diagnose the issue.

Anyway, most likely your mysqld process has run out of one of the
system resources (memory, open files, etc.) due to a very large
transaction that your load has generated, thus it seem to get stuck
silently without leaving any trace in the log, as you can see.

As for the 5M default innodb log file size (notice
/etc/mysql/conf.d/mysqld_safe_syslog.cnf):

121121 0:22:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup 'joiner'
'10.179.249.32' '' '/var/lib/mysql/'
'/etc/mysql/conf.d/mysqld_safe_syslog.cnf' '26798' 2>sst.err'

- there is a bug
(https://bugs.launchpad.net/codership-mysql/+bug/1079892) in current PXC
where it will use a config file from the bottom of include hierarchy,
rather then the top level one. You either need to avoid using include
directives in my.cnf (as well as .my.cnf), or add required configuration
options (data_dir, innodb_log_file_size, etc.) to the one that is used
in the command line above.

Regards,
Alex

amol

unread,

Nov 21, 2012, 12:38:20 PM11/21/12

to percona-d...@googlegroups.com

yes i was reading about the bug, but never saw a mention about adding the similar settings to /etc/mysql/conf.d/mysqld_safe_syslog.cnf file

currently the file looks like this

# more /etc/mysql/conf.d/mysqld_safe_syslog.cnf

[mysqld_safe]

syslog

regarding hardware and memory information, here is the current memory usage

free -m

total used free shared buffers cached

Mem: 4012 3969 43 0 315 3284

-/+ buffers/cache: 369 3643

Swap: 8189 0 8189

i just don't have the historic memory information from when this happened, i was importing the data using "mysql -u username -p dbname < dbfile.sql" command

Regarding other hardware if you can send me a specific command i can send you the output as this is a cloud server, dont have much control on the hardware

here is the cpuinfo output

# more /proc/cpuinfo

processor : 0

vendor_id : AuthenticAMD

cpu family : 16

model : 4

model name : Quad-Core AMD Opteron(tm) Processor 2374 HE

stepping : 2

cpu MHz : 2182601.405

cache size : 512 KB

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misal

ignsse 3dnowprefetch

bogomips : 4400.18

TLB size : 1024 4K pages

clflush size : 64

cache_alignment : 64

address sizes : 48 bits physical, 48 bits virtual

power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 1

vendor_id : AuthenticAMD

cpu family : 16

model : 4

model name : Quad-Core AMD Opteron(tm) Processor 2374 HE

stepping : 2

cpu MHz : 2182601.405

cache size : 512 KB

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misal

ignsse 3dnowprefetch

bogomips : 4400.18

TLB size : 1024 4K pages

clflush size : 64

cache_alignment : 64

address sizes : 48 bits physical, 48 bits virtual

power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 2

vendor_id : AuthenticAMD

cpu family : 16

model : 4

model name : Quad-Core AMD Opteron(tm) Processor 2374 HE

stepping : 2

cpu MHz : 2182601.405

cache size : 512 KB

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misal

ignsse 3dnowprefetch

bogomips : 4400.18

TLB size : 1024 4K pages

clflush size : 64

cache_alignment : 64

address sizes : 48 bits physical, 48 bits virtual

power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 3

vendor_id : AuthenticAMD

cpu family : 16

model : 4

model name : Quad-Core AMD Opteron(tm) Processor 2374 HE

stepping : 2

cpu MHz : 2182601.405

cache size : 512 KB

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misal

ignsse 3dnowprefetch

bogomips : 4400.18

TLB size : 1024 4K pages

clflush size : 64

cache_alignment : 64

address sizes : 48 bits physical, 48 bits virtual

power management: ts ttp tm stc 100mhzsteps hwpstate

amol

unread,

Nov 21, 2012, 12:42:09 PM11/21/12

to percona-d...@googlegroups.com

and regarding your statement

"You either need to avoid using include
directives in my.cnf (as well as .my.cnf), or add required configuration
options (data_dir, innodb_log_file_size, etc.) to the one that is used
in the command line above. "

will it be ok if i just do a

cp /etc/mysql/my.cnf /etc/mysql/conf.d/mysqld_safe_syslog.cnf

will this work? and do i have to do another SST (or is their workaround before i get to this current state again ) since a complete SST is taking few hours..

On Wednesday, November 21, 2012 12:29:41 PM UTC-5, Alexey Yurchenko wrote:

Alex Yurchenko

unread,

Nov 21, 2012, 2:02:03 PM11/21/12

to percona-d...@googlegroups.com

On 2012-11-21 19:38, amol wrote:
> yes i was reading about the bug, but never saw a mention about adding
> the
> similar settings to /etc/mysql/conf.d/mysqld_safe_syslog.cnf file
>
> currently the file looks like this
>
> # more /etc/mysql/conf.d/mysqld_safe_syslog.cnf
> [mysqld_safe]
> syslog

So that's what xtrabackup uses for configuration file - there is no
mysqld settings at all so it uses default.

I see 2 options:

1) use different SST method (rsync, mysqldump)

2) comment out all 'include' directives in /etc/my.cnf and instead do

cat /etc/mysql/conf.d/* >> /etc/my.cnf

> regarding hardware and memory information, here is the current memory
> usage
>
> free -m
> total used free shared buffers
> cached
> Mem: 4012 3969 43 0 315
> 3284
> -/+ buffers/cache: 369 3643
> Swap: 8189 0 8189
>
> i just don't have the historic memory information from when this
> happened,
> i was importing the data using "mysql -u username -p dbname <
> dbfile.sql"
> command

So, what's inside dbfile.sql?

Are you using the latest galera package?

amol

unread,

Nov 21, 2012, 2:17:13 PM11/21/12

to percona-d...@googlegroups.com

the dbfile.sql is a backup of an entire schema which has about 20k tables and some tables have 10k+ rows

yes i am using latest galera package

$ dpkg --list percona*

Desired=Unknown/Install/Remove/Purge/Hold

| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

||/ Name Version Description

+++-=============================================================-=============================================================-==========================================================================================================================================

ii percona-xtrabackup 2.0.3-470.lucid Open source backup tool for InnoDB and XtraDB

un percona-xtradb-client-5.0 <none> (no description available)

un percona-xtradb-cluster-client-5.1 <none> (no description available)

ii percona-xtradb-cluster-client-5.5 5.5.27-23.6-356.lucid Percona Server database client binaries

ii percona-xtradb-cluster-common-5.5 5.5.27-23.6-356.lucid Percona Server database common files (e.g. /etc/mysql/my.cnf)

un percona-xtradb-cluster-galera <none> (no description available)

ii percona-xtradb-cluster-galera-2.x 115.lucid Galera components of Percona XtraDB Cluster

un percona-xtradb-cluster-server-5.1 <none> (no description available)

ii percona-xtradb-cluster-server-5.5 5.5.27-23.6-356.lucid Percona Server database server binaries

un percona-xtradb-server-5.0 <none> (no description available)

but about your 2nd options

2) comment out all 'include' directives in /etc/my.cnf and instead do

cat /etc/mysql/conf.d/* >> /etc/my.cnf

so before i saw this i copied the /etc/mysql/my.cnf to /etc/mysql/conf.d/mysqld_safe_syslog.cnf

and then restarted mysql and after 2+ hours the node1 is up

but when i started mysql i got these messages

# /etc/init.d/mysql start

Warning: skipping '!includedir /etc/mysql/conf.d/' directive as maximum includerecursion level was reached in file /etc/mysql/conf.d/mysqld_safe_syslog.cnf at line 161

* Starting MySQL (Percona XtraDB Cluster) database server mysqld Warning: skipping '!includedir /etc/mysql/conf.d/' directive as maximum includerecursion level was reached in file /etc/mysql/conf.d/mysqld_safe_syslog.cnf at line 161

Warning: skipping '!includedir /etc/mysql/conf.d/' directive as maximum includerecursion level was reached in file /etc/mysql/conf.d/mysqld_safe_syslog.cnf at line 161

[fail]

i will start the sql file import process again now to migrate my data from production into this cluster db

Alex Yurchenko

unread,

Nov 21, 2012, 3:41:19 PM11/21/12

to percona-d...@googlegroups.com

On 2012-11-21 21:17, amol wrote:
> the dbfile.sql is a backup of an entire schema which has about 20k
> tables

what was the command line you made the backup with?

> and some tables have 10k+ rows

You probably mean 10M+ rows?

> yes i am using latest galera package

Actually no, see
http://www.percona.com/downloads/Percona-XtraDB-Cluster/LATEST/deb/squeeze/x86_64/

Consider to upgrade, a lot of bugs fixed there, should give you much
better mileage. (but not the one with conf file)

> *2) comment out all 'include' directives in /etc/my.cnf and instead
> do
>
> cat /etc/mysql/conf.d/* >> /etc/my.cnf
> *

amol

unread,

Nov 21, 2012, 3:57:22 PM11/21/12

to percona-d...@googlegroups.com

the backup was taken using..

mysqldump --opt -u $MyUSER -h $MyHOST -p$MyPASS $db --lock-tables=false > $FILE

no i mean 10K+ rows (yes it seems quite a small number)

ok and i also upgraded the percona libraries to

$ dpkg --list percona*

||/ Name Version Description

+++-=============================================================-=============================================================-==========================================================================================================================================

ii percona-xtrabackup 2.0.3-470.lucid Open source backup tool for InnoDB and XtraDB

un percona-xtradb-client-5.0 <none> (no description available)

un percona-xtradb-cluster-client-5.1 <none> (no description available)

ii percona-xtradb-cluster-client-5.5 5.5.28-23.7-369.lucid Percona Server database client binaries

ii percona-xtradb-cluster-common-5.5 5.5.28-23.7-369.lucid Percona Server database common files (e.g. /etc/mysql/my.cnf)

un percona-xtradb-cluster-galera <none> (no description available)

ii percona-xtradb-cluster-galera-2.x 115.lucid Galera components of Percona XtraDB Cluster

un percona-xtradb-cluster-server-5.1 <none> (no description available)

ii percona-xtradb-cluster-server-5.5 5.5.28-23.7-369.lucid Percona Server database server binaries

un percona-xtradb-server-5.0 <none> (no description available)

once i upgraded node1 i see these messages in the error.log on node2 and node3

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'File_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user add File_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113266

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 52, g: 113266, s: 113265, d: 113265, ts: 1353530928104784573)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user add Grant_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113267

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 53, g: 113267, s: 113266, d: 113266, ts: 1353530928107066319)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host add Grant_priv enum('N','Y') NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113268

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 54, g: 113268, s: 113267, d: 113267, ts: 1353530928108170521)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db add Grant_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113269

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 55, g: 113269, s: 113268, d: 113268, ts: 1353530928109811164)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'ssl_type'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user

ADD ssl_type enum('','ANY','X509', 'SPECIFIED') COLLATE utf8_general_ci NOT NULL,

ADD ssl_cipher BLOB NOT NULL,

ADD x509_issuer BLOB NOT NULL,

ADD x509_subject BLOB NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113273

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 59, g: 113273, s: 113272, d: 113272, ts: 1353530928115429941)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate key name 'Grantor'' on query. Default database: 'mysql'. Query: 'ALTER TABLE tables_priv

ADD KEY Grantor (Grantor)', Error_code: 1061

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113275

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 61, g: 113275, s: 113274, d: 113274, ts: 1353530928135069693)

121121 15:48:48 [ERROR] Slave SQL: Error 'Unknown column 'Type' in 'columns_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE columns_priv

CHANGE Type Column_priv set('Select','Insert','Update','References')

COLLATE utf8_general_ci DEFAULT '' NOT NULL', Error_code: 1054

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113278

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 64, g: 113278, s: 113277, d: 113277, ts: 1353530928208311204)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'type'' on query. Default database: 'mysql'. Query: 'ALTER TABLE func add type enum ('function','aggregate') COLLATE utf8_general_ci NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113281

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 67, g: 113281, s: 113280, d: 113280, ts: 1353530928238622562)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Show_db_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user

ADD Show_db_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Alter_priv,

ADD Super_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Show_db_priv,

ADD Create_tmp_table_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Super_priv,

ADD Lock_tables_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_tmp_table_priv,

ADD Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Lock_tables_priv,

ADD Repl_slave_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Execute_priv,

ADD Repl_client_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Repl_slave_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113282

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 68, g: 113282, s: 113281, d: 113281, ts: 1353530928241551333)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'max_questions'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user

ADD max_questions int(11) NOT NULL DEFAULT 0 AFTER x509_subject,

ADD max_updates int(11) unsigned NOT NULL DEFAULT 0 AFTER max_questions,

ADD max_connections int(11) unsigned NOT NULL DEFAULT 0 AFTER max_updates', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113284

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 70, g: 113284, s: 113283, d: 113283, ts: 1353530928244061124)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_tmp_table_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db

ADD Create_tmp_table_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL,

ADD Lock_tables_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113285

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 71, g: 113285, s: 113284, d: 113284, ts: 1353530928245147048)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_tmp_table_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host

ADD Create_tmp_table_priv enum('N','Y') DEFAULT 'N' NOT NULL,

ADD Lock_tables_priv enum('N','Y') DEFAULT 'N' NOT NULL', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113286

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 72, g: 113286, s: 113285, d: 113285, ts: 1353530928246267878)

121121 15:48:48 [ERROR] Slave SQL: Error 'You cannot 'ALTER' a log table if logging is enabled' on query. Default database: 'mysql'. Query: 'ALTER TABLE slow_log

MODIFY start_time TIMESTAMP NOT NULL,

MODIFY user_host MEDIUMTEXT NOT NULL,

MODIFY query_time TIME NOT NULL,

MODIFY lock_time TIME NOT NULL,

MODIFY rows_sent INTEGER NOT NULL,

MODIFY rows_examined INTEGER NOT NULL,

MODIFY db VARCHAR(512) NOT NULL,

MODIFY last_insert_id INTEGER NOT NULL,

MODIFY insert_id INTEGER NOT NULL,

MODIFY server_id INTEGER UNSIGNED NOT NULL,

MODIFY sql_text MEDIUMTEXT NOT NULL', Error_code: 1580

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113301

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 87, g: 113301, s: 113300, d: 113300, ts: 1353530928488648079)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Lock_tables_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113303

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 89, g: 113303, s: 113302, d: 113302, ts: 1353530928524697415)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Lock_tables_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113305

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 91, g: 113305, s: 113304, d: 113304, ts: 1353530928542460155)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Create_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Repl_client_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113307

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 93, g: 113307, s: 113306, d: 113306, ts: 1353530928560591976)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Show_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113309

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 95, g: 113309, s: 113308, d: 113308, ts: 1353530928582964796)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Show_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113311

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 97, g: 113311, s: 113310, d: 113310, ts: 1353530928605956589)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Show_view_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Show_view_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113313

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 99, g: 113313, s: 113312, d: 113312, ts: 1353530928625323970)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Show_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113316

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 102, g: 113316, s: 113315, d: 113315, ts: 1353530928648730462)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Show_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113318

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 104, g: 113318, s: 113317, d: 113317, ts: 1353530928666369064)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Create_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Show_view_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113320

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 106, g: 113320, s: 113319, d: 113319, ts: 1353530928682323973)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Alter_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113322

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 108, g: 113322, s: 113321, d: 113321, ts: 1353530928702137853)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Alter_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113324

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 110, g: 113324, s: 113323, d: 113323, ts: 1353530928719970584)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Alter_routine_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Alter_routine_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Create_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113326

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 112, g: 113326, s: 113325, d: 113325, ts: 1353530928738648238)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Execute_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Alter_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113328

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 114, g: 113328, s: 113327, d: 113327, ts: 1353530928759296854)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Execute_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Execute_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Alter_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113330

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 116, g: 113330, s: 113329, d: 113329, ts: 1353530928778325723)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'max_user_connections'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD max_user_connections int(11) unsigned DEFAULT '0' NOT NULL AFTER max_connections', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113335

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 121, g: 113335, s: 113334, d: 113334, ts: 1353530928802469661)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_user_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Create_user_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Alter_routine_priv', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113336

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 122, g: 113336, s: 113335, d: 113335, ts: 1353530928804111384)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'Routine_type'' on query. Default database: 'mysql'. Query: 'ALTER TABLE procs_priv

ADD Routine_type enum('FUNCTION','PROCEDURE')

COLLATE utf8_general_ci NOT NULL AFTER Routine_name', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113342

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 128, g: 113342, s: 113341, d: 113341, ts: 1353530928873758002)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'character_set_client'' on query. Default database: 'mysql'. Query: 'ALTER TABLE proc ADD character_set_client

char(32) collate utf8_bin DEFAULT NULL

AFTER comment', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113347

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 133, g: 113347, s: 113346, d: 113346, ts: 1353530928941203818)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'collation_connection'' on query. Default database: 'mysql'. Query: 'ALTER TABLE proc ADD collation_connection

char(32) collate utf8_bin DEFAULT NULL

AFTER character_set_client', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113350

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 136, g: 113350, s: 113349, d: 113349, ts: 1353530928959300710)

121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name 'db_collation'' on query. Default database: 'mysql'. Query: 'ALTER TABLE proc ADD db_collation

char(32) collate utf8_bin DEFAULT NULL

AFTER collation_connection', Error_code: 1060

121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113353

121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 139, g: 113353, s: 113352, d: 113352, ts: 1353530928979110976)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'body_utf8'' on query. Default database: 'mysql'. Query: 'ALTER TABLE proc ADD body_utf8 longblob DEFAULT NULL

AFTER db_collation', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113356

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 142, g: 113356, s: 113355, d: 113355, ts: 1353530928997102942)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Event_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user add Event_priv enum('N','Y') character set utf8 DEFAULT 'N' NOT NULL AFTER Create_user_priv', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113359

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 145, g: 113359, s: 113358, d: 113358, ts: 1353530929036856994)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Event_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db add Event_priv enum('N','Y') character set utf8 DEFAULT 'N' NOT NULL', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113362

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 148, g: 113362, s: 113361, d: 113361, ts: 1353530929069417604)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'sql_mode'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD sql_mode set ('NOT_USED') AFTER on_completion', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113366

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 152, g: 113366, s: 113365, d: 113365, ts: 1353530929181203862)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'originator'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD COLUMN originator INT UNSIGNED NOT NULL AFTER comment', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113370

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 156, g: 113370, s: 113369, d: 113369, ts: 1353530929232660436)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'time_zone'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD COLUMN time_zone char(64) CHARACTER SET latin1

NOT NULL DEFAULT 'SYSTEM' AFTER originator', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113372

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 158, g: 113372, s: 113371, d: 113371, ts: 1353530929244593970)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'character_set_client'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD character_set_client

char(32) collate utf8_bin DEFAULT NULL

AFTER time_zone', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113373

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 159, g: 113373, s: 113372, d: 113372, ts: 1353530929246321272)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'collation_connection'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD collation_connection

char(32) collate utf8_bin DEFAULT NULL

AFTER character_set_client', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113375

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 161, g: 113375, s: 113374, d: 113374, ts: 1353530929261746658)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'db_collation'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD db_collation

char(32) collate utf8_bin DEFAULT NULL

AFTER collation_connection', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113377

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 163, g: 113377, s: 113376, d: 113376, ts: 1353530929283629068)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'body_utf8'' on query. Default database: 'mysql'. Query: 'ALTER TABLE event ADD body_utf8 longblob DEFAULT NULL

AFTER db_collation', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113379

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 165, g: 113379, s: 113378, d: 113378, ts: 1353530929301893007)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Trigger_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Event_priv', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113381

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 167, g: 113381, s: 113380, d: 113380, ts: 1353530929318926387)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Trigger_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host ADD Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113383

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 169, g: 113383, s: 113382, d: 113382, ts: 1353530929341130759)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Trigger_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db ADD Trigger_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113385

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 171, g: 113385, s: 113384, d: 113384, ts: 1353530929358018068)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'Create_tablespace_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD Create_tablespace_priv enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL AFTER Trigger_priv', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113388

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 174, g: 113388, s: 113387, d: 113387, ts: 1353530929378583146)

121121 15:48:49 [ERROR] Slave SQL: Error 'Duplicate column name 'plugin'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user ADD plugin char(64) DEFAULT '', ADD authentication_string TEXT', Error_code: 1060

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113394

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 180, g: 113394, s: 113393, d: 113393, ts: 1353530929408431747)

121121 15:48:49 [ERROR] Slave SQL: Error 'Table 'mysql.tmp_proxies_priv' doesn't exist' on query. Default database: 'mysql'. Query: 'INSERT INTO tmp_proxies_priv VALUES ('localhost', 'root', '', '', TRUE, '', now())', Error_code: 1146

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113397

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 183, g: 113397, s: 113396, d: 113396, ts: 1353530929448009462)

121121 15:48:49 [ERROR] Slave SQL: Error 'Table 'mysql.tmp_proxies_priv' doesn't exist' on query. Default database: 'mysql'. Query: 'INSERT INTO proxies_priv SELECT * FROM tmp_proxies_priv WHERE @had_proxies_priv_table=0', Error_code: 1146

121121 15:48:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 113398

121121 15:48:49 [Warning] WSREP: Ignoring error for TO isolated action: source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 184, g: 113398, s: 113397, d: 113397, ts: 1353530929449175906)

121121 15:52:23 [Note] WSREP: declaring d6e6ae13-341c-11e2-0800-2c95537c9b81 stable

Alex Yurchenko

unread,

Nov 21, 2012, 4:38:03 PM11/21/12

to percona-d...@googlegroups.com

On 2012-11-21 22:57, amol wrote:
> the backup was taken using..
> mysqldump --opt -u $MyUSER -h $MyHOST -p$MyPASS $db
> --lock-tables=false > $FILE

With --opt you may want to add --skip-create-options if you have any
MyISAM tables there.

> no i mean 10K+ rows (yes it seems quite a small number)
>
> ok and i also upgraded the percona libraries to
>

> ii percona-xtradb-cluster-galera-2.x
> 115.lucid
> Galera components
> of
> Percona XtraDB Cluster

This still seems to be old.

> once i upgraded node1 i see these messages in the error.log on node2
> and
> node3
>
> 121121 15:48:48 [ERROR] Slave SQL: Error 'Duplicate column name
> 'File_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE
> user
> add File_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL',
> Error_code:
> 1060
> 121121 15:48:48 [Warning] WSREP: RBR event 1 Query apply warning: 1,
> 113266
> 121121 15:48:48 [Warning] WSREP: Ignoring error for TO isolated
> action:
> source: d6e6ae13-341c-11e2-0800-2c95537c9b81 version: 2 local: 0
> state:
> APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 52, g: 113266, s:
> 113265, d: 113265, ts: 1353530928104784573)

Well, you seem to be updating system tables (running mysql_upgrade?) -
which already have these columns. Generally this should be harmless, but
you should check that it is really what you want.

amol

unread,

Nov 21, 2012, 5:02:42 PM11/21/12

to percona-d...@googlegroups.com

yes for the myisam tables i am anyways editing the sql file by using this command

:%s/ENGINE=MyISAM/ENGINE=InnoDB/g

so all tables are created as innodb

for the libraries i just did an apt-get update and

sudo apt-get install -V percona-xtrabackup percona-xtradb-cluster-client-5.5 percona-xtradb-cluster-common-5.5 percona-xtradb-cluster-galera-2.x percona-xtradb-cluster-server-5.5

and about the errors i got those on all the 3 nodes, so am not so sure what i have to do about them?...

amol

unread,

Nov 25, 2012, 11:05:25 AM11/25/12

to percona-d...@googlegroups.com

Hi,

So my node 1 crashed again last night while i was migrating the live data and i had to abort the migration

scenario: i was logged in to the server from 2 shell sessions. First session was inserting data into the database using mysql -u username -p schema_name1 < dump file1.sql

In the second session i was using vi dump_file2.sql to ":%s/ENGINE=MyISAM/ENGINE=InnoDB/g"

so that all the tables are created with innodb, this file was about 2GB

and i got this erors again

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

what could be the reason this is happening over and over again my VM has 4GB RAM and innodb buffer pool is set to abt 2GB

So finally had to call the host provider and get the VM rebooted

and secondly now the node1 does not comeup again

here is the innobackup.prepare.log from node1

121125 04:46:03 innobackupex: Restarting xtrabackup with command: xtrabackup_55 --defaults-file="/etc/mysql/conf.d/mysqld_safe_syslog.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysql

for creating ib_logfile*

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)

xtrabackup: cd to /var/lib/mysql

xtrabackup: This target seems to be already prepared.

xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'.

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 134217728

121125 4:46:03 InnoDB: Using Linux native AIO

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 134217728

121125 4:46:03 InnoDB: Using Linux native AIO

xtrabackup: Starting InnoDB instance for recovery.

xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

121125 4:46:03 InnoDB: The InnoDB memory heap is disabled

121125 4:46:03 InnoDB: Mutexes and rw_locks use GCC atomic builtins

121125 4:46:03 InnoDB: Compressed tables use zlib 1.2.3

121125 4:46:03 InnoDB: Using Linux native AIO

121125 4:46:03 InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead

121125 4:46:03 InnoDB: Initializing buffer pool, size = 100.0M

121125 4:46:03 InnoDB: Completed initialization of buffer pool

121125 4:46:03 InnoDB: Log file ./ib_logfile0 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile0 size to 128 MB

InnoDB: Database physically writes the file full: wait...

InnoDB: Progress in MB: 100

121125 4:46:10 InnoDB: Log file ./ib_logfile1 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile1 size to 128 MB

InnoDB: Database physically writes the file full: wait...

InnoDB: Progress in MB: 100

121125 4:46:15 InnoDB: highest supported file format is Barracuda.

InnoDB: The log sequence number in ibdata files does not match

InnoDB: the log sequence number in the ib_logfiles!

121125 4:46:15 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Last MySQL binlog file position 0 17886101, file name /var/log/mysql/mysql-bin.000082

121125 4:55:13 InnoDB: Waiting for the background threads to start

121125 4:55:14 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 15744097292

[notice (again)]

If you use binary log and don't use any hack of group commit,

the binary log position seems to be:

InnoDB: Last MySQL binlog file position 0 17886101, file name /var/log/mysql/mysql-bin.000082

xtrabackup: starting shutdown with innodb_fast_shutdown = 1

121125 4:55:58 InnoDB: Starting shutdown...

121125 4:55:59 InnoDB: Shutdown completed; log sequence number 15744097292

121125 04:55:59 innobackupex: completed OK!

Please suggest me..also let me know if you need more log files..

amol

unread,

Nov 25, 2012, 6:14:34 PM11/25/12

to percona-d...@googlegroups.com

now when i reboot the node, some process of --wsre--recover starts

on node 1

$ ps -ef | grep mysql

root 1192 1 0 16:07 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe

mysql 2668 1192 5 16:07 ? 00:00:02 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --open-files-limit=65535 --pid-file=/var/lib/mysql/db-clusternode1.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --log_error=/tmp/tmp.w5J6VnBiPE --wsrep-recover

akedar 3618 2900 0 16:08 pts/0 00:00:00 grep --color=auto mysql

and then in the mysql log file i see

121125 16:07:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

121125 16:07:21 mysqld_safe WSREP: 'wsrep_urls' is DEPRECATED! Use wsrep_cluster_address to specify multiple addresses instead.

121125 16:07:21 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.w5J6VnBiPE

my /etc/mysql/my.cnf

# * Basic Settings

#

server_id=1

binlog_format=ROW

wsrep_provider=/usr/lib64/libgalera_smm.so

wsrep_slave_threads=2

wsrep_cluster_name=iform_cluster

wsrep_sst_method=rsync

wsrep_node_name=db-clusternode1

innodb_locks_unsafe_for_binlog=1

innodb_autoinc_lock_mode=2

log_slave_updates

wsrep_replicate_myisam=1

wsrep_sst_receive_address=10.179.249.32

but still the rsync process does not start, it is still doing an xtrabackup? how can i correct that

and here is the innobackup.prepare.log from node1

# more innobackup.prepare.log

This software is published under

the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

IMPORTANT: Please check that the apply-log run completes successfully.

At the end of a successful apply-log run innobackupex

prints "completed OK!".

121125 17:01:54 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/mysql/conf.d/mysqld_safe_syslog.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysql

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)

xtrabackup: cd to /var/lib/mysql

xtrabackup: This target seems to be not prepared yet.

xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(15744101787)

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 1

xtrabackup: innodb_log_file_size = 2097152

121125 17:01:54 InnoDB: Using Linux native AIO

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 1

xtrabackup: innodb_log_file_size = 2097152

121125 17:01:54 InnoDB: Using Linux native AIO

xtrabackup: Starting InnoDB instance for recovery.

xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

121125 17:01:54 InnoDB: The InnoDB memory heap is disabled

121125 17:01:54 InnoDB: Mutexes and rw_locks use GCC atomic builtins

121125 17:01:54 InnoDB: Compressed tables use zlib 1.2.3

121125 17:01:54 InnoDB: Using Linux native AIO

121125 17:01:54 InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead

121125 17:01:54 InnoDB: Initializing buffer pool, size = 100.0M

121125 17:01:54 InnoDB: Completed initialization of buffer pool

121125 17:01:54 InnoDB: highest supported file format is Barracuda.

InnoDB: The log sequence number in ibdata files does not match

InnoDB: the log sequence number in the ib_logfiles!

121125 17:01:54 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Last MySQL binlog file position 0 1636, file name /var/log/mysql/mysql-bin.000083

121125 17:07:35 InnoDB: Waiting for the background threads to start

121125 17:07:36 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 15744101787

[notice (again)]

If you use binary log and don't use any hack of group commit,

the binary log position seems to be:

InnoDB: Last MySQL binlog file position 0 1636, file name /var/log/mysql/mysql-bin.000083

xtrabackup: starting shutdown with innodb_fast_shutdown = 1

121125 17:30:26 InnoDB: Starting shutdown...

121125 17:30:31 InnoDB: Shutdown completed; log sequence number 15744101787

121125 17:30:31 innobackupex: Restarting xtrabackup with command: xtrabackup_55 --defaults-file="/etc/mysql/conf.d/mysqld_safe_syslog.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysql

for creating ib_logfile*

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)

xtrabackup: cd to /var/lib/mysql

xtrabackup: This target seems to be already prepared.

xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'.

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 134217728

121125 17:30:31 InnoDB: Using Linux native AIO

xtrabackup: Temporary instance for recovery is set as followings.

xtrabackup: innodb_data_home_dir = ./

xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend

xtrabackup: innodb_log_group_home_dir = ./

xtrabackup: innodb_log_files_in_group = 2

xtrabackup: innodb_log_file_size = 134217728

121125 17:30:31 InnoDB: Using Linux native AIO

xtrabackup: Starting InnoDB instance for recovery.

xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

121125 17:30:31 InnoDB: The InnoDB memory heap is disabled

121125 17:30:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins

121125 17:30:31 InnoDB: Compressed tables use zlib 1.2.3

121125 17:30:31 InnoDB: Using Linux native AIO

121125 17:30:31 InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead

121125 17:30:31 InnoDB: Initializing buffer pool, size = 100.0M

121125 17:30:31 InnoDB: Completed initialization of buffer pool

121125 17:30:31 InnoDB: Log file ./ib_logfile0 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile0 size to 128 MB

InnoDB: Database physically writes the file full: wait...

InnoDB: Progress in MB: 100

121125 17:30:36 InnoDB: Log file ./ib_logfile1 did not exist: new to be created

InnoDB: Setting log file ./ib_logfile1 size to 128 MB

InnoDB: Database physically writes the file full: wait...

InnoDB: Progress in MB: 100

121125 17:30:41 InnoDB: highest supported file format is Barracuda.

InnoDB: The log sequence number in ibdata files does not match

InnoDB: the log sequence number in the ib_logfiles!

121125 17:30:41 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information from the .ibd files...

InnoDB: Last MySQL binlog file position 0 1636, file name /var/log/mysql/mysql-bin.000083

121125 17:39:21 InnoDB: Waiting for the background threads to start

121125 17:39:22 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 15744101900

[notice (again)]

If you use binary log and don't use any hack of group commit,

the binary log position seems to be:

InnoDB: Last MySQL binlog file position 0 1636, file name /var/log/mysql/mysql-bin.000083

xtrabackup: starting shutdown with innodb_fast_shutdown = 1

121125 17:40:17 InnoDB: Starting shutdown...

121125 17:40:21 InnoDB: Shutdown completed; log sequence number 15744101900

121125 17:40:21 innobackupex: completed OK!

Alex Yurchenko

unread,

Nov 25, 2012, 8:14:04 PM11/25/12

to percona-d...@googlegroups.com

On 2012-11-25 18:05, amol wrote:
> Hi,
> So my node 1 crashed again last night while i was migrating the live
> data
> and i had to abort the migration
>
> scenario: i was logged in to the server from 2 shell sessions. First
> session was inserting data into the database using mysql -u username
> -p
> schema_name1 < dump file1.sql
> In the second session i was using vi dump_file2.sql to
> ":%s/ENGINE=MyISAM/ENGINE=InnoDB/g"
> so that all the tables are created with innodb, this file was about
> 2GB

Do you have swap enabled? How much of it? I'm not sure how vi does it,
but 2GB in buffer pool + 2GB file to edit can be pretty risky if you
have only 4GB in RAM and no swap. Had you use --skip-create-options to
dump the database, you would not have needed that edit stage.

> and i got this erors again
>
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message
>
> what could be the reason this is happening over and over again my VM
> has
> 4GB RAM and innodb buffer pool is set to abt 2GB
> So finally had to call the host provider and get the VM rebooted

What sort of virtualization does your provider use (xen, kvm, openvz)?
What top says in this situation?
Do you use innodb_file_per_table option? What is your open file limit
there?

> *and secondly now the node1 does not comeup again*

Error logs?

amol

unread,

Nov 25, 2012, 10:28:08 PM11/25/12

to percona-d...@googlegroups.com

On Sunday, November 25, 2012 8:14:07 PM UTC-5, Alexey Yurchenko wrote:

On 2012-11-25 18:05, amol wrote:
> Hi,
> So my node 1 crashed again last night while i was migrating the live
> data
> and i had to abort the migration
>
> scenario: i was logged in to the server from 2 shell sessions. First
> session was inserting data into the database using mysql -u username
> -p
> schema_name1 < dump file1.sql
> In the second session i was using vi dump_file2.sql to
> ":%s/ENGINE=MyISAM/ENGINE=InnoDB/g"
> so that all the tables are created with innodb, this file was about
> 2GB

Do you have swap enabled? How much of it? I'm not sure how vi does it,
but 2GB in buffer pool + 2GB file to edit can be pretty risky if you
have only 4GB in RAM and no swap.

yes i do have swap enabled here is the memory info,

# free -m

total used free shared buffers cached

Mem: 4012 1391 2621 0 202 931

-/+ buffers/cache: 257 3755

Swap: 8189 0 8189

Had you use --skip-create-options to dump the database, you would not have needed that edit stage.

but my table structure is dynamic, as we keep creating tables and adding columns everyday, so i need that point in time table structure plus there are more that 30000 tables

> and i got this erors again
>
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message
>
> what could be the reason this is happening over and over again my VM
> has
> 4GB RAM and innodb buffer pool is set to abt 2GB
> So finally had to call the host provider and get the VM rebooted

What sort of virtualization does your provider use (xen, kvm, openvz)? not sure about this since its done by cloud server provider...

What top says in this situation? can't get to top as the machine becomes unresponsive and only way out is to reboot the node

Do you use innodb_file_per_table option? What is your open file limit there?

yes

open_files_limit = 65535

innodb_file_per_table = 1

> *and secondly now the node1 does not comeup again*

Error logs?

nothing really in the error log

So now when i deleted everything in the /var/lib/mysql directory and added rsync as an option in /etc/mysql/conf.d/mysqld_safe_syslog.cnf

i get these errors in the error.log

121125 20:09:42 [Note] WSREP: Quorum results:

version = 2,

component = PRIMARY,

conf_id = 102,

members = 2/3 (joined/total),

act_id = 249440,

last_appl. = -1,

protocols = 0/4/2 (gcs/repl/appl),

group UUID = b70128b0-28fd-11e2-0800-85a302db6d44

121125 20:09:42 [Note] WSREP: Flow-control interval: [14, 28]

121125 20:09:42 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 249440)

121125 20:09:42 [Note] WSREP: State transfer required:

Group state: b70128b0-28fd-11e2-0800-85a302db6d44:249440

Local state: 00000000-0000-0000-0000-000000000000:-1

121125 20:09:42 [Note] WSREP: New cluster view: global state: b70128b0-28fd-11e2-0800-85a302db6d44:249440, view# 103: Primary, number of nodes: 3, my index: 2, protocol version 2

121125 20:09:42 [Warning] WSREP: Gap in state sequence. Need state transfer.

121125 20:09:44 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.179.249.32' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/conf.d/mysqld_safe_syslog.cnf' --parent '19974''

failed to create pid file /var/lib/mysql//rsync_sst.pid: Invalid argument

cat: /var/lib/mysql//rsync_sst.pid: No such file or directory

amol

unread,

Nov 25, 2012, 10:58:01 PM11/25/12

to percona-d...@googlegroups.com

also on the donor node (node 3) i see this error in the error.log during the xtrabackup SST

121125 22:40:22 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 249440)

121125 22:40:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

121125 22:40:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address '10.179.249.32:4444/xtrabackup_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/conf.d/mysqld_safe_syslog.cnf' --gtid 'b70128b0-28fd-11e2-0800-85a302db6d44:249440''

121125 22:40:22 [Note] WSREP: sst_donor_thread signaled with 0

mktemp: too few X's in template `wsrep_sst_xtrabackupXXXX.pid'

amol

unread,

Nov 26, 2012, 1:17:43 AM11/26/12

to percona-d...@googlegroups.com

ok once i sycned the time on all the 3 nodes and tried to restart node1, now it crashes with this errors in the log

121126 00:42:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

121126 00:42:26 mysqld_safe WSREP: 'wsrep_urls' is DEPRECATED! Use wsrep_cluster_address to specify multiple addresses instead.

121126 00:42:26 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.w3avGLPBgn

121126 00:42:35 mysqld_safe WSREP: Failed to recover position:

121126 0:42:36 [Note] WSREP: Read nil XID from storage engines, skipping position init

121126 0:42:36 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'

121126 0:42:36 [Note] WSREP: wsrep_load(): Galera 2.2(r115) by Codership Oy <in...@codership.com> loaded succesfully.

121126 0:42:36 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1

121126 0:42:36 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.

121126 0:42:36 [Note] WSREP: Passing config to GCS: base_host = aa.bb.cc.212; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://0.0.0.0:4567; ist.recv_addr = 10.179.249.32:4568; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3

121126 0:42:36 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1

121126 0:42:36 [Note] WSREP: wsrep_sst_grab()

121126 0:42:36 [Note] WSREP: Start replication

121126 0:42:36 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1

121126 0:42:36 [Note] WSREP: protonet asio version 0

121126 0:42:36 [Note] WSREP: backend: asio

121126 0:42:36 [Note] WSREP: GMCast version 0

121126 0:42:36 [Note] WSREP: (155603d7-378c-11e2-0800-22c17158d6ed, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

121126 0:42:36 [Note] WSREP: (155603d7-378c-11e2-0800-22c17158d6ed, 'tcp://0.0.0.0:4567') multicast: , ttl: 1

121126 0:42:36 [Note] WSREP: EVS version 0

121126 0:42:36 [Note] WSREP: PC version 0

121126 0:42:36 [Note] WSREP: gcomm: connecting to group 'cluster', peer '10.178.232.141:4567'

121126 0:42:36 [Note] WSREP: (155603d7-378c-11e2-0800-22c17158d6ed, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.178.232.146:4567

121126 0:42:36 [Note] WSREP: (155603d7-378c-11e2-0800-22c17158d6ed, 'tcp://0.0.0.0:4567') turning message relay requesting off

121126 0:42:37 [Note] WSREP: declaring 99a78c91-341e-11e2-0800-9fbbe4a6d569 stable

121126 0:42:37 [Note] WSREP: declaring e4e1dafe-341e-11e2-0800-8175a79f5788 stable

121126 0:42:37 [Note] WSREP: view(view_id(PRIM,155603d7-378c-11e2-0800-22c17158d6ed,112) memb {

155603d7-378c-11e2-0800-22c17158d6ed,

99a78c91-341e-11e2-0800-9fbbe4a6d569,

e4e1dafe-341e-11e2-0800-8175a79f5788,

} joined {

} left {

} partitioned {

})

121126 0:42:37 [Note] WSREP: gcomm: connected

121126 0:42:37 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636

121126 0:42:37 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)

121126 0:42:37 [Note] WSREP: Opened channel 'cluster'

121126 0:42:37 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

121126 0:42:37 [Note] WSREP: Waiting for SST to complete.

121126 0:42:37 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 15ef9809-378c-11e2-0800-9a7240636586

121126 0:42:37 [Note] WSREP: STATE EXCHANGE: sent state msg: 15ef9809-378c-11e2-0800-9a7240636586

121126 0:42:37 [Note] WSREP: STATE EXCHANGE: got state msg: 15ef9809-378c-11e2-0800-9a7240636586 from 0 (db-clusternode1)

121126 0:42:37 [Note] WSREP: STATE EXCHANGE: got state msg: 15ef9809-378c-11e2-0800-9a7240636586 from 1 (db-clusternode3)

121126 0:42:37 [Note] WSREP: STATE EXCHANGE: got state msg: 15ef9809-378c-11e2-0800-9a7240636586 from 2 (db-clusternode2)

121126 0:42:37 [Note] WSREP: Quorum results:

version = 2,

component = PRIMARY,

conf_id = 109,

members = 2/3 (joined/total),

act_id = 249440,

last_appl. = -1,

protocols = 0/4/2 (gcs/repl/appl),

group UUID = b70128b0-28fd-11e2-0800-85a302db6d44

121126 0:42:37 [Note] WSREP: Flow-control interval: [14, 28]

121126 0:42:37 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 249440)

121126 0:42:37 [Note] WSREP: State transfer required:

Group state: b70128b0-28fd-11e2-0800-85a302db6d44:249440

Local state: 00000000-0000-0000-0000-000000000000:-1

121126 0:42:37 [Note] WSREP: New cluster view: global state: b70128b0-28fd-11e2-0800-85a302db6d44:249440, view# 110: Primary, number of nodes: 3, my index: 0, protocol version 2

121126 0:42:37 [Warning] WSREP: Gap in state sequence. Need state transfer.

121126 0:42:39 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.179.249.32' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/conf.d/mysqld_safe_syslog.cnf' --parent '1825''

121126 0:42:40 [Note] WSREP: Prepared SST request: xtrabackup|10.179.249.32:4444/xtrabackup_sst

121126 0:42:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

121126 0:42:40 [Note] WSREP: Assign initial position for certification: 249440, protocol version: 2

121126 0:42:40 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (b70128b0-28fd-11e2-0800-85a302db6d44): 1 (Operation not permitted)

at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.

121126 0:42:40 [Note] WSREP: Node 0 (db-clusternode1) requested state transfer from '*any*'. Selected 1 (db-clusternode3)(SYNCED) as donor.

121126 0:42:40 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 249440)

121126 0:42:40 [Note] WSREP: Requesting state transfer: success, donor: 1

121126 0:57:25 [Note] WSREP: (155603d7-378c-11e2-0800-22c17158d6ed, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.178.232.141:4567

121126 0:57:25 [Note] WSREP: declaring e4e1dafe-341e-11e2-0800-8175a79f5788 stable

121126 0:57:25 [Note] WSREP: view(view_id(PRIM,155603d7-378c-11e2-0800-22c17158d6ed,113) memb {

155603d7-378c-11e2-0800-22c17158d6ed,

e4e1dafe-341e-11e2-0800-8175a79f5788,

} joined {

} left {

} partitioned {

99a78c91-341e-11e2-0800-9fbbe4a6d569,

})

121126 0:57:25 [Note] WSREP: forgetting 99a78c91-341e-11e2-0800-9fbbe4a6d569 (tcp://10.178.232.146:4567)

05:57:25 UTC - mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Please help us make Percona Server better by reporting any

bugs at http://bugs.percona.com/

key_buffer_size=0

read_buffer_size=131072

max_used_connections=0

max_threads=300

thread_count=2

connection_count=2

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 656594 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0 thread_stack 0x40000

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7f2dd5]

/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x6bc344]

/lib/libpthread.so.0(+0xf8f0)[0x7fc4416588f0]

/lib/libc.so.6(+0x7c577)[0x7fc440875577]

/lib/libc.so.6(__libc_malloc+0x6e)[0x7fc440876ade]

/usr/lib/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x59)[0x7fc43d937d79]

/usr/lib/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x2b)[0x7fc43d93874b]

/usr/lib/libstdc++.so.6(_ZNSs7reserveEm+0x3c)[0x7fc43d93927c]

/usr/lib/libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0xaf)[0x7fc43d93307f]

/usr/lib/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x35)[0x7fc43d9373c5]

/usr/lib/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE13_M_insert_intImEES3_S3_RSt8ios_basecT_+0x14b)[0x7fc43d91ca6b]

/usr/lib/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE6do_putES3_RSt8ios_basecm+0x16)[0x7fc43d91cbe6]

/usr/lib/libstdc++.so.6(_ZNSo9_M_insertImEERSoT_+0x9e)[0x7fc43d92f0ee]

/usr/lib64/libgalera_smm.so(_ZNK5gcomm4UUID9to_streamERSo+0xa6)[0x7fc43e2f1716]

/usr/lib64/libgalera_smm.so(+0x166afb)[0x7fc43e2edafb]

/usr/lib64/libgalera_smm.so(gcs_core_recv+0xf4)[0x7fc43e2e3474]

/usr/lib64/libgalera_smm.so(+0x163460)[0x7fc43e2ea460]

/lib/libpthread.so.0(+0x69ca)[0x7fc44164f9ca]

/lib/libc.so.6(clone+0x6d)[0x7fc4408e216d]

You may download the Percona Server operations manual by visiting

http://www.percona.com/software/percona-server/. You may find information

in the manual which will help you identify the cause of the crash.

121126 00:57:25 mysqld_safe mysqld from pid file /var/lib/mysql/db-clusternode1.pid ended

Johncan Lam

unread,

Nov 25, 2012, 7:37:18 PM11/25/12

to percona-d...@googlegroups.com

Maybe it's a bug about kernel 2.6.32,just upgrade your kernel

see the url: http://rhn.redhat.com/errata/RHBA-2012-0124.html

2012/11/26 amol <ajk...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.

Johncan Lam

unread,

Nov 25, 2012, 7:31:10 PM11/25/12

to percona-d...@googlegroups.com

Maybe it's a bug about kernel 2.6.32,just upgrade your kernel

see the url: http://rhn.redhat.com/errata/RHBA-2012-0124.html

2012/11/26 amol <ajk...@gmail.com>

now when i reboot the node, some process of --wsre--recover starts

--

amol

unread,

Nov 26, 2012, 2:59:00 PM11/26/12

to percona-d...@googlegroups.com

i am not running on red hat i am using ubuntu

node1:~$ uname -a

Linux node1 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

node1:~$ cat /etc/issue

Ubuntu 10.04.4 LTS \n \l

Alex Yurchenko

unread,

Nov 26, 2012, 8:10:58 PM11/26/12

to percona-d...@googlegroups.com

It is still the same kernel version and the build seems to be dated to
before the bug discovery (at least to before that page).

On 2012-11-26 21:59, amol wrote:
> i am not running on red hat i am using ubuntu
>
> node1:~$ uname -a
> Linux node1 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC
> 2011
> x86_64 GNU/Linux
>
> node1:~$ cat /etc/issue
> Ubuntu 10.04.4 LTS \n \l
>
>
>
> On Sunday, November 25, 2012 7:37:18 PM UTC-5, Johncan Lam wrote:
>>
>> Maybe it's a bug about kernel 2.6.32,just upgrade your kernel
>>
>> see the url: http://rhn.redhat.com/errata/RHBA-2012-0124.html
>>

Alex Yurchenko

unread,

Nov 26, 2012, 8:27:25 PM11/26/12

to percona-d...@googlegroups.com

Ok that may be the problem. What does

# ulimit -n

say? If it is less than 65535, you need to go and fix the system-wide
open file limit (/etc/security/limits) to be no less than the value
above.

Alex Yurchenko

unread,

Nov 26, 2012, 8:30:13 PM11/26/12

to percona-d...@googlegroups.com

On 2012-11-26 08:17, amol wrote:
> ok once i sycned the time on all the 3 nodes and tried to restart
> node1,
> now it crashes with this errors in the log
>
> 121126 00:42:26 mysqld_safe Starting mysqld daemon with databases
> from
> /var/lib/mysql
> 121126 00:42:26 mysqld_safe WSREP: 'wsrep_urls' is DEPRECATED! Use
> wsrep_cluster_address to specify multiple addresses instead.
> 121126 00:42:26 mysqld_safe WSREP: Running position recovery with
> --log_error=/tmp/tmp.w3avGLPBgn
> 121126 00:42:35 mysqld_safe WSREP: Failed to recover position:
> 121126 0:42:36 [Note] WSREP: Read nil XID from storage engines,
> skipping
> position init
> 121126 0:42:36 [Note] WSREP: wsrep_load(): loading provider library
> '/usr/lib64/libgalera_smm.so'
> 121126 0:42:36 [Note] WSREP: wsrep_load(): Galera 2.2(r115) by
> Codership
> Oy <in...@codership.com> loaded succesfully.

You are still not using the latest Galera version.

>> the *xtrabackup* SST

Amol Kedar

unread,

Nov 26, 2012, 10:25:44 PM11/26/12

to percona-d...@googlegroups.com

yes the output of
ulimit -n is
1024

so i need to find the system wide settings in ubuntu ..and i also need to upgrade the kernel (to ubuntu 12.04 LTS)?

yes i got the rsync SST working by removing the /etc/mysql/conf.d/mysqld_safe_syslog.cnf file and restarting the database and rsync worked fine...

and regarding the galera

$ sudo apt-get install percona-xtradb-cluster-galera-2.x
Reading package lists... Done
Building dependency tree
Reading state information... Done
percona-xtradb-cluster-galera-2.x is already the newest version.

$ sudo dpkg --list percona*
||/ Name Version Description
+++-=====================================================-=====================================================-==========================================================================================================================

ii percona-xtrabackup 2.0.3-470.lucid Open source backup tool for InnoDB and XtraDB

ii percona-xtradb-cluster-client-5.5 5.5.28-23.7-369.lucid Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.28-23.7-369.lucid Percona Server database common files (e.g. /etc/mysql/my.cnf)

ii percona-xtradb-cluster-galera-2.x 115.lucid Galera components of Percona XtraDB Cluster

ii percona-xtradb-cluster-server-5.5 5.5.28-23.7-369.lucid Percona Server database server binaries

so has it been pushed to the repository?

Alex Yurchenko

unread,

Nov 27, 2012, 4:48:19 AM11/27/12

to percona-d...@googlegroups.com

On 2012-11-27 05:25, Amol Kedar wrote:
> yes the output of
> ulimit -n is
> 1024
>
> so i need to find the system wide settings in ubuntu ..and i also
> need to upgrade the kernel (to ubuntu 12.04 LTS)?

yes, and the rest of the packages as well: apt-get update && apt-get
upgrade

http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster/LATEST/deb/lucid/x86_64/percona-xtradb-cluster-galera-2.x_117.lucid_amd64.deb

I'm not sure what's wrong there, but maybe you need to manually remove
the old version and install the new one.

Amol Kedar

unread,

Nov 27, 2012, 2:50:01 PM11/27/12

to percona-d...@googlegroups.com

On Nov 27, 2012, at 4:48 AM, Alex Yurchenko <alexey.y...@codership.com> wrote:

On 2012-11-27 05:25, Amol Kedar wrote:
yes the output of
ulimit -n is
1024

so i need to find the system wide settings in ubuntu ..and i also
need to upgrade the kernel (to ubuntu 12.04 LTS)?

yes, and the rest of the packages as well: apt-get update && apt-get upgrade

yes i edited the ulimit conf and also ran a apt-get update and apt-get upgrade

but that does not upgrade the kernel or the ubuntu release

:~$ uname -a

Linux iform-db-clusternode1 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

:~$ cat /etc/lsb-release

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=10.04

DISTRIB_CODENAME=lucid

DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"

ok i manually installed the package, thanks for the link

sudo dpkg --list percona*

||/ Name Version Description

+++-==============================================================-==============================================================-

ii percona-xtrabackup 2.0.3-470.lucid Open source backup tool for InnoDB and XtraDB

ii percona-xtradb-cluster-client-5.5 5.5.28-23.7-369.lucid Percona Server database client binaries

ii percona-xtradb-cluster-common-5.5 5.5.28-23.7-369.lucid Percona Server database common files (e.g. /etc/mysql/my.cnf)

ii percona-xtradb-cluster-galera-2.x 117.lucid Galera components of Percona XtraDB Cluster

ii percona-xtradb-cluster-server-5.5 5.5.28-23.7-369.lucid Percona Server database server binaries

Alex Yurchenko

unread,

Nov 27, 2012, 3:01:39 PM11/27/12

to percona-d...@googlegroups.com

On 2012-11-27 21:50, Amol Kedar wrote:
> On Nov 27, 2012, at 4:48 AM, Alex Yurchenko
> <alexey.y...@codership.com> wrote:
>
>> On 2012-11-27 05:25, Amol Kedar wrote:
>>> yes the output of
>>> ulimit -n is
>>> 1024
>>>
>>> so i need to find the system wide settings in ubuntu ..and i also
>>> need to upgrade the kernel (to ubuntu 12.04 LTS)?
>>
>> yes, and the rest of the packages as well: apt-get update && apt-get
>> upgrade
>>
>
> yes i edited the ulimit conf and also ran a apt-get update and
> apt-get upgrade
>
> but that does not upgrade the kernel or the ubuntu release

It should not upgrade the release or kernel version, but it should
update all packages to the latest patch levels. Have you rebooted after
doing this?

Amol Kedar

unread,

Nov 27, 2012, 3:28:21 PM11/27/12

to percona-d...@googlegroups.com

yes i did reboot all the nodes and they all did come up fine

and i did modify some innodb parameters in the my.cnf to reduce the memory usage

innodb_log_files_in_group = 2

innodb_log_file_size = 64M # reduced the log file size from 128M to 64M

innodb_flush_log_at_trx_commit = 1

innodb_file_per_table = 1

innodb_buffer_pool_size = 1G # reduced the buffer pool from 2GB to 1GB

$ ulimit -n

65535

few questions i have now

1) can i use one of the 3 nodes to configure as a master for another slave machine..for e.g

on node 2

setup

log_bin = /var/log/mysql/mysql-bin.log

max_binlog_size = 100M

binlog_ignore_db = performance_schema

binlog_ignore_db = information_schema

binlog_ignore_db = mysql

so this ways i can configures another slave server which can read off these binary logs...

so now, will there be a problem if node2 now is used as a donor for other joining nodes during SST? becuase it has binary logging turned on..

2) i read that using the parameter #innodb_flush_method = O_DIRECT is efficient on ubuntu/linux

but when i add the parameter to the configuration, i always get these kind of warnings

121126 17:57:43 InnoDB: Failed to set O_DIRECT on file ./emobileCdb/UserP.ibd: OPEN: Invalid argument, continuing anyway

121126 17:57:43 InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662

is this a concern? i disabled the parameter after i saw these errors but will that cause my db to be slower? due to caching on the OS memory and db memory?

Alex Yurchenko

unread,

Nov 27, 2012, 4:07:19 PM11/27/12

to percona-d...@googlegroups.com

On 2012-11-27 22:28, Amol Kedar wrote:
> yes i did reboot all the nodes and they all did come up fine

And what does uname -a say after reboot? Does apt-get upgrade tell you
"The following packages have been kept back: ...."

there should not be.

> 2) i read that using the parameter #innodb_flush_method =
> O_DIRECT is efficient on ubuntu/linux
> but when i add the parameter to the configuration, i always get these
> kind of warnings
>
> 121126 17:57:43 InnoDB: Failed to set O_DIRECT on file
> ./emobileCdb/UserP.ibd: OPEN: Invalid argument, continuing anyway
> 121126 17:57:43 InnoDB: O_DIRECT is known to result in 'Invalid
> argument' on Linux on tmpfs, see MySQL Bug#26662

Oh boy... I guess this explains a lot. Are you sure you want to keep
your data in tmpfs? How big is your dataset?

> is this a concern? i disabled the parameter after i saw these errors
> but will that cause my db to be slower? due to caching on the OS
> memory and db memory?

Amol Kedar

unread,

Nov 27, 2012, 4:19:53 PM11/27/12

to percona-d...@googlegroups.com

On Nov 27, 2012, at 4:07 PM, Alex Yurchenko <alexey.y...@codership.com> wrote:

> On 2012-11-27 22:28, Amol Kedar wrote:
>> yes i did reboot all the nodes and they all did come up fine
>
> And what does uname -a say after reboot? Does apt-get upgrade tell you "The following packages have been kept back: ...."

yes here is the output

$ uname -a
Linux node2 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

$ sudo apt-get upgrade

Reading package lists... Done
Building dependency tree
Reading state information... Done

The following packages have been kept back:

linux-image-virtual linux-virtual
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

the dataset is between 15-20GB right now, i am not sure about the advantage of tmpfs?

>
>> is this a concern? i disabled the parameter after i saw these errors
>> but will that cause my db to be slower? due to caching on the OS
>> memory and db memory?
>
> --
> Alexey Yurchenko,
> Codership Oy, www.codership.com
> Skype: alexey.yurchenko, Phone: +358-400-516-011
>

Alex Yurchenko

unread,

Nov 27, 2012, 4:46:33 PM11/27/12

to percona-d...@googlegroups.com

well, you should read about it. Judging by this your MySQL servers are
severely misconfigured and would not work even standalone. You should
first fix this before attempting setting up Galera cluster.

Amol Kedar

unread,

Nov 27, 2012, 5:29:05 PM11/27/12

to percona-d...@googlegroups.com

well i used the percona tool for generating the cnf paramters?
can you give me a pointer to where they would be severely misconfigured.?
here is my.cnf file

$ more /etc/mysql/my.cnf

# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
wsrep_urls = gcomm://10.179.249.32:4567,gcomm://10.178.232.141:4567,gcomm://10.178.232.146:4567,gcomm://

[mysqld]
#

# * Basic Settings
#
server_id=1
binlog_format=ROW
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_slave_threads=2

wsrep_cluster_name=cluster
wsrep_sst_method=xtrabackup
wsrep_node_name=node1

innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
log_slave_updates
wsrep_replicate_myisam=1
wsrep_sst_receive_address=10.179.249.32

wsrep_provider_options = "gmcast.listen_addr=tcp://0.0.0.0:4567; ist.recv_addr=10.179.249.32:4568; "

user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer_size = 32M
myisam_recover = FORCE,BACKUP
max_connect_errors = 1000000
max_allowed_packet = 16M
thread_stack = 256K
max_sp_recursion_depth = 10
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
tmp_table_size = 32M
max_heap_table_size = 32M
max_connections = 300
thread_cache_size = 50
open_files_limit = 65535
table_definition_cache = 4096
table_open_cache = 10240

#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
# LOGGING #
log_error = /var/log/mysql/error.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log

# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
#expire_logs_days = 10
#max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = performance_schema
#binlog_ignore_db = information_schema
#binlog_ignore_db = mysql

#
# * InnoDB
#
#innodb_flush_method = O_DIRECT

innodb_log_files_in_group = 2
innodb_log_file_size = 64M

innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = 1
innodb_buffer_pool_size = 1G

# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 32M

#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Alex Yurchenko

unread,

Nov 27, 2012, 5:56:34 PM11/27/12

to percona-d...@googlegroups.com

The fact that you place your data files on tmpfs without knowing the
consequences (and data size being far greater than RAM size) means that
you're largely out of control over your machines. The problem is not in
my.cnf, it is in overall machine setup. Read about tmpfs and decide what
you need to do. First you need to learn how to configure standalone
MySQL servers and only then go to clusters.

Justin Swanhart

unread,

Nov 27, 2012, 6:04:39 PM11/27/12

to percona-d...@googlegroups.com

Hi,

First try to determine why your data is on tmpfs if you don't expect
it to be there.

Provide the output of the following commands:
mysqladmin var | egrep "datadir|basedir|innodb"
sudo mount
ls -ldh /var/lib/mysql

--Justin

On Tue, Nov 27, 2012 at 2:56 PM, Alex Yurchenko

Amol Kedar

unread,

Nov 27, 2012, 6:53:34 PM11/27/12

to percona-d...@googlegroups.com

ok here is the information.

$ mysqladmin -u xxx -p var | egrep "datadir|basedir|innodb"

Enter password:

| basedir | /usr

| datadir | /var/lib/mysql/

| have_innodb | YES

| ignore_builtin_innodb | OFF

| innodb_adaptive_flushing | ON

| innodb_adaptive_flushing_method | estimate

| innodb_adaptive_hash_index | ON

| innodb_adaptive_hash_index_partitions | 1

| innodb_additional_mem_pool_size | 8388608

| innodb_autoextend_increment | 8

| innodb_autoinc_lock_mode | 2

| innodb_blocking_buffer_pool_restore | OFF

| innodb_buffer_pool_instances | 1

| innodb_buffer_pool_populate | OFF

| innodb_buffer_pool_restore_at_startup | 0

| innodb_buffer_pool_shm_checksum | ON

| innodb_buffer_pool_shm_key | 0

| innodb_buffer_pool_size | 1073741824

| innodb_change_buffering | all

| innodb_changed_pages_limit | 1000000

| innodb_checkpoint_age_target | 0

| innodb_checksums | ON

| innodb_commit_concurrency | 0

| innodb_concurrency_tickets | 500

| innodb_corrupt_table_action | assert

| innodb_data_file_path | ibdata1:10M:autoextend

| innodb_data_home_dir |

| innodb_dict_size_limit | 0

| innodb_disallow_writes | OFF

| innodb_doublewrite | ON

| innodb_doublewrite_file |

| innodb_fake_changes | OFF

| innodb_fast_checksum | OFF

| innodb_fast_shutdown | 1

| innodb_file_format | Antelope

| innodb_file_format_check | ON

| innodb_file_format_max | Antelope

| innodb_file_per_table | ON

| innodb_flush_log_at_trx_commit | 1

| innodb_flush_method |

| innodb_flush_neighbor_pages | area

| innodb_force_load_corrupted | OFF

| innodb_force_recovery | 0

| innodb_ibuf_accel_rate | 100

| innodb_ibuf_active_contract | 1

| innodb_ibuf_max_size | 536854528

| innodb_import_table_from_xtrabackup | 0

| innodb_io_capacity | 200

| innodb_kill_idle_transaction | 0

| innodb_large_prefix | OFF

| innodb_lazy_drop_table | 0

| innodb_lock_wait_timeout | 50

| innodb_locks_unsafe_for_binlog | ON

| innodb_log_block_size | 512

| innodb_log_buffer_size | 8388608

| innodb_log_file_size | 67108864

| innodb_log_files_in_group | 2

| innodb_log_group_home_dir | ./

| innodb_max_dirty_pages_pct | 75

| innodb_max_purge_lag | 0

| innodb_merge_sort_block_size | 1048576

| innodb_mirrored_log_groups | 1

| innodb_old_blocks_pct | 37

| innodb_old_blocks_time | 0

| innodb_open_files | 300

| innodb_page_size | 16384

| innodb_purge_batch_size | 20

| innodb_purge_threads | 1

| innodb_random_read_ahead | OFF

| innodb_read_ahead | linear

| innodb_read_ahead_threshold | 56

| innodb_read_io_threads | 4

| innodb_recovery_stats | OFF

| innodb_recovery_update_relay_log | OFF

| innodb_replication_delay | 0

| innodb_rollback_on_timeout | OFF

| innodb_rollback_segments | 128

| innodb_show_locks_held | 10

| innodb_show_verbose_locks | 0

| innodb_spin_wait_delay | 6

| innodb_stats_auto_update | 1

| innodb_stats_method | nulls_equal

| innodb_stats_on_metadata | ON

| innodb_stats_sample_pages | 8

| innodb_stats_update_need_lock | 1

| innodb_strict_mode | OFF

| innodb_support_xa | ON

| innodb_sync_spin_loops | 30

| innodb_table_locks | ON

| innodb_thread_concurrency | 0

| innodb_thread_concurrency_timer_based | OFF

| innodb_thread_sleep_delay | 10000

| innodb_track_changed_pages | OFF

| innodb_use_global_flush_log_at_trx_commit | ON

| innodb_use_native_aio | ON

| innodb_use_sys_malloc | ON

| innodb_use_sys_stats_table | OFF

| innodb_version | 1.1.8-rel29.1

| innodb_write_io_threads | 4

$ sudo mount

/dev/xvda1 on / type ext3 (rw,noatime,errors=remount-ro)

proc on /proc type proc (rw)

none on /sys type sysfs (rw,noexec,nosuid,nodev)

none on /sys/fs/fuse/connections type fusectl (rw)

none on /sys/kernel/debug type debugfs (rw)

none on /sys/kernel/security type securityfs (rw)

none on /dev type devtmpfs (rw,mode=0755)

none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)

none on /dev/shm type tmpfs (rw,nosuid,nodev)

none on /var/run type tmpfs (rw,nosuid,mode=0755)

none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)

none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)

none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)

none on /proc/xen type xenfs (rw)

/var/lib/ezncrypt/storage/encrypted_private on /var/lib/ezncrypt/ezncrypted type ecryptfs (rw,ecryptfs_sig=b9289e421e22f5b4,ecryptfs_unlink_sigs,ecryptfs_cipher=aes,ecryptfs_key_bytes=32)

/var/lib/ezncrypt/ezncrypted on /var/lib/ezncrypt/ezncrypted type ezncryptfs (rw,dir_default=/var/lib/ezncrypt/storage/unencrypted_private,dir_crypt=/var/lib/ezncrypt/ezncrypted)

$ ls -ldh /var/lib/mysql

lrwxrwxrwx 1 mysql root 50 2012-11-07 15:34 /var/lib/mysql -> /var/lib/ezncrypt/ezncrypted/mysqld//var/lib/mysql

(don't be alarmed by this, we use an encryption software for the files..)

$ ls -ldh /var/lib/mysql/

drwx------ 8 mysql mysql 4.0K 2012-11-27 14:21 /var/lib/mysql/

another thing is http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_method

does not mention about tmpfs?

Justin Swanhart

unread,

Nov 29, 2012, 4:31:56 PM11/29/12

to percona-d...@googlegroups.com

Hi,

Well, the tmpfs warning in the error log appears to be a red herring.
You aren't using tmpfs, just a filesystem that doesn't support direct
i/o. So don't use innodb_flush_method=O_DIRECT since your filesystem
doesn't support it.

The encrypted filesystem could be causing issues if IO isn't
completing in a reasonable amount of time (or sometimes never
completing)..

--Justin

Amol Kedar

unread,

Nov 29, 2012, 4:43:23 PM11/29/12

to percona-d...@googlegroups.com

Yes thanks justin,

I think it is related to the encrypted file system too..

As i have been testing the import in the past few days i saw some crash errors related to the ecryptfs on ubuntu...

something like

[92691.171350] ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-27]

[92691.171370] ecryptfs_writepage: Error encrypting page (upper index [0x00000000000014f4])

[92880.180090] INFO: task mysqld:9380 blocked for more than 120 seconds.

[92880.180106] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

yeah the next test is going to be, running the db in a unencrypted files mode...

Amol Kedar

unread,

Dec 1, 2012, 10:36:11 AM12/1/12

to percona-d...@googlegroups.com

This morning while testing some sql create and insert scripts on the cluster, node 1 and node 2 db crashed

I am attaching the error.log from node1 and node 2

Node 3 is still running fine. but i think if i try to bring up the other nodes node 3 might get stopped and i might have some downtime?

I have copied my.cnf as /etc/mysql/conf.d/mysqld_safe_syslog.cnf so that i can use xtrabackup (due to a certain bug in galera..?i guess)

in the error logs i see that the files

node 1

90 121201 9:25:26 [ERROR] Slave SQL: Could not execute Write_rows event on table mobiledb._data10958_th_ipatrol; Duplicate entry '20166' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1005, Error_code: 1062

91 121201 9:25:26 [Warning] WSREP: RBR event 2 Write_rows apply warning: 121, 2443703

92 121201 9:25:26 [ERROR] WSREP: receiving IST failed, node restart required: Failed to apply app buffer: @^SºP^S^C, seqno: 2443703, status: WSREP_FATAL

93 at galera/src/replicator_smm.cpp:apply_wscoll():49

94 at galera/src/replicator_smm.cpp:apply_trx_ws():120

node 2

53 121201 9:25:13 [ERROR] Slave SQL: Could not execute Write_rows event on table mobiledb._data10958_th_ipatrol; Duplicate entry '24182' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master

log FIRST, end_log_pos 1012, Error_code: 1062

54 121201 9:25:13 [Warning] WSREP: RBR event 2 Write_rows apply warning: 121, 2443704

55 121201 9:25:13 [ERROR] WSREP: receiving IST failed, node restart required: Failed to apply app buffer: A^SºP^S^C, seqno: 2443704, status: WSREP_FATAL

56 at galera/src/replicator_smm.cpp:apply_wscoll():49

57 at galera/src/replicator_smm.cpp:apply_trx_ws():120

is this the only reason for the crash? and what scenario could this happen?

another point to note is that i ha node 1 and node 3 as xtrabackup and node 2 as rsync (by mistake i left it like that)

# dpkg --list percona*

Desired=Unknown/Install/Remove/Purge/Hold

| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

||/ Name Version Description

+++-=====================================================-=====================================================-==========================================================================================================================