Percona xtrabackup plugin return code 3

[Steffen]

Hi,

i'm trying to setup the percona xtrabackup plugin but can't get it to work.

When i run the xtrabackup command manually on the client everything works, but when bareos is executing the command I get the following error:

client-fd JobId 43: Fatal error: python-fd: Dump command returned non-zero value: 3, command: "xtrabackup --backup --datadir=/srv/mysql/ --stream=xbstream --extra-lsndir=/srv/tmp/backup" message:

bareos-dir Version: 16.2.4 (01 July 2016) x86_64-pc-linux-gnu debian Debian GNU/Linux 8.0 (jessie) Debian_8.0 x86_64

Client OS: Debian Jessie
Client bareos version: 16.2.4 (01 July 2016)
Python 2.7.9
xtrabackup version 2.4.6 based on MySQL server 5.7.13 Linux (x86_64) (revision id: 54967d1)

My mysql datadir is located in /srv/mysql

My lsndir exists and has rwxrwxrwx rights.

I've also tried to change the data dir in https://github.com/bareos/bareos-contrib/blob/master/fd-plugins/bareos_percona/BareosFdPercona.py#L90 to run everything else with default settings, which results in exactly the same error. I also tried to set the stdErr to PIPE here ( https://github.com/bareos/bareos-contrib/blob/master/fd-plugins/bareos_percona/BareosFdPercona.py#L256 which didn't help ) as i hoped to get more details what's going on and included the stdOut in the error message. Which unfortunately didn't work. So i replaced the file again with the original.

Same results when I set the path to the my.cnf or not in my FileSet plugin module conf.

I went through those 3 threads even if they don't describe exactly my problem but hoped i get some more hints.

https://groups.google.com/forum/#!searchin/bareos-users/percona|sort:relevance/bareos-users/4cQoJ_d8HtA/WT0bKs4fDwAJ
https://groups.google.com/forum/#!searchin/bareos-users/percona|sort:relevance/bareos-users/eI5hZqVPVmE/ISHwlIiLDAAJ
https://groups.google.com/forum/#!searchin/bareos-users/percona|sort:relevance/bareos-users/aaWLMDxUbWU/l53-K773AQAJ

Attached is the trace file.

Any help is really appreciated, thank you in advance.

[Steffen]

Trace file attached

[Kjetil Torgrim Homme]

Steffen <steffen...@gmail.com> writes:

> i'm trying to setup the percona xtrabackup plugin but can't get it to work.
>
> When i run the xtrabackup command manually on the client everything
> works,

so you get a proper xbstream on stdout, and the exit status is 0?

> but when bareos is executing the command I get the following
> error:
>
> client-fd JobId 43: Fatal error: python-fd: Dump command returned
> non-zero value: 3, command: "xtrabackup --backup --datadir=/srv/mysql/
> --stream=xbstream --extra-lsndir=/srv/tmp/backup" message:

xtrabackup does not document what exit code 3 means. I think you need
to work more on how to reproduce the error outside Bareos. things to
try is to run xtrabackup in an empty environment (no $HOME etc.).

one thing I did to help debug xtrabackup was basically

mv xtrabackup xtrabackup.real

and make xtrabackup a shell script which does

#! /bin/sh
$0.real "$@" | tee /tmp/xtrabackup.output
ret=$?
echo "exit code $ret" >> /tmp/xtrabackup.exit
exit $ret

season to taste :-)

--
Kjetil T. Homme
Redpill Linpro AS - Changing the game

[Steffen Persch]

Thank you very much for your response.

The shellscript idea didn’t show me the actual problem but it helped me further debug everything.

I’ve found the error with changing https://github.com/bareos/bareos-contrib/blob/master/fd-plugins/bareos_percona/BareosFdPercona.py#L256 to:

log_fd = os.open("/tmp/xtra.log", os.O_WRONLY|os.O_CREAT)
self.stream = Popen(self.dumpcommand, shell=True, stdout=log_fd, stderr=log_fd)

Which finally gave me an error message to work with:

130523 13:33:54 InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.

The problem is that the ulimit for the process is set to 1024. I’ve increased the ulimit in the shell script and it seems to be working now.

I’m not sure where that limit comes from, i checked all my configs and I can’t find anything related to that number. All users have 65536 as limit, so maybe it’s something specific to bareos?

The next problem which occurs is that if you set a custom lsndir row 216 ( https://github.com/bareos/bareos-contrib/blob/master/fd-plugins/bareos_percona/BareosFdPercona.py#L216 ) will fail because it always looks for the xtrabackup_checkpoints file in the tmp folder. So i had to adjust this too.

The backup job completes now but i still need to do some more tests to confirm that everything is working as expected.

Thank’s again for your help.

Best,

Steffen

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/Z0UDn4Pb1ew/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.