Restore xtrabackup No lsn information found

[George Kontostanos]

Hello,

I am trying to restore a successful backup using xtrabackup.

The web UI shows the size of the backup as NaN undefined. I am not sure if this is normal:

https://s1.postimg.org/3stincso73/Screen_Shot_2017-10-06_at_12.58.39.png

When the restore finishes I get the following error:

dir-fd JobId 2933: Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000002899 from job 2899

I tried it again in debug mode like this:

*setdebug client=my.client trace=on level=200
2000 OK setdebug=200 trace=0 hangup=0 timestamp=0 tracefile=/var/lib/bareos/dir-fd.trace

However, it does not seem to create a trace file.

Dir and FD run CentOS 7.4 Bareos Version: 16.2.4

Any idea on how to troubleshoot this further?

Thanks

George

[Maik Aussendorf]

Hi,

Am 06.10.2017 um 13:02 schrieb George Kontostanos:
> Hello,
>
> I am trying to restore a successful backup using xtrabackup.
>
> The web UI shows the size of the backup as NaN undefined. I am not sure if this is normal:
>
> https://s1.postimg.org/3stincso73/Screen_Shot_2017-10-06_at_12.58.39.png
>
> When the restore finishes I get the following error:
>
> dir-fd JobId 2933: Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000002899 from job 2899
>
> I tried it again in debug mode like this:
>
> *setdebug client=my.client trace=on level=200

it must be trace=1 (not trace=on).

Then check on backup, if the LSN was found.

Regards
Maik

> 2000 OK setdebug=200 trace=0 hangup=0 timestamp=0 tracefile=/var/lib/bareos/dir-fd.trace
>
> However, it does not seem to create a trace file.
>
> Dir and FD run CentOS 7.4 Bareos Version: 16.2.4
>
> Any idea on how to troubleshoot this further?
>
> Thanks
>
> George
>
>
>

--
With kind regards // Mit freundlichen Grüßen
--
Open Source Backup Conference Sept. 25-26 2017 http://osbconf.org
---
Maik Außendorf maik.au...@bareos.com
Bareos GmbH & Co. KG Phone: +49221630693-93
http://www.bareos.com Fax: +49221630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, M. Außendorf,
J. Steffens, P. Storz

[George Kontostanos]

On Friday, October 6, 2017 at 3:07:01 PM UTC+3, maik.aussendorf wrote:
> Hi,
>
Hi Mike

> Am 06.10.2017 um 13:02 schrieb George Kontostanos:
> > Hello,
> >
> > I am trying to restore a successful backup using xtrabackup.
> >
> > The web UI shows the size of the backup as NaN undefined. I am not sure if this is normal:
> >
> > https://s1.postimg.org/3stincso73/Screen_Shot_2017-10-06_at_12.58.39.png
> >
> > When the restore finishes I get the following error:
> >
> > dir-fd JobId 2933: Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000002899 from job 2899
> >
> > I tried it again in debug mode like this:
> >
> > *setdebug client=my.client trace=on level=200
> it must be trace=1 (not trace=on).

Thanks, this time I got the trace :(https://drive.google.com/file/d/0B6SfXxoaUYIVRU9laUlEbmF0cmM/view?usp=sharing)

> Then check on backup, if the LSN was found.

Same error again: dir-fd JobId 2937: Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000002899 from job 2899

> Regards
> Maik
>
> > 2000 OK setdebug=200 trace=0 hangup=0 timestamp=0 tracefile=/var/lib/bareos/dir-fd.trace
> >
> > However, it does not seem to create a trace file.
> >
> > Dir and FD run CentOS 7.4 Bareos Version: 16.2.4
> >
> > Any idea on how to troubleshoot this further?
> >
> > Thanks
> >
> > George
> >
> >
> >
>
> --
> With kind regards // Mit freundlichen Grüßen
> --
> Open Source Backup Conference Sept. 25-26 2017 http://osbconf.org
> ---
> Maik Außendorf maik.au...@bareos.com
> Bareos GmbH & Co. KG Phone: +49221630693-93
> http://www.bareos.com Fax: +49221630693-10
>
> Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
> Komplementär: Bareos Verwaltungs-GmbH
> Geschäftsführer: Stephan Dühr, M. Außendorf,
> J. Steffens, P. Storz

Regards

George

[Maik Aussendorf]

Hi George,

please also post a trace of the backup job. I assume something has gone
wrong with the original backup. Please use the latest plugin version
from Git, as it includes a better way to get the LSN and logging.

I am also wondering what you try to do with

python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mysqluser=bareos:mysqlpassword=wae5Vi5shai

the options mysqluser and mysqlpassowrd, that are no valid options for
the plugin. You need to place credentials using the my.cnf mechanism.

https://github.com/bareos/bareos-contrib/tree/master/fd-plugins/bareos_percona

Regards

Maik

[George Kontostanos]

On Friday, October 6, 2017 at 6:03:16 PM UTC+3, maik.aussendorf wrote:
> Hi George,
>
> please also post a trace of the backup job. I assume something has gone
> wrong with the original backup. Please use the latest plugin version
> from Git, as it includes a better way to get the LSN and logging.

Hi Mike, I am actually using the plugins from git.

>
> I am also wondering what you try to do with
>
> python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mysqluser=bareos:mysqlpassword=wae5Vi5shai
>
> the options mysqluser and mysqlpassowrd, that are no valid options for
> the plugin. You need to place credentials using the my.cnf mechanism.

I understand however this is the way that the mysqldump plugin works. I would prefer to avoid having passwords in my.cnf

I will try it and let you know.
>
> https://github.com/bareos/bareos-contrib/tree/master/fd-plugins/bareos_percona
>
> Regards
>
> Maik

Thanks

George

[George Kontostanos]

Hi Mike,

I have tried it and I am getting now the following error when try to backup:

dir-fd JobId 2986: Fatal error: python-fd: Dump command returned non-zero value: 1, command: "xtrabackup /etc/my.cnf--backup --stream=xbstream --extra-lsndir=/tmp/tmpC8aCfq" message:

my.cnf:

[mysql]
user=bkpuser
password=s3cret

FileSet {
Name = "client-name"
Include {
Options {
compression=GZIP
signature = MD5
}
Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mycnf=/etc/my.cnf"
}
}

If I run the following command in the FD the backup works:

xtrabackup --user=bkpuser --password=s3cret --backup --target-dir=/tmp

I could not get a trace

Any thoughts?

Thanks

George

[Jörg Steffens]

isn't there a space missing between "/etc/my.cnf" and "--backup"? Or is
this a copy and paste mistake?

--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:

S. Dühr, M. Außendorf, Jörg Steffens, P. Storz

[George Kontostanos]

Indeed there is a space missing here:

dir-fd JobId 3009: Fatal error: python-fd: Dump command returned non-zero value: 1, command: "xtrabackup /etc/my.cnf--backup --stream=xbstream --extra-lsndir=/tmp/tmpenhtus" message: 

The command is the FileSet is:

Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mycnf=/etc/my.cnf"

Do you think I should try adding a space like: Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mycnf=/etc/my.cnf " ?

Nice catch 

George

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/7pwz9oT0ANE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jörg Steffens]

On 07.10.2017 at 16:25 wrote George Kontostanos:
> Indeed there is a space missing here:
>
> dir-fd JobId 3009: Fatal error: python-fd: Dump command returned
> non-zero value: 1, command: "xtrabackup /etc/my.cnf--backup
> --stream=xbstream --extra-lsndir=/tmp/tmpenhtus" message: 
>
> The command is the FileSet is:
>
> Plugin =
> "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mycnf=/etc/my.cnf"
>
> Do you think I should try adding a space like: Plugin =
> "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-percona:mycnf=/etc/my.cnf
> " ?

Properly this will not help. I assume a bug introduced with the latest
modifications. I assume, it will be fixed soon.

Jörg

[George Kontostanos]

It did not :) I tried it and got the same error again.

Thanks

George

[Maik Aussendorf]

Hi,

the mycnf parameter was introduced with one of the latest pull-requests, and we haven't tested all option combinations. Should work now.

In case of problems: get the xtrabackup command from the joblog and make sure it works, when you start it manually the same way.

Regards

Maik

You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.

To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
With kind regards // Mit freundlichen Grüßen
---
 Maik Außendorf                             maik.au...@bareos.com

 Bareos GmbH & Co. KG                       Phone: +49221630693-93
 http://www.bareos.com                      Fax: +49221630693-10

 Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
 Komplementär: Bareos Verwaltungs-GmbH
 Geschäftsführer: Stephan Dühr, M. Außendorf, 
 J. Steffens, P. Storz

[George Kontostanos]

Hi Maik,

Thanks for the update. This time I managed to finish the backup. However, the restore produced the same error:

Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000003091 from job 3091

Please see a new trace here: https://drive.google.com/a/netactuate.com/file/d/0B6SfXxoaUYIVVmhLUXRmVHRTWWc/view?usp=sharing

Best Regards

George

[George Kontostanos]

Just some more info of that helps. Although the restore returns an error, it appears that there is a full restore in /tmp/bareos-restores/_percona/3111

Inside that directory I can see all mysql data plus the following:

xtrabackup_checkpoints

xtrabackup_info

xtrabackup_logfile:

uuid = e73bedbe-acc7-11e7-841e-00163e7b9c3e

name = 

tool_name = xtrabackup

tool_command = --defaults-extra-file=/etc/my.cnf --backup --stream=xbstream --extra-lsndir=/tmp/tmpV_6Toz

tool_version = 2.4.8

ibbackup_version = 2.4.8

server_version = 5.6.37-82.2

start_time = 2017-10-09 03:15:34

end_time = 2017-10-09 04:00:23

lock_time = 0

binlog_pos = 

innodb_from_lsn = 0

innodb_to_lsn = 43699025276

partial = N

incremental = N

format = xbstream

compact = N

compressed = N

encrypted = N

Which looks valid to me.

Thanks

George

[Maik Aussendorf]

Hello,

please post the joblogs _and_ debug-logs of _both_ the backup job _and_ the restore job.

Regards
Maik

[George Kontostanos]

Hi Maik,

The job logs:

2017-10-09 08:26:37 bareos-dir JobId 3091: Bareos bareos-dir 16.2.4 (01Jul16):

Build OS: x86_64-redhat-linux-gnu redhat CentOS Linux release 7.0.1406 (Core) 

JobId: 3091

Job: server.tld-sql-job01.2017-10-09_07.41.15_53

Backup Level: Full

Client: "server.tld" 16.2.4 (01Jul16) x86_64-redhat-linux-gnu,redhat,CentOS Linux release 7.0.1406 (Core) ,CentOS_7,x86_64

FileSet: "server.tld-data" 2017-10-07 09:01:01

Pool: "server.tld-fd-sql-pool01" (From command line)

Catalog: "MyCatalog" (From Client resource)

Storage: "ams-bu-1" (From Pool resource)

Scheduled time: 09-Oct-2017 07:41:15

Start time: 09-Oct-2017 07:41:20

End time: 09-Oct-2017 08:26:37

Elapsed time: 45 mins 17 secs

Priority: 10

FD Files Written: 2

SD Files Written: 2

FD Bytes Written: 10,062,426,869 (10.06 GB)

SD Bytes Written: 10,062,427,477 (10.06 GB)

Rate: 3703.5 KB/s

Software Compression: 73.7 % (gzip)

VSS: no

Encryption: no

Accurate: no

Volume name(s): server.tld-fd-sql-pool01-2917|server.tld-fd-sql-pool01-3142

Volume Session Id: 142

Volume Session Time: 1502825051

Last Volume Bytes: 9,412,787,248 (9.412 GB)

Non-fatal FD errors: 0

SD Errors: 0

FD termination status: OK

SD termination status: OK

Termination: Backup OK

2017-10-09 08:26:37 bareos-dir JobId 3091: Begin pruning Jobs older than 1 month 15 days .

2017-10-09 08:26:37 bareos-dir JobId 3091: No Jobs found to prune.

2017-10-09 08:26:37 bareos-dir JobId 3091: Begin pruning Files.

2017-10-09 08:26:37 bareos-dir JobId 3091: No Files found to prune.

2017-10-09 08:26:37 bareos-dir JobId 3091: End auto prune.

2017-10-09 08:26:36 bareos-sd JobId 3091: Elapsed time=00:45:13, Transfer rate=3.708 M Bytes/second

2017-10-09 08:00:23 dir-fd JobId 3091: python-fd: Starting backup of /_percona/xtrabackup_checkpoints

2017-10-09 07:43:54 bareos-sd JobId 3091: New volume "server.tld-fd-sql-pool01-3142" mounted on device "FileStorage1" (/datastore/dir2) at 09-Oct-2017 07:43.

2017-10-09 07:43:53 bareos-dir JobId 3091: Created new Volume "server.tld-fd-sql-pool01-3142" in catalog.

2017-10-09 07:43:53 bareos-sd JobId 3091: Labeled new Volume "server.tld-fd-sql-pool01-3142" on device "FileStorage1" (/datastore/dir2).

2017-10-09 07:43:53 bareos-sd JobId 3091: Wrote label to prelabeled Volume "server.tld-fd-sql-pool01-3142" on device "FileStorage1" (/datastore/dir2)

2017-10-09 07:43:52 bareos-sd JobId 3091: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage1" (/datastore/dir2).

2017-10-09 07:43:52 bareos-sd JobId 3091: End of medium on Volume "server.tld-fd-sql-pool01-2917" Bytes=10,737,356,084 Blocks=166,440 at 09-Oct-2017 07:43.

2017-10-09 07:41:20 bareos-sd JobId 3091: Volume "server.tld-fd-sql-pool01-2917" previously written, moving to end of data.

2017-10-09 07:41:20 bareos-sd JobId 3091: Warning: For Volume "server.tld-fd-sql-pool01-2917":

The sizes do not match! Volume=10075075915 Catalog=10075075115

Correcting Catalog

2017-10-09 07:41:19 bareos-dir JobId 3091: Using Device "FileStorage1" to write.

2017-10-09 07:41:17 bareos-dir JobId 3091: Start Backup JobId 3091, Job=server.tld-sql-job01.2017-10-09_07.41.15_53

2017-10-09 07:15:34 dir-fd JobId 3091: python-fd: Starting backup of /_percona/xbstream.0000003091

----------

2017-10-09 09:48:21 bareos-dir JobId 3111: Error: Bareos bareos-dir 16.2.4 (01Jul16):

Build OS: x86_64-redhat-linux-gnu redhat CentOS Linux release 7.0.1406 (Core) 

JobId: 3111

Job: RestoreFiles.2017-10-09_09.28.43_42

Restore Client: server.tld

Start time: 09-Oct-2017 09:28:45

End time: 09-Oct-2017 09:48:21

Elapsed time: 19 mins 36 secs

Files Expected: 1

Files Restored: 1

Bytes Restored: 38,327,618,793

Rate: 32591.5 KB/s

FD Errors: 1

FD termination status: Error

SD termination status: OK

Termination: *** Restore Error ***

2017-10-09 09:48:21 bareos-dir JobId 3111: Begin pruning Files.

2017-10-09 09:48:21 bareos-dir JobId 3111: No Files found to prune.

2017-10-09 09:48:21 bareos-dir JobId 3111: End auto prune.

2017-10-09 09:48:16 bareos-sd JobId 3111: End of Volume at file 2 on device "FileStorage1" (/datastore/dir2), Volume "server.tld-fd-sql-pool01-3142"

2017-10-09 09:48:16 bareos-sd JobId 3111: End of all volumes.

2017-10-09 09:30:13 bareos-sd JobId 3111: End of Volume at file 2 on device "FileStorage1" (/datastore/dir2), Volume "server.tld-fd-sql-pool01-2917"

2017-10-09 09:30:13 bareos-sd JobId 3111: Ready to read from volume "server.tld-fd-sql-pool01-3142" on device "FileStorage1" (/datastore/dir2).

2017-10-09 09:30:13 bareos-sd JobId 3111: Forward spacing Volume "server.tld-fd-sql-pool01-3142" to file:block 0:252.

2017-10-09 09:28:47 bareos-sd JobId 3111: Ready to read from volume "server.tld-fd-sql-pool01-2917" on device "FileStorage1" (/datastore/dir2).

2017-10-09 09:28:47 bareos-sd JobId 3111: Forward spacing Volume "server.tld-fd-sql-pool01-2917" to file:block 2:1485141323.

2017-10-09 09:28:46 bareos-dir JobId 3111: Using Device "FileStorage1" to read.

2017-10-09 09:28:45 bareos-dir JobId 3111: Start Restore Job RestoreFiles.2017-10-09_09.28.43_42

2017-10-09 09:02:01 dir-fd JobId 3111: Error: python-fd: No lsn information found in restore object for file /tmp/bareos-restores//_percona/xbstream.0000003091 from job 3091

I am not sure how to get the debug-logs

Thanks

George

[Maik Aussendorf]

Hi,

just as you did for restore: create a tracefile run backup job.

You've also mentioned a file xtrabackup_checkpoints in your restore, please send the contents. The plugin grabs the LSN from that file after having called xtrabackup for creating the backup files..

Regards

Maik

[George Kontostanos]

Hi Maik,

I will install a new smaller database so that the traces don't get so huge. 
I will keep you posted

Thanks

George