[[ Xpost from Bareos-users ]] Backup S3 to Tape, and restore

[Sean Sullivan]

Hello All!
I am brand new to bareos/bacula and tape. I've recently inherited a t50e and a server to backup our local S3 data to tape.

I am still going through the documentation on setting everything up and just finished writing the test backup of /usr/sbin to my Test001 volume successfully. I am looking around and I see many writeups offering to backup to S3//Amazon VTL but not from S3 to tape. Has anyone done this reliably? Does anyone have any resources or a direction to point me in. I have a feeling I have a long way to go.

I found an old post in bareos-devel that mentioned doing what I am trying to do ( https://groups.google.com/forum/?fromgroups#!searchin/bareos-devel/s3cmd%7Csort:relevance/bareos-devel/EkX36z3BtBk/tVM4ATsJCZQJ ). I am having issues with the first part of this though:
/usr/lib64/bareos/plugins/BareosFdPluginBaseclass.py in <module>()
25 # Functions taken and adapted from bareos-fd.py
26
---> 27 import bareosfd

ImportError: No module named bareosfd

I don't see this module on my system and I am not sure where it can be found. Any help would be appreciated.

[Stephan Dühr]

Hello Sean,

to backup S3 data, you already found out yourself the best way to go,
that is writing a filedaemon plugin using python.

As an introduction I'd suggest to work through the following slides from OSBConf 2015:
http://osbconf.org/wp-content/uploads/2015/10/Bareos-Python-Plugins-Hacking-Workshop.pdf

I don't know if Trygve Vea managed to finish his approach, would be nice to know.
Looks like he used code from S3cmd. I would suggest to look at boto3 instead:
https://boto3.readthedocs.io/en/latest/reference/services/s3.html
which should also be available packaged for common linux dists.

Regards,

Stephan

Just install the bareos-filedaemon-python-plugin package, that should fix this.

--
Stephan Dühr stepha...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-90
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf,
J. Steffens, Philipp Storz, M. v. Wieringen

[Sean Sullivan]

Thanks Stephan!

It turns out ( apparently according to someone on the #bareos irc room) that he did finish the plugin but he has yet to release it. I thought the same thing ( use boto3 ) and am trying to do that now. I found the mysql backup example on github ( https://github.com/bareos/bareos-contrib/tree/master/fd-plugins/mysql-python )

I'm relatively new to everything programming and python and I've seemed to hit a beginner snag. I can't seem to import the bareosfd module::

/usr/lib64/bareos/plugins/BareosFdPluginBaseclass.py in <module>()
25 # Functions taken and adapted from bareos-fd.py
26
---> 27 import bareosfd

I can't seem to find any bareosfd.so either. May I ask what I need to install to use this module? I'm running Centos7 but I can switch to any os or compile from github if need be.

Thanks again Stephan! Sorry for the trouble

[Stephan Dühr]

Please install the package bareos-filedaemon-python-plugin using
yum install bareos-filedaemon-python-plugin

That package contains python-fd.so, then
import bareosfd
will work when used within the python plugin.

Regards,

Stephan

>
>
> Thanks again Stephan! Sorry for the trouble
>

[Sean Sullivan]

Sorry for the late reply. I'm on CentOS7 with the bareos package installed::

[root@kg10-19 ~]# yum list installed | grep -iE "bareos"
bareos.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-bconsole.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-client.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-database-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-database-mysql.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-database-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-director.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-director-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-filedaemon.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-filedaemon-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 <<< The plugin is installed
bareos-storage.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-storage-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-storage-tape.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
bareos-webui.noarch 15.2.2-41.1.el7 @bareos_bareos-15.2
libfastlz.x86_64 0.1-7.1.el7 @bareos_bareos-15.2

still when I try to import bareosfd it fails::
[root@kg10-19 ~]# python -m bareosfd
/bin/python: No module named bareosfd

Is there something I am missing? Do I need to change my environment to something else prior to being able to import this?

In [1]: import os, sys

In [2]: print os.environ['PYTHONPATH']
:/usr/lib64/bareos/plugins/:/usr/lib64/bareos/

In [3]: import bareosfd
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-3-4bb231fac54f> in <module>()
----> 1 import bareosfd

ImportError: No module named bareosfd

Thank you so much for your help with this. I hope I am not the only one with these issues.

[Maik Aussendorf]

Hello Sean,

do you want to write or modify an existing python plugin? The bareos-fd
starts a Python interpreter and loads the modules in that context.

I recommend that you start with one of the simple example plugins like
BareosFdPluginLocalFileset.py (which is part of the rpm), try to get it
running and adapt it. Running FD in debug mode level >=100 gives you
some output about what the plugin is doing.

Regards
Maik

Am 29.09.2016 um 22:10 schrieb Sean Sullivan:
> Sorry for the late reply. I'm on CentOS7 with the bareos package installed::
>
> [root@kg10-19 ~]# yum list installed | grep -iE "bareos"
> bareos.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-bconsole.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-client.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-database-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.

> bareos-database-mysql.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-database-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-director.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-director-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-filedaemon.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-filedaemon-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 <<< The plugin is installed
> bareos-storage.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2

> bareos-storage-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2Hello Se

> bareos-storage-tape.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-webui.noarch 15.2.2-41.1.el7 @bareos_bareos-15.2
> libfastlz.x86_64 0.1-7.1.el7 @bareos_bareos-15.2
>
>
> still when I try to import bareosfd it fails::
> [root@kg10-19 ~]# python -m bareosfd
> /bin/python: No module named bareosfd
>
>
> Is there something I am missing? Do I need to change my environment to something else prior to being able to import this?
>
> In [1]: import os, sys
>
> In [2]: print os.environ['PYTHONPATH']
> :/usr/lib64/bareos/plugins/:/usr/lib64/bareos/
>
> In [3]: import bareosfd
> ---------------------------------------------------------------------------
> ImportError Traceback (most recent call last)
> <ipython-input-3-4bb231fac54f> in <module>()
> ----> 1 import bareosfd
>
> ImportError: No module named bareosfd
>
> Thank you so much for your help with this. I hope I am not the only one with these issues.
>

--
With kind regards // Mit freundlichen Grüßen
--
Maik Außendorf maik.au...@bareos.com
Bareos GmbH & Co. KG Phone: +49221630693-93
http://www.bareos.com Fax: +49221630693-10
**Open Source Backup Conf 2016 - September 26 to 27 - osbconf.org **

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH

Geschäftsführer: Stephan Dühr, M. Außendorf,
J. Steffens, P. Storz

[Stephan Dühr]

Hi Sean,

On 09/29/2016 10:10 PM, Sean Sullivan wrote:
> Sorry for the late reply. I'm on CentOS7 with the bareos package installed::
>
> [root@kg10-19 ~]# yum list installed | grep -iE "bareos"
> bareos.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-bconsole.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-client.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-database-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-database-mysql.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-database-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-director.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-director-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-filedaemon.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-filedaemon-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 <<< The plugin is installed
> bareos-storage.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-storage-python-plugin.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-storage-tape.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2
> bareos-webui.noarch 15.2.2-41.1.el7 @bareos_bareos-15.2
> libfastlz.x86_64 0.1-7.1.el7 @bareos_bareos-15.2
>
>
> still when I try to import bareosfd it fails::
> [root@kg10-19 ~]# python -m bareosfd
> /bin/python: No module named bareosfd

This won't work, a Bareos fd python plugin can't be run with the normal python interpreter.
As Maik already said, it's the bareos-fd process that invokes the interpreter and only
then it's possible to
import bareosfd

It's really best to take the local fileset demo plugin as a template to get started, eg.

cd /usr/lib64/bareos/plugins
cp bareos-fd-local-fileset.py bareos-fd-plugin-s3.py
cp BareosFdPluginLocalFileset.py BareosFdPluginS3.py

Did you read
http://osbconf.org/wp-content/uploads/2015/10/Bareos-Python-Plugins-Hacking-Workshop.pdf ?

If it leaves open questions, please ask.

Do you think it would be helpful to have some more detailed step-by-step how-to
for getting started with Bareos python plugin development?

Regards,

Stephan

>
>
> Is there something I am missing? Do I need to change my environment to something else prior to being able to import this?
>
> In [1]: import os, sys
>
> In [2]: print os.environ['PYTHONPATH']
> :/usr/lib64/bareos/plugins/:/usr/lib64/bareos/
>
> In [3]: import bareosfd
> ---------------------------------------------------------------------------
> ImportError Traceback (most recent call last)
> <ipython-input-3-4bb231fac54f> in <module>()
> ----> 1 import bareosfd
>
> ImportError: No module named bareosfd
>
> Thank you so much for your help with this. I hope I am not the only one with these issues.
>

[Sean Sullivan]

I finally made this. It's junk honestly but it seems to successfully back up s3 objects. I took the mysql plugin and just smashed an s3 downloader into it.

https://github.com/lookcrabs/bareos-contrib/tree/master/fd-plugins/s3-python

It's not the best but its simple and seems to work. If anyone could help me fix this up or with any tips to make this work better I would be very appreciative.

Mine uses boto3 so it works with AWS s3, Ceph, Cleversave and hopefully any S3 like storage (minio etc). You just need to adjust the signature for awsv4 if you want the newer style otherwise it should work fine as is.

Currently, I can't get estimates to work properly but backups/restores seem to work.

I'm trying to find another plugin that has estimates working so I can more or less steal and adjust their bit. Any help would be greatly appreciated.