Any luck with 17.2.4 bareos-storage-droplet package?

[Patrick McShane]

Hi,

We've setup an Amazon S3 bucket with nearly wide open permissions.

We supplied all the parameters as per documentation.

When we start a test backup job it seems to queue and then enter a "running" state for a long period of time with no obvious failures or error codes/messages.

After the job has been running for a few minutes, a subsequent check of messages from the bconsole CLI shows that a mount request exists for the S3 volume because the mount was not automatic.  We then honor the S3 volume mount request using console "mount" command only to see another mount request. 

Has anyone experienced this "mount request loop" when testing the bareos-storage-droplet S3 addon package?

Any general recommendations on what configuration options are causing this repeating mount request behavior when attempting to write to an S3 object store? 

Any recommendations on AWS S3 bucket attributes on the Amazon side that may fix this? 

Thanks, 

Pat 

[Jan Tymiński]

I have exactly the same issue.
I have turned on verbose monitoring on my S3 bucket which I am using for droplet and it is not even being hit by Bareos at all.
When I try the bucket with awscli or with S3FS, it works without any issues with exactly the same credentials, but it doesn't from S3 droplet.
AWS doesn't report any get/put/head requests, it doesn't return any 5xx nor 4xx errors.

I also tried adding "https://" in the bucket address, but then it returns:
17-May-2018 13:36:00.402878 17-May-2018 13:36:00.402918 bareos-sd (850): message.c:1129-3865 DIRECTOR for following msg: bareos-sd: ERROR in droplet_device.c:103 error: src/addrlist.c:638: dpl_addrlist_add: cannot lookup host https: Unknown host
17-May-2018 13:36:00.402979 bareos-storage-lon-sd (10): droplet_device.c:103-3865 ERROR: error: src/profile.c:293: conf_cb_func: error parsing address list
17-May-2018 13:36:00.403006 bareos-sd (850): message.c:858-3865 Enter dispatch_message type=4 msg=bareos-storage-lon-sd: ERROR in droplet_device.c:103 error: src/profile.c:293: conf_cb_func: error parsing address list
17-May-2018 13:36:00.403036 bareos-sd (850): message.c:1129-3865 DIRECTOR for following msg: bareos-sd: ERROR in droplet_device.c:103 error: src/profile.c:293: conf_cb_func: error parsing address list
17-May-2018 13:36:00.403096 bareos-sd (100): droplet_device.c:704-3865 droplet_device.c:703 Failed to create a new context using config profile=/etc/bareos/bareos-sd.d/device/droplet/droplet.profile,bucket=name-of-my-bucket,iothreads=1,ioslots=1,chunksize=100M

I have tried switching use_https to False too, but it didn't change anything, exactly the mount loop was happening.
This is what setdebug=999 for storage reports when the loop happens:
16-May-2018 09:55:49.133920 bareos-sd (100): droplet_device.c:292-0 Flushing chunk /S3DropletTest-Full-2886/0001
16-May-2018 09:55:49.158134 bareos-sd (100): chunked_device.c:413-0 Enqueueing chunk 1 of volume S3DropletTest-Full-2886 for retry of upload later
16-May-2018 09:55:49.166766 bareos-sd (100): chunked_device.c:413-0 Enqueueing chunk 0 of volume S3DropletTest-Full-2886 for retry of upload later
16-May-2018 09:59:15.203428 bareos-sd (100): chunked_device.c:398-0 Flushing chunk 0 of volume S3DropletTest-Full-2906 by thread 0x6100007f86617fa7
16-May-2018 09:59:15.203475 bareos-sd (100): droplet_device.c:292-0 Flushing chunk /S3DropletTest-Full-2906/0000
16-May-2018 09:59:15.231959 bareos-sd (100): chunked_device.c:398-0 Flushing chunk 1 of volume S3DropletTest-Full-2906 by thread 0x3900007f8660ff97
16-May-2018 09:59:15.231995 bareos-sd (100): droplet_device.c:292-0 Flushing chunk /S3DropletTest-Full-2906/0001
16-May-2018 09:59:15.271106 bareos-sd (100): chunked_device.c:413-0 Enqueueing chunk 0 of volume S3DropletTest-Full-2906 for retry of upload later
16-May-2018 09:59:15.306617 bareos-sd (100): chunked_device.c:413-0 Enqueueing chunk 1 of volume S3DropletTest-Full-2906 for retry of upload later

First I had used 5 threads with 5 slots and for 3 jobs in parallel, so I thought I have reached S3 bucket limits for requests, but then I changed to 1 thread with 1 slot and the issue remained exactly the same - this was the moment when I started to check AWS monitoring for buckets.

I'd love to move on with that.
If needed, I can provide obfuscated configs related to the droplet backend.

Best Regards,
Jan.

[Aron Schüler]

I had success by using following configuration, all on a fresh setup centos7:

$ cat bareos-sd.d/device/droplet/droplet.profile
use_https = false
host = <bucketname>.s3-eu-central-1.amazonaws.com
access_key = my-access-key
secret_key = my-private-key
pricing_dir = ""
backend = s3
aws_auth_sign_version = 4

$ cat bareos-sd.d/device/S3_ObjectStorage.conf
Device {
Name = S3_ObjectStorage
Media Type = s3bucket
Archive Device = S3 Object Storage

Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/droplet.profile,bucket=<bucketname>,iothreads=2,location=eu-central-1"
Device Type = droplet
LabelMedia = yes # lets Bareos label unlabeled media
Random Access = yes
AutomaticMount = yes # when device opened, read it
RemovableMedia = no
AlwaysOpen = no
Description = "S3 Object device. A connecting Director must have the same Name and MediaType."
Maximum File Size = 200000000 # 200 MB (Allows for seeking to small portions of the Volume)
}

$ cat bareos-dir.d/storage/s3bucket.conf
Storage {
Name = s3bucket
Address = centos7-droplet-test
Password = #####
Device = S3_ObjectStorage
Media Type = s3bucket
}

I had the same error @Jan had with his https configuration because i copied the whole url into the configuration. Now, I can write to the bucket, even though it is really slow (14 chunks with 300B each after 30 minutes of sd writing).

[Aron Schüler]

While the backup runs as expected but still slow, I cant get the restoring to work.
All I get is the following when trying to restore:

"Warning: acquire.c:296 Read acquire: Requested Volume "Incremental-0006" on "S3_ObjectStorage" (S3) is not a Bareos labeled Volume, because: ERR=block.c:1024 Read zero bytes at 0:0 on device "S3_ObjectStorage" (S3)."

[Rafael Castellar das Neves]

Hi Patrick!

I would like to answer this question, but I'm still trying to understand how to set my bareos with my S3. I heard about bareos-storage-droplet and others, but I'm confuse.. Is there any documentation or howto you could share with me?

By now, I just have my Bareos 17 (compiled from source) working good in my home network along with bareos-webui.

Tks!!