Backup data to AWS S3 bucket using the BareOS utility
983 views
Skip to first unread message
Kaushal Shriyan
Mar 16, 2021, 12:02:09 PM3/16/21
to bareos-users
Hi,
Is there a way to push the backup data to the AWS S3 bucket using the BareOS utility? For example, if I backup both configurations and data directory of GitLab SCM services using BareOS utility, Can it be pushed to AWS S3 bucket instead of storing it locally in the BareOS server.
Please advise. Thanks in advance.
Best Regards,
Kaushal
Spadajspadaj
Mar 17, 2021, 3:28:20 AM3/17/21
to bareos...@googlegroups.com
Yes, there is a plugin for storage daemon to store data in S3.
See the docs.
--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/655c1c45-4a5e-47d3-9895-7c87f47a0af6n%40googlegroups.com.
Kaushal Shriyan
Mar 23, 2021, 6:33:48 AM3/23/21
to bareos-users
Hi,
I am running bareos 20.0 on CentOS Linux release 7.9.2009 (Core). I have installed bareos 20.0 using https://download.bareos.org/bareos/release/20/CentOS_7/x86_64/ I am trying to push the backup to AWS S3 by following the document https://docs.bareos.org/TasksAndConcepts/StorageBackends.html. When I execute the run process I am encountering
23-Mar 10:28 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 10:28 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
#cat /etc/bareos/bareos-sd.d/device/droplet/aws.profile
# Generic host, but can't access buckets younger than 24h:
# Region specific host name. Can access also new buckets.
host="s3.amazonaws.com"
use_https = "true"
backend = "s3"
aws_region = "ap-south-1"
aws_auth_sign_version = "4"
access_key = "XXXXXXXXXXXXXXXXXXX"
secret_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
pricing_dir = ""
cat /etc/bareos/bareos-sd.d/device/AWS_S3_1-00.conf
Device {
Name = "AWS_S3_1-00"
Media Type = "S3_Object1"
Archive Device = "AWS S3 Storage"
Device Type = droplet
Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/aws.profile,bucket=droplet-bareos,chunksize=100M"
Label Media = yes # Lets Bareos label unlabeled media
Random Access = yes
Automatic Mount = yes # When device opened, read it
Removable Media = no
Always Open = no
Maximum Concurrent Jobs = 1
}
cat /etc/bareos/bareos-dir.d/storage/S3_Object.conf
Storage {
Name = "S3_Object"
Address = "krithilinux" # N.B. Use a fully qualified name here (do not use "localhost" here).
Password = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
Maximum Concurrent Jobs = 1
Collect Statistics = yes
Allow Compression = Yes
Device = "AWS_S3_1-00"
Media Type = "S3_Object1"
}
#rpm -qa |grep bareos
bareos-database-postgresql-20.0.1-3.el7.x86_64
bareos-client-20.0.1-3.el7.x86_64
bareos-storage-droplet-20.0.1-3.el7.x86_64
bareos-common-20.0.1-3.el7.x86_64
bareos-database-common-20.0.1-3.el7.x86_64
bareos-database-tools-20.0.1-3.el7.x86_64
bareos-filedaemon-20.0.1-3.el7.x86_64
bareos-tools-20.0.1-3.el7.x86_64
bareos-20.0.1-3.el7.x86_64
bareos-webui-20.0.1-3.el7.x86_64
bareos-bconsole-20.0.1-3.el7.x86_64
bareos-director-20.0.1-3.el7.x86_64
bareos-storage-20.0.1-3.el7.x86_64
Please let me know if anyone needs any additional information. Thanks in Advance.
Best Regards,
Kaushal
Kaushal Shriyan
Mar 23, 2021, 7:14:27 AM3/23/21
to bareos-users
Hi,
I am attaching the screenshot and the bareos logs for your reference.
==> bareos.log <==
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd JobId 44: Warning: stored/label.cc:389 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/Full-0003, ERR=Success
23-Mar 16:42 bareos-sd JobId 44: Warning: stored/mount.cc:276 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/Full-0003, ERR=Success
23-Mar 16:42 apigeeapicrafterprodbackup-fd JobId 44: Fatal error: filed/dir_cmd.cc:2697 Bad response to Append Data command. Wanted 3000 OK data
, got 3903 Error append data
23-Mar 16:42 bareos-dir JobId 44: Fatal error: Director's comm line to SD dropped.
23-Mar 16:42 bareos-dir JobId 44: Error: Bareos bareos-dir 20.0.1 (02Mar21):
Build OS: CentOS Linux release 7.6.1810 (Core)
JobId: 44
Job: apigeebackup.2021-03-23_16.17.19_08
Backup Level: Full
Client: "apigeeapicrafterprodbackup" 20.0.0 (16Dec20) CentOS Linux release 7.6.1810 (Core),redhat
FileSet: "LinuxConfig" 2021-03-22 19:26:06
Pool: "Full" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "S3_Object" (From Job resource)
Scheduled time: 23-Mar-2021 16:17:19
Start time: 23-Mar-2021 16:17:21
End time: 23-Mar-2021 16:42:58
Elapsed time: 25 mins 37 secs
Priority: 10
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s):
Volume Session Id: 1
Volume Session Time: 1616496336
Last Volume Bytes: 0 (0 B)
Non-fatal FD errors: 1
SD Errors: 0
FD termination status: Fatal Error
SD termination status: Error
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: *** Backup Error ***
Best Regards,
Kaushal
Kaushal Shriyan
Mar 25, 2021, 11:11:32 PM3/25/21
to bareos-users
Hi,
Checking in again if someone can pitch in for my earlier post to the bareos-users google group. Thanks in advance. I look forward to hearing from you.
Best Regards,
Kaushal
Larry G. Wapnitsky
Mar 27, 2021, 7:48:41 PM3/27/21
to bareos-users
Curious about this as well. Haven't used BareOS in a few years, but want to set it up in my homelab with Wasabi, and am running into the same errors.
Kaushal Shriyan
Mar 28, 2021, 1:27:37 AM3/28/21
to Larry G. Wapnitsky, bareos-users
On Sun, Mar 28, 2021 at 5:18 AM Larry G. Wapnitsky <la...@wapnitsky.com> wrote:
Curious about this as well. Haven't used BareOS in a few years, but want to set it up in my homelab with Wasabi, and am running into the same errors.
Hi,
I have already raised a bug report in https://bugs.bareos.org/view.php?id=1331. Thanks in Advance.
Best Regards,
Kaushal Shriyan
Mar 30, 2021, 1:18:03 PM3/30/21
to bareos-users
Hi,
Checking in again if someone can pitch in for my earlier post to the bareos-users google group. I have already raised a bug report in https://bugs.bareos.org/view.php?id=1331. Thanks in advance. I look forward to hearing from you.
Best Regards,
Kaushal
Yavor Marinov
Mar 30, 2021, 3:21:53 PM3/30/21
to Kaushal Shriyan, bareos-users
Hello,
on totally unrelated topic it's not BareOS, but simple Bareos. Had this issues on some other apps and i can confirm that the issue was caused because of SSL_ERROR_SYSCALL along with the wsa error_code 10054 which both says that the connection has been terminated by the remote host - also, had experience with old curl / certificates / javastores packages which could be possible reason, just because of the deprecated or old or whatever functions. Overall, i don't think this could be set as bug, just because simple firewall, or NAT-ed VM/baremetal could cause this issue.
Strongly suggesting you to turn on some debugging and see what's exactly makes the remote host to drop your connection.
~br
--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/16eabca4-8f95-4417-8353-909bcffb6fdan%40googlegroups.com.
Yavor Marinov
Mar 31, 2021, 3:55:32 PM3/31/21
to Kaushal Shriyan, bareos-users
All i was able to locate in bareos-dir.trace was just evidence that 99% it's an ssl issue:
* if you are using any self-signed certificate on Bareos (highly suggest you start using it)
* i saw that your Bareos director is called on few lines with >>> hostname <<< which is not recommended
* if you are using internal/NAT-ed network, > ensure < that every other module of network/iptables is loaded
you can check if your Bareos has enabled SSL by simply using bconsole and it should produce something liket this:
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 20.0.0 (16 December 2020)
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 20.0.0 (16 December 2020)
I can think of only one other thing that you might missing - postgresql - check SSL, try to login with user/pass from localhost using bareos credentials. everything from top to bottom.
Not sure how many installation you made either Bareos or Bacula, but having such issues is step one of learning "what's exactly is Bareos" - the config structure, the definitions of file-daemon, storage-daemon and director
~br
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/dbdceb53-636e-4836-b248-1ed5726f8d19n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages