Problems backing up ESXI 6.5 vms
Egoitz Aurrekoetxea
Good afternoon,
Just to let you know, as I was slightly aware, it fails mates...
I do attach the job log with vadp_dumper in verbose mode working
for the fd plugin of Vmware. I'll try to solve it but if someones
knows how could
be solved please tell me as I will be very thankful for that. I
should say I tested this too, with 17.2.3 although this attached
test is from 16.2.3. Have reproduced it too by changing vddk
libraries by the
5.5.5 version too....
Best regards,
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
Egoitz Aurrekoetxea
It seems the problem to be :
Warning: HostAgent is not a VirtualCenter, cannot disable
sVmotion. Error 20 at 2530.
Log: VixDiskLibVim: Failed to load license manager.
Failed to Prepare For Access: The command is not supported on
remote objects [20]
Log: VixDiskLib: VixDiskLib_ConnectEx: Establish connection using
(null).
Log: VixDiskLib: VixDiskLib_Connect: Establish connection.
Log: 2017-09-25T17:12:39.044+02:00 [7F2B8E730700 warning 'Libs']
SSL_VerifyCbHelper: Certificate verification is disabled, so
connection will proceed despite the error
Log: 2017-09-25T17:12:39.044+02:00 [7F2B8E730700 warning 'Libs']
SSL_VerifyCbHelper: Certificate verification is disabled, so
connection will proceed despite the error
Log: 2017-09-25T17:12:39.044+02:00 [7F2B8E730700 warning 'Libs']
SSL_VerifyCbHelper: Certificate verification is disabled, so
connection will proceed despite the error
Log: 2017-09-25T17:12:39.058+02:00 [7F2B96F7D740 warning
'Default'] Closing Response processing in unexpected state: 3
Log: 2017-09-25T17:12:39.062+02:00 [7F2B96DE8700 warning 'Libs']
SSL_VerifyCbHelper: Certificate verification is disabled, so
connection will proceed despite the error
They say that is fixed in 5.5.4 vddk....
root@sareveam bareos]# rpm -qa | grep -i disklib
bareos-vmware-vix-disklib-5.5.4_2454786-1.1.el7.x86_64
So... this should not be the issue... although as said before have "exchanged" momentaneously the libraries by the 5.5.5 ones and a fd restart but no successs....
Cheers!
Stephan Duehr
I don't know if VDDK 5.5.4 is supposed to work with vSphere 6.5, may be bareos_vadp_dumper must
be adapted and built against VDDK 6.x to get it working with vSphere 6.5.
There are some more errors, like
Log: Cnx_Connect: Error message:
Warning: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error:
Log: NBD_ClientOpen: Couldn't connect to 172.16.8.125:902
Log: DISKLIB-DSCPTR: : "ha-nfcssl://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : Failed to open NBD extent.
Log: DISKLIB-LINK : "ha-nfcssl://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : failed to open
(NBD_ERR_NETWORK_CONNECT).
Log: DISKLIB-CHAIN : "ha-nfcssl://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : failed to open
(NBD_ERR_NETWORK_CONNECT).
Log: DISKLIB-LIB : Failed to open 'ha-nfcssl://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902' with flags 0x1e
NBD_ERR_NETWORK_CONNECT (2338).
Log: NBD_ClientOpen: attempting to create connection to ha-nfc://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902
Warning: SSL: connect failed
Log: CnxAuthdConnect: Returning false because SSL_ConnectAndVerify failed
Log: CnxConnectAuthd: Returning false because CnxAuthdConnect failed
Log: Cnx_Connect: Returning false because CnxConnectAuthd failed
Log: Cnx_Connect: Error message:
Warning: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error:
Log: NBD_ClientOpen: Couldn't connect to 172.16.8.125:902
Log: DISKLIB-DSCPTR: : "ha-nfc://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : Failed to open NBD extent.
Log: DISKLIB-LINK : "ha-nfc://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : failed to open
(NBD_ERR_NETWORK_CONNECT).
Log: DISKLIB-CHAIN : "ha-nfc://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902" : failed to open
(NBD_ERR_NETWORK_CONNECT).
Log: DISKLIB-LIB : Failed to open 'ha-nfc://[datastore1] DEBIAN91/DEBIAN9...@172.16.8.125:902' with flags 0x1e
NBD_ERR_NETWORK_CONNECT (2338).
Failed to open [datastore1] DEBIAN91/DEBIAN91_0.vmdk : You do not have access rights to this file [13]
May be it's a SSL cert issue, and 6.5 does not allow NBD acccess without SSL any more? Don't know.
Sometimes VMware error messages can be confusing and misleading.
Does the FD run on a VM that has access to the Datastore or on a separate system?
From the logs, it looks like it's trying different access methods, but all fail.
If it's a VM, it should be able to use the hotadd method automatically, which should
not require any SSL stuff.
Regards,
Stephan
>> *Egoitz Aurrekoetxea*
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> www.sarenet.es <http://www.sarenet.es>
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
> --
>
>
> *Egoitz Aurrekoetxea*
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> www.sarenet.es <http://www.sarenet.es>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
> --
> To unsubscribe from this group and stop receiving emails from it, send an email to
> bareos-users...@googlegroups.com <mailto:bareos-users...@googlegroups.com>.
> To post to this group, send email to bareos...@googlegroups.com <mailto:bareos...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
--
Stephan Dühr stepha...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-90
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz
Egoitz Aurrekoetxea
Good afternoon,
Apologies for a so late answering. How did you adapted and build it?. You mean, you installed vddk 6.x and then linked against it?. Did you modified the code of Bareos, some part in order for it to work with ESXI 6.5?.
Best regards,
Stephan Duehr
in the current state, the Bareos VMware Plugin is using VDDK 5.5.4.
As of the release notes
https://vdc-download.vmware.com/vmwb-repository/dcr-public/682a33d4-4139-46dd-8432-fbcafb52f8da/13ec58e7-65dd-4185-9ccb-ad01f286d187/vddk-550-releasenotes.html
it is not compatible with vSphere 6.5:
"The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other
words, VDDK 5.5 and all of its update releases support vSphere 5.0, 5.1, and 6.0 (except new features)."
we are currently working on moving to VDDK 6.5 for the upcoming Bareos version 17.2, as of release notes
https://vdc-download.vmware.com/vmwb-repository/dcr-public/ab4ca212-278a-4221-a4ce-4718dc9370d0/da5d8973-14cd-48a0-8574-4797d8184661/vsphere-vddk-65-release-notes.html
it should be compatible as follows:
"The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other
words, VDDK 6.5 and all its update releases support vSphere 5.5, 6.0, and the next major release (except new features)."
However, this requires code changes and testing, so it may take some time.
Regards,
Stephan
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> bareos-users...@googlegroups.com <mailto:bareos-users...@googlegroups.com>.
> To post to this group, send email to bareos...@googlegroups.com <mailto:bareos...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
--
Egoitz Aurrekoetxea
Good morning,
I have adapted it to work with VDDK 6.5. Let me check it properly and I could provide you the diffs...
Best regards,
Jörg Steffens
> Good morning,
>
>
> I have adapted it to work with VDDK 6.5. Let me check it properly and I
> could provide you the diffs...
this in more detail later, as I'm not sure, how far it has been tested.
There are now Bareos packages for vddk 6.5 available in
http://download.bareos.org/bareos/experimental/nightly/
and
http://download.bareos.org/bareos/release/17.2/
There is also updated documentation on it:
http://doc.bareos.org/master/html/bareos-manual-main-reference.html#VMwarePlugin
If you compare your changes with the changes in
https://github.com/bareos/bareos-vmware you can maybe point out, if
there is still something missing.
regards,
Jörg
--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
OJK
I've just tried the latest RC 17.2.4 against our vSphere 6.5.
Following the documentation, and configuring a job to backup one VM, I get the following Error:
16-Nov 14:56 client1 JobId 148: Fatal error: python-fd: check_dumper(): bareos_vadp_dumper returncode: 1 error output:
Failed to Prepare For Access: One of the parameters was invalid [3]
Failed to connect to vserver : One of the parameters was invalid [3]
16-Nov 14:56 client1 JobId 148: Fatal error: python-fd: plugin_io[IO_CLOSE]: bareos_vadp_dumper returncode: 1 error output:
Failed to Prepare For Access: One of the parameters was invalid [3]
Failed to connect to vserver : One of the parameters was invalid [3]
I'm calling the Plugin in the FileSet:
FileSet {
Name = "VMware"
Description = "Backup VMware"
Include {
Options {
Signature = MD5
compression = GZIP
}
Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=DC1:folder=/DEV:vmname=dev01:vcserver=vserver:vcuser=bar...@vsphere.local:vcpass=password:vcthumbprint=<HASH>:transport=nbdssl"
}
}
Did I configure something wrong? I can provide any additional Documentation/Configuration that is required to fix this problem.
Regards,
OJK
Stephan Duehr
sorry, my fault. There were some commits missing on
https://github.com/bareos/bareos-vmware/tree/bareos-17.2
We are currently building new packages and will publish them as soon as possible.
Regards,
Stephan
Stephan Dühr stepha...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-90
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Stephan Duehr
fixed RC 17.2.4 vmware plugin packages have been published now at
http://download.bareos.org/bareos/release/17.2/
It should work fine now. Feedback is welcome. I ran some tests against vSphere 6.5 and
was surprised to see nbd and nbdssl faster than hotadd. I always thought that hotadd
would be faster. May be that changed with 6.5. Was on VMFS6 also, that may matter.
May be you want to share your results.
Regards,
Stephan
Egoitz Aurrekoetxea
Hi mates!!
I don't really have really seen your diffs but I did basically :
- Add Thumbprint (now required in 6.5 and optional in 6.0 vddk)
to /usr/lib64/bareos/plugins/BareosFdPluginVMware.py... I planned
to fetch properly the values from conf by modifying properly it's
source.... I now was
entering vcenter thumbprint by hand :
656 self.si =
SmartConnect(host=self.options['vcserver'],
657
user=self.options['vcuser'],
658
pwd=self.options['vcpass'],
---------> there
659 port=443)
660 self.si_last_keepalive = int(time.time())
661
- Later in Vadp Dumper something like :
--- bareos_vadp_dumper.cpp-defecto 2017-11-17
10:50:50.655490669 +0100
+++ bareos_vadp_dumper.cpp 2017-11-18 12:55:50.277593390
+0100
@@ -391,6 +391,7 @@
static void cleanup(void)
{
VixError err;
+ char elthumbprint[] =
"8F:61:8C:BC:E7:7C:3F:CE:32:65:9B:55:C6:FC:E6:8C:29:AB:39:B5";
if (info) {
VixDiskLib_FreeInfo(info);
@@ -406,6 +407,8 @@
write_diskHandle = NULL;
}
+ cnxParams.thumbPrint = elthumbprint;
+
if (connection) {
VixDiskLib_Disconnect(connection);
if (cleanup_on_disconnect) {
@@ -484,6 +487,7 @@
int succeeded = 0;
VixError err;
const char *snapshot_moref = NULL;
+ char elthumbprint[] =
"8F:61:8C:BC:E7:7C:3F:CE:32:65:9B:55:C6:FC:E6:8C:29:AB:39:B5";
memset(&cnxParams, 0, sizeof(cnxParams));
@@ -576,6 +580,8 @@
snapshot_moref = json_string_value(object);
}
+ cnxParams.thumbPrint = elthumbprint;
+
if (!local_vmdk) {
err = VixDiskLib_PrepareForAccess(&cnxParams,
BAREOS_VADPDUMPER_IDENTITY);
if (VIX_FAILED(err)) {
@@ -588,6 +594,7 @@
}
}
+ cnxParams.thumbPrint = elthumbprint;
err = VixDiskLib_ConnectEx(&cnxParams, (readonly) ? TRUE :
FALSE, snapshot_moref, force_transport, &connection);
if (VIX_FAILED(err)) {
char *error_txt;
And I'd swear that worked to me without issues (with the backups
ending fine and so...) with 17.2 and Debian packages... although
in my last set up with Centos, yesterday's or one more day ago (at
most) set up vm (with from that moment official
packages of Bareos for Centos) with Bareos and Centos, it seemed
it was not able to re-enable vmotion after the snapshot was
removed... Fortunately.... I have read this thread, done a yum
update and works now... although I have to check
all properply....
I'll check all of it and will be in contact.....
By the way, OJK the error below is because vadp dumper is not passing to vddk the thumprint (properly at least).... just it....
Cheers!
OJK
I've tried the Version: 17.2.4-5.1.el7.x86_64 as well as: 17.2.4-6.1.el7.x86_64 from the Nightly/Experimental Builds. And I'm getting a new Error:
20-Nov 09:03 bareos01 JobId 34: No prior Full backup Job record found.
20-Nov 09:03 bareos01 JobId 34: No prior or suitable Full backup found in catalog. Doing FULL backup.
20-Nov 09:03 bareos01 JobId 34: Start Backup JobId 34, Job=vmdk_DUS01_DEVEL.2017-11-20_09.03.35_09
20-Nov 09:03 bareos01 JobId 34: Using Device "dcnas01" to write.
20-Nov 09:03 bareos01 JobId 34: Volume "DEV-VMware-Full-0004" previously written, moving to end of data.
20-Nov 09:03 bareos01 JobId 34: Ready to append to end of Volume "DEV-VMware-Full-0004" size=37595931518
20-Nov 09:03 bareos01 JobId 34: python-fd: Starting backup of /VMS/DUS01/DEVEL/dev01/[nas01] dev01/dev01.vmdk_cbt.json
20-Nov 09:03 bareos01 JobId 34: python-fd: Starting backup of /VMS/DUS01/DEVEL/dev01/[nas01] dev01/dev01.vmdk
20-Nov 09:32 bareos01 JobId 34: User defined maximum volume capacity 53,687,091,200 exceeded on device "dcnas01" (/var/lib/bareos/storage).
20-Nov 09:32 bareos01 JobId 34: End of medium on Volume "DEV-VMware-Full-0004" Bytes=53,687,030,232 Blocks=832,204 at 20-Nov-2017 09:32.
20-Nov 09:32 bareos01 JobId 34: Created new Volume "DEV-VMware-Full-0009" in catalog.
20-Nov 09:32 bareos01 JobId 34: Labeled new Volume "DEV-VMware-Full-0009" on device "dcnas01" (/var/lib/bareos/storage).
20-Nov 09:32 bareos01 JobId 34: Wrote label to prelabeled Volume "DEV-VMware-Full-0009" on device "dcnas01" (/var/lib/bareos/storage)
20-Nov 09:32 bareos01 JobId 34: New volume "DEV-VMware-Full-0009" mounted on device "dcnas01" (/var/lib/bareos/storage) at 20-Nov-2017 09:32.
20-Nov 09:36 bareos01 JobId 34: Fatal error: python-fd: Traceback (most recent call last):
File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 42, in start_backup_file
return bareos_fd_plugin_object.start_backup_file(context, savepkt)
File "/usr/lib64/bareos/plugins/BareosFdPluginVMware.py", line 176, in start_backup_file
self.vadp.disk_device_to_backup = self.vadp.disk_devices.pop(0)
IndexError: pop from empty list
20-Nov 09:36 bareos01 JobId 34: Elapsed time=00:33:16, Transfer rate=9.135 M Bytes/second
20-Nov 09:36 bareos01 JobId 34: Error: Bareos bareos01 16.2.4 (01Jul16):
Build OS: x86_64-redhat-linux-gnu redhat CentOS Linux release 7.0.1406 (Core)
JobId: 34
Job: vmdk_DUS01_DEVEL.2017-11-20_09.03.35_09
Backup Level: Full (upgraded from Incremental)
Client: "bareos01" 16.2.4 (01Jul16) x86_64-redhat-linux-gnu,redhat,CentOS Linux release 7.0.1406 (Core) ,CentOS_7,x86_64
FileSet: "VMware_DUS01_DEVEL" 2017-11-17 15:53:22
Pool: "DEV-VMware-Full" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "dcnas01" (From Pool resource)
Scheduled time: 20-Nov-2017 09:02:45
Start time: 20-Nov-2017 09:03:37
End time: 20-Nov-2017 09:36:53
Elapsed time: 33 mins 16 secs
Priority: 10
FD Files Written: 2
SD Files Written: 2
FD Bytes Written: 18,234,137,568 (18.23 GB)
SD Bytes Written: 18,234,139,681 (18.23 GB)
Rate: 9135.3 KB/s
Software Compression: 57.5 % (gzip)
VSS: no
Encryption: yes
Accurate: yes
Volume name(s): DEV-VMware-Full-0004|DEV-VMware-Full-0009
Volume Session Id: 1
Volume Session Time: 1511164938
Last Volume Bytes: 2,161,090,120 (2.161 GB)
Non-fatal FD errors: 1
SD Errors: 0
FD termination status: Fatal Error
SD termination status: Canceled
Termination: *** Backup Error ***
Any help would be appreciated.
Regards,
OJK
Egoitz Aurrekoetxea
Hi!
Can you be trying to backup local files?. I mean local in the own machine instead of connecting to an esx server/vcenter??. Could you have the local transport flag enabled for vadp dumper¿?...
Cheers,
OJK
thanks for the answer.
The bareos01 is a physical Server that connects to vCenter/ESX via the VMware Plugin. Do you mean backup from the local bareos Server? That works without problems.
A "normal" File Backup from VMs running on our ESX Cluster also works without problems. Or do you mean Backup to the Local HDD in the Bareos Server and not our Backup NAS?
And afaik I didn't configure the vadp dumper, so - don't know where I could enable/disable that flag?
Regards,
OJK
OJK
> Hi,
>
> thanks for the answer.
>
> The bareos01 is a physical Server that connects to vCenter/ESX via the VMware Plugin. Do you mean backup from the local bareos Server? That works without problems.
> A "normal" File Backup from VMs running on our ESX Cluster also works without problems. Or do you mean Backup to the Local HDD in the Bareos Server and not our Backup NAS?
I tried a backup directly to the local HDD of the bareos-dir/sd server => Same Error.
OJK
Hello,
is there any update/idea regarding my problem? Or should I provide additional conf/log files?
Regards,
OJK
Игорь Дементьев
When I test backups with VMWare environment version 6.0(on other servers for backup host and ESXi) all was OK.
But in VMWare 6.5 VM backups fail. All configs and configuration steps was same.
I tried many combinations with vcthumbprint and transport (nbdssl and nbd).
BareOS 16.2 error:
Warning: SSL: connect failed
Warning: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error:
Failed to open [DS_2] BACKUP-TEST/BACKUP-TEST_1.vmdk : You do not have access rights to this file [13]
BareOS 17.2 error:
Failed to Prepare For Access: One of the parameters was invalid [3]
Failed to connect to 10.17.14.10 : One of the parameters was invalid [3]
Now I start experiment with workaround from Egoitz Aurrekoetxea. But I have not VDDK libraries and can't compile bareos_vadp_dumper.cpp
And VMWare say: "Your download request has been received and is under review for download eligibility".
"
bareos_vadp_dumper.cpp:51:24: fatal error: vixDiskLib.h: No such file or directory
#include <vixDiskLib.h>
^
compilation terminated.
"
Stephan Duehr
does this VM have multiple disks?
Are you on vSphere 6.5?
However, at least against vSphere 5.5 I can't reproduce this error, the plugin
works with VMs with multiple disks, and also when spanning multiple storage volumes.
Regards,
Stephan
OJK
thanks for your answer. It seems to be a configuration error on my side.
I defined 2 VMs in my fileset:
FileSet {
Name = "DC_DEVEL"
Include {
Options {
Signature = MD5 # calculate md5 checksum per file
compression = LZ4
}
Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=DC:folder=/DEVEL:vmname=dev02:vcserver=vcenter01:vcuser=bar...@vsphere.local:vcpass=<PASSWORD>:vcthumbprint=<THUMBPRINT>:transport=nbdssl"
Plugin = "python:module_path=/usr/lib64/bareos/plugins:module_name=bareos-fd-vmware:dc=DC:folder=/DEVEL:vmname=dev01:vcserver=vcenter01:vcuser=bar...@vsphere.local:vcpass=<PASSWORD>:vcthumbprint=<THUMBPRINT>:transport=nbdssl"
}
}
And using only one plugin line (and deleting the old backups from the bareos Volume), it seems to be working. Only disabling one plugin (which I've tried last week) didn't change anything. Only after deleting the old (successful) Backups it worked.
Is there a way to configure multiple VMs in one Plugin line? Or to say, backup all VMs in one folder? Because defining a different fileset for each VM is rather complicated to manage.
Regards,
OJK
Александр Канайкин
Today i've tried to update to 17.2.4 and backup vmware 6.5 vm
i've added new conf parameters. But error still the same.
bareos-fd JobId 520: Fatal error: python-fd: plugin_io[IO_CLOSE]: bareos_vadp_dumper returncode: 1 error output:
Warning: VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
Warning: SSL: connect failed
Warning: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error:
Failed to open [CCM] wikitest.ccm.loc/wikitest.ccm.loc_1.vmdk : You do not have access rights to this file [13]
Filename to backup was correctly selected. But as earlier... No rights to file.
Александр Канайкин
I updated bareos:
sudo yum update bareos
Then remove bareos-vmware-plugin and fresh install it.
But one packet bareos-vmware-vix-disklib-5.5.4 left unupdated. Then i updated to bareos-vmware-vix-disklib-6.5.2.
And it works.