Backuping files from a network drive (shared folder) on Windows

[Olivier Gagnon]

Hello,

I am trying to backup files from our SAN sharing fils via CIFS. I need to use the windows client. I have tried using mapping the shared folder to a drive (i.e Q:\) but I get this error:

Could not stat "q:/": ERR=The system cannot find the path specified.

I get the same error using UNC (//san/folder or \\san\folder).

Is there a way to do that with Bareos ?

Thanks,

O.

[Marco van Wieringen]

Keep in mind that Bareos runs as a service and under a certain system
account. I am myself not a windows user but I seem to rember that when
you do a net mapping that it is for a user only and not for the whole
system so maybe you are bitten by that.

A google on "windows map network drive global" gives some hits which
I think exactly explain your problem and workaround.

--
Marco van Wieringen marco.van...@bareos.com
Bareos GmbH & Co. KG Phone: +49-221-63069389
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, M. Außendorf, J. Steffens,
P. Storz, M. v. Wieringen

[Olivier Gagnon]

Hi,

Thanks for the comment.

With some tools like procmon.exe and wireshark and with the win32 api help, I succeeded to get the path right. The path must use the Long UNC path: ie. \\?\UNC\Server\Folder

Now while using this, in procmob, I see the client access the share with Success but I get errors Access is denied for a folder and Bad file handle for a file.

I run the bareos client via a command line with debugging on so the credentials are OK , as wireshark says.

Yours truly,

O.

[Michael Mol]

Marco had it exactly right. You need to make sure the service the
Barios runs under has access to the network share you're trying to back
up. This is consistent with the symptoms you are seeing.

--
I send GPG-encrypted email by default. If I sent this message to you,
and not to a list, it means I probably checked to see if you had a
published key.

[Olivier Gagnon]

Hi Michael,

Thanks for the update.

As I said, the service is running as a domain user with the necessary credentials. Actually, I even tried with a Domain Admin credentials.

Procmon.exe lists 'bareos-fd.exe' as running with the correct domain user. Wireshark shows no authentication nor authorization error.

All communications between the client and the SMB server are successful.

A Win32 programmer guru at my job lead me to some clues so I am trying recompiling the Windows client with his input.

I will let you know if I succeed. Anyhow, any other lead would be appreciated.

Yours truly,

O.

[Olivier Gagnon]

I have succeeded to cross compile the win32 client using mingw and I've been testing things since this morning.

(Btw, I may have some patches to make the compiling more friendly).

I'm using the latest GIT code btw.

As I have said earlier, you definitely have to use the syntax "UNC\Server\Share" in the File directive.

I have set up two guest shares, one on the SAN and another using SAMBA.

I have tried using the Domain Admin default account (DOMAIN\Administrator) and the Domain User we use for backup, with the Backup Operators rights.

In all those situations, I can see that the Bareos-fd.exe client, running as a service or from the command line with debugging, is connecting to the network shared folder with the defined credentials from the logs.

In all cases, I get an Access Denied error.

However, after playing with the code, I succeeded to modify it to backup files directly using the form "UNC\Server\Share\file.ext". I also got the ACEs NT rights saved and restored.

According to the Microsoft API Dev Center, the function GetFileAttributes cannot be used on shares. By bypassing that call in stat2() in compat.c, I can get the listing of the files/directories with the subsequent FindFirstFile() and FindNextFile() calls.

I'm continuing my tests but there is definitely something there.

Yours truly,

O.

On Fri, Nov 1, 2013 at 12:51 PM, Michael Mol <mik...@gmail.com> wrote:

[Marco van Wieringen]

Olivier Gagnon <ogagnon <at> ludia.com> writes:

> I have succeeded to cross compile the win32 client using mingw and I've
> been testing things since this morning.
> (Btw, I may have some patches to make the compiling more friendly).

The current setup is light years ahead of what it was in Bacula where
cross compiling was a art not mastered by many due to their archaic
Makefile we rewrote those from scratch but geared towards using
the SUSE Open Build Service (OBS) mingw infrastructure. So it might
not work to great if you try an other path to do cross compiling
although it should be much easier then before. As to patches if they
make sure they don't interfere with building it in OBS then we can see
if the make sense if they do I don't think its something we want to
change now as we have to much time invested in building everything
with OBS that makes it possible to rebuild a whole release in a
couple of hours including running the release through CI.

> I'm using the latest GIT code btw.
> As I have said earlier, you definitely have to use the syntax
> "UNC\Server\Share" in the File directive.
> I have set up two guest shares, one on the SAN and another using SAMBA.
> I have tried using the Domain Admin default account
> (DOMAIN\Administrator) and the Domain User we use for backup, with
> the Backup Operators rights.
> In all those situations, I can see that the Bareos-fd.exe client,
> running as a service or from the command line with debugging, is
> connecting to the network shared folder with the defined credentials
> from the logs.
>
> In all cases, I get an Access Denied error.
>

I think the whole stuff contributed to Bacula in 2008 has never seen
lots of changes ever since. We have quite a load of changes in the
development pipeline just because the code needed so much changes
to be able to cope with the new stuff Windows added over the year
(dedup, VMPs, new API's). We hope to have that ready to merge into
master before the end of the year it just needs a lot of testing as
we had to do quite some surgery to get the code in a somewhat better
shape then it was.

> However, after playing with the code, I succeeded to modify it to backup
> files directly using the form "UNC\Server\Share\file.ext". I also got
> the ACEs NT rights saved and restored.

Ok interesting. I hope if you come up with some patches that you
implement it in a somewhat generic way as I think backing up SMB
filesystems via a client is not the "wanted" way of backing up data.
Most systems like NetAPP won't like be backed up via NFS or SMB
they have other ways of doing it in a less resource intensive way
(NDMP etc.).

> According to the Microsoft API Dev Center, the function
> GetFileAttributes cannot be used on shares. By bypassing that
> call in stat2() in compat.c, I can get the listing of the
> files/directories with the subsequent FindFirstFile() and
> FindNextFile() calls.

Ok interesting I think the whole code uses quite some API that is
either not implemented on shares or implemented differently on shares.
Like I said I think it was never designed to be used in that way.

> I'm continuing my tests but there is definitely something there.

Ok great we have a big update for windows pending to make the code
a lot more clean and future proof. But due to the fact that we want
to test this new code first (and because its big it takes time.)
It should get out of the development pipeline rather sooner then
later but with new code you never know.

[Olivier Gagnon]

On Tuesday, November 5, 2013 1:17:24 PM UTC-5, Marco van Wieringen wrote:
> Olivier Gagnon <ogagnon <at> ludia.com> writes:
>
>
>
>
>
> > I have succeeded to cross compile the win32 client using mingw and I've
>
> > been testing things since this morning.
>
> > (Btw, I may have some patches to make the compiling more friendly).
>
> The current setup is light years ahead of what it was in Bacula where
>
> cross compiling was a art not mastered by many due to their archaic
>
> Makefile we rewrote those from scratch but geared towards using
>
> the SUSE Open Build Service (OBS) mingw infrastructure. So it might
>
> not work to great if you try an other path to do cross compiling
>
> although it should be much easier then before. As to patches if they
>
> make sure they don't interfere with building it in OBS then we can see
>
> if the make sense if they do I don't think its something we want to
>
> change now as we have to much time invested in building everything
>
> with OBS that makes it possible to rebuild a whole release in a
>
> couple of hours including running the release through CI.
>
>

I understand you want to keep your setup for CI. Since I am building on
Debian, I had to made some specific changes. But, there are things that
I think could be globally.

I have finally succeeded to implement everything to backup shared folders.

The code was not implemented to support shares. The only exception on "special folder" was on disk drive (ie. c:). I had to add exceptions on network drives
beginning with long path UNC (UNC\Server\Share) as described by MSDN. Strangely,
according to MSDN, short path UNC (\\Server\Share) should have work but it didn't. And I tried !

So, from the code I have here, I have made backups of our SAN's shared folders via Windows 7 and the total files and bytes from the 'estimate' bconsole command are exactly the same as what the share's 'windows properties' says.
All the ACEs rights are preserved in the backups and restores.

I've made tests with Incremental and Differential level backups and
the supposed behaviour is working.

>
>
> > I'm continuing my tests but there is definitely something there.
>
> Ok great we have a big update for windows pending to make the code
>
> a lot more clean and future proof. But due to the fact that we want
>
> to test this new code first (and because its big it takes time.)
>
> It should get out of the development pipeline rather sooner then
>
> later but with new code you never know.
>

Tell me if you want my updates/patches. I think it could help !

O.

P.S.:

Maybe this thread should belong in bareos-devel.

[Marco van Wieringen]

Olivier Gagnon <ogagnon <at> ludia.com> writes:
> I understand you want to keep your setup for CI. Since I am building on
> Debian, I had to made some specific changes. But, there are things that
> I think could be globally.

I think we want to keep things mostly generic I don't want to go back
to the nightmare the build system for windows was on Bacula. But feel
free to propose changes that help others. If its something simple
like changing the patch to the cross compiler etc I think it doesn't
make much sense as it will be different on every platform we tried to
make things so configurable that you could easily change that kind of
settings but within reason we can look at possible enhancements there.

> I have finally succeeded to implement everything to backup shared folders.
>
> The code was not implemented to support shares. The only exception on
> "special folder" was on disk drive
> (ie. c:). I had to add exceptions on network drives
> beginning with long path UNC (UNC\Server\Share) as described by MSDN.
> Strangely, according to MSDN, short path UNC (\\Server\Share) should
> have work but it didn't. And I tried !

Some things on windows tend to work differently on different versions
of the OS.

>
> So, from the code I have here, I have made backups of our SAN's shared
> folders via Windows 7 and the total files and bytes from the
> 'estimate' bconsole command are exactly the same as what the
> share's 'windows properties' says.
> All the ACEs rights are preserved in the backups and restores.
>

Ok great as its using the windows backup API it probably picks up all
needed information.

> I've made tests with Incremental and Differential level backups and
> the supposed behaviour is working.
>

Ok so it seems the needed changes are rather limited then. Like I said
before its outside the normal design envelop but if the patches are
not to big we can look at integrating them into a next release.

>
> Tell me if you want my updates/patches. I think it could help !
>

Sure we can at least take a look, see

https://www.bareos.org/en/howto-contribute.html

on how to contribute to the project.

>
> Maybe this thread should belong in bareos-devel.

Now that we are into the inner guts of the implementation it
might make sense to start a new thread there.