Extended attribute in cifs share error

[winsoz winsoz]

Hello everybody, I use bup in my KUbuntu distro with Kup, the bup version is 0.32. I've two backup plans on remote CIFS share (windows server) and one on my home directory.

The share is mounted in fstab with the line: "//SERVER/FOLDER /mnt/FOLDER/ cifs credentials=/home/USER/.smbcredentials,uid=1000,gid=1000 0 0". Unfortunately after the distro upgrade (from KUbubtu 22.04.1 to 22.10) the plans of the remote share fails.

Digging into the bup code I found the problem in this for loop https://github.com/bup/bup/blob/0.32.x/lib/cmd/save-cmd.py#L383 and from there in https://github.com/bup/bup/blob/0.32.x/lib/bup/metadata.py#L900.

When the script try to get the extended attribute of /mnt/FOLDER it gets the "bad file descriptor" error or "permission denied" error (I know is strange that the error isn't unique, but whatever...).

To test the problem I wrote a simple program that uses the listxattr() function of include <attr/xattr.h> and the problem remains (also the alternate error code)

I understand that the error is not direct fault of bup, but it fails completely the backup process (even if I think it goes well) and I'm without this beautiful bup since 2 months ago.

If there is something else that I can do/test, here I'm. Thank you all for your support.

[Rob Browning]

winsoz winsoz <win...@gmail.com> writes:

> When the script try to get the extended attribute of /mnt/FOLDER it gets

> the "bad file descriptor" error or "permission denied" error.

Sounds like it's returning errors other than the EOPNOTSUPP we already
handle in _add_linux_xattr()?

If so, do you know what the exact exceptions and/or errno values are
(and ideally what provokes each). And if the output on error includes
any details, could you send that too?

Thanks for the report.
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Sounds like it's returning errors other than the EOPNOTSUPP we already
> handle in _add_linux_xattr()?

I figured out how to set up a trivial test vm, and there, attempting to
save a home dir mounted on /mnt that contains a bup src tree fails like
this:

./bup index /mnt
read Linux attr: [Errno 13] Permission denied: '/mnt/bup/main/test/tmp/test-ls-PfuFpfS/src/socket'
Ignoring attr EINVAL; if you're not using ntfs-3g, please report: /mnt/bup/main/test/tmp/test-ls-PfuFpfS/src/file
Indexing: 944, done (739 paths/s).
WARNING: 1 errors encountered.

Is that similar to what you were seeing? i.e. the real error there is
the EACCES for .../src/socket, which seems like an odd error, and if
that's the norm there, I'd be hesitant universally quash that there,
without understanding why it's happening better.

I also suppose, if I arranged the test right, we might want to add cifs
to the ntfs-3g message.

[Winsoz]

Hello Rob,

below my answer to your question.

Thank you for your help.

Il giorno sabato 18 febbraio 2023 alle 18:24:52 UTC+1 r...@defaultvalue.org ha scritto:

winsoz winsoz <win...@gmail.com> writes:

> When the script try to get the extended attribute of /mnt/FOLDER it gets
> the "bad file descriptor" error or "permission denied" error.

Sounds like it's returning errors other than the EOPNOTSUPP we already
handle in _add_linux_xattr()?

Yes the permission denied error (the recurring one nowadays) is the errno 13 (EACCES). In fact is not handle in that function.

If so, do you know what the exact exceptions and/or errno values are
(and ideally what provokes each). And if the output on error includes
any details, could you send that too?

Unfortunately I have no idea what can provoke it. The output in the log file is "[Errno 13] Permission denied".

[Winsoz]

Rob,

about this mail I want to specify that the error shows up during the bup save stage. The previously bup index stage goes well.

[Rob Browning]

Winsoz <win...@gmail.com> writes:

> about this mail I want to specify that the error shows up during the bup
> save stage. The previously bup index stage goes well.

OK, ideally I'd like to understand the problem better before attempting
any solution, i.e. I'd like to know why that's an error, does it depend
on the server, the mount options, the local kernel or library...

Higher level, is it a bug we're working around somewhere else, or is
this the intended behavior for this arrangement? (Not knowing much
about cifs makes that a bit more difficult.)

I did look through some of the previous reports of trouble with cifs,
and I didn't see any issues with xattrs.

In any case, it might be useful (now or later) to know a bit more about
the arrangement, i.e.

- what kind of machine is running bup (linux, a bsd, macos, ...)
- what's the version of python-pyxattr
- what's the cifs server like, etc.
- if the server is samba, what's the remote filesystem like (ext4?)

Oh, and as a local hack, if this is the only problem, I suppose you
could consider just adding

xattr = None

at the top-level, after the block at the top of metadata.py that checks
for xattr support, i.e. the block guarded by

if sys.platform.startswith('linux'):

assuming that xattrs aren't crtiical to you right now. You could also
just remove python-pyxattr from the system, but of course that might not
be acceptable.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Higher level, is it a bug we're working around somewhere else, or is
> this the intended behavior for this arrangement?

For example, I suppose it could just be that cifs wants/needs to return
EACCES for xattrs in some situations. (Though why the top-level mount
point in your case?)

If so, and maybe even if not, we could consider deferring that error the
way we do right now for EACCES elsewhere (e.g. for attrs). Say
something like this in _add_linux_xattr():

diff --git a/lib/bup/metadata.py b/lib/bup/metadata.py
index 0a7514d4d..4412da5b8 100644
--- a/lib/bup/metadata.py
+++ b/lib/bup/metadata.py
@@ -643,6 +643,10 @@ class Metadata:
self.linux_xattr = xattr.get_all(path, nofollow=True)
except EnvironmentError as e:
if e.errno == errno.EOPNOTSUPP:
+ return
+ if e.errno == errno.EACCES:
+ add_error('read Linux xattr: %s' % e)
+ else:
raise

def _same_linux_xattr(self, other):

With that, the save could finish (if EACCES were the only problem) with
a non-zero exit.

Of course at some point we probably need to figure out a sufficiently
flexible way to suppress certain expected errors, or disable relevant
features: "--no-xattr" or "--ignore-errors xattr,posix1e" or ..., but
that's a potentially much more complex topic (depending on what we end
up thinking we need).

[Winsoz]

Il giorno lunedì 20 febbraio 2023 alle 19:03:20 UTC+1 r...@defaultvalue.org ha scritto:

Winsoz <win...@gmail.com> writes:

> about this mail I want to specify that the error shows up during the bup
> save stage. The previously bup index stage goes well.

OK, ideally I'd like to understand the problem better before attempting
any solution, i.e. I'd like to know why that's an error, does it depend
on the server, the mount options, the local kernel or library...

Higher level, is it a bug we're working around somewhere else, or is
this the intended behavior for this arrangement? (Not knowing much
about cifs makes that a bit more difficult.)

That's the reason because I wrote a simple program. I mean obviously is not a bup fault, but something with the system upgrade has changed. In fact now listxattr() function returns that error (yeah we don't know what it returned before, but for sure all worked great)

I did look through some of the previous reports of trouble with cifs,
and I didn't see any issues with xattrs.

In any case, it might be useful (now or later) to know a bit more about
the arrangement, i.e.

- what kind of machine is running bup (linux, a bsd, macos, ...)
- what's the version of python-pyxattr
- what's the cifs server like, etc.
- if the server is samba, what's the remote filesystem like (ext4?)

The machine is a KUbuntu 22.10 with kernel 5.19.0-31-generic x86_64.

xattr.__version__ = '0.7.2'.
The file server is running on Windows Server 2019, local filesystem is ext4, for what is worth my Samba version is 4.16.8-Ubuntu, but I know this is not the file server share.

Oh, and as a local hack, if this is the only problem, I suppose you
could consider just adding

xattr = None

at the top-level, after the block at the top of metadata.py that checks
for xattr support, i.e. the block guarded by

if sys.platform.startswith('linux'):

assuming that xattrs aren't crtiical to you right now. You could also
just remove python-pyxattr from the system, but of course that might not
be acceptable.

How do I know if xattr is needed to bup for backup without lost features or something else?

I can modify the metadata.py to ignore xattr module or whatever but I don't if this breaks something in the bup workflow.

[Winsoz]

Il giorno lunedì 20 febbraio 2023 alle 19:22:47 UTC+1 r...@defaultvalue.org ha scritto:

Rob Browning <r...@defaultvalue.org> writes:

> Higher level, is it a bug we're working around somewhere else, or is
> this the intended behavior for this arrangement?

For example, I suppose it could just be that cifs wants/needs to return
EACCES for xattrs in some situations. (Though why the top-level mount
point in your case?)

I try to follow the source code of the function in the kernel, but really i don't understand what is happening to return such an error.

If so, and maybe even if not, we could consider deferring that error the
way we do right now for EACCES elsewhere (e.g. for attrs). Say
something like this in _add_linux_xattr():

diff --git a/lib/bup/metadata.py b/lib/bup/metadata.py
index 0a7514d4d..4412da5b8 100644
--- a/lib/bup/metadata.py
+++ b/lib/bup/metadata.py
@@ -643,6 +643,10 @@ class Metadata:
self.linux_xattr = xattr.get_all(path, nofollow=True)
except EnvironmentError as e:
if e.errno == errno.EOPNOTSUPP:
+ return
+ if e.errno == errno.EACCES:
+ add_error('read Linux xattr: %s' % e)
+ else:
raise

def _same_linux_xattr(self, other):

With that, the save could finish (if EACCES were the only problem) with
a non-zero exit.

I thought  a similar code, but I'm not so inside the bup internals to understand if such error handling is the right thing to do.

Maybe it's correct for the root node of the mount point, but is wrong to do at some folder below that point.

It's better to discuss with grown-up people. :-)

[Greg Troxel]

Please adjust your MUA to quote properly in plain text, or if you can't,
don't include the previous text. I can't figure out what is quoting
Rob and what is new.

[Winsoz]

Sorry to hear that, but I'm using the google groups to interface with the mailing list.

I see two different colors for the quoted text and for normal text.

[Greg Troxel]

Winsoz <win...@gmail.com> writes:

> Sorry to hear that, but I'm using the google groups to interface with the
> mailing list.
> I see two different colors for the quoted text and for normal text.

I am not seeing colors. I am asking that the text/plain part (which
does not have colors, but just characters) be readable, which usually
involves indentation or quoting indicators.

[Winsoz]

Greg,

this is a screenshot of the google interface and the visualization is similar to the gmail Android app.

I use the answer button and it automatically add the '>' character to the quoting text. I think that your mail client has to interpret it as quoted text.

Am I wrong?

[Nix]

On 20 Feb 2023, win...@gmail.com outgrape:

> Yes the permission denied error (the recurring one nowadays) is the errno
> 13 (EACCES). In fact is not handle in that function.

Looking at the kernel (probably not Ubuntu's, this is just a random 6.2
one I happen to have lying around), cifs_xattr_get() can directly return
only -EOPNOTSUPP or -ERANGE, but functions it calls (like
cifs_sb_tlink()) can return -ENOMEM, -EACCES, and no doubt a lot else.
Since virtually all cifs functions call cifs_sb_tlink(), virtually all
cifs functions can return -EACCES (or -ENOMEM, but I doubt we should
even try to do anything if we hit that, we are just doomed).

It looks like this happens if the TCP connection to the server is
somehow interrupted and an attempt to bring it back with
cifs_construct_tcon() fails (alas, the actual error gets eaten by
cifs_sb_tlink()).

I'm a bit surprised that anything else on that cifs connection works
after this, really. I wonder what wireshark would show...

--
NULL && (void)

[Rob Browning]

Winsoz <win...@gmail.com> writes:

> Sorry to hear that, but I'm using the google groups to interface with the
> mailing list.
> I see two different colors for the quoted text and for normal text.

For what it's worth, I have the same issue -- i.e. it was hard to tell
what you wrote and what you were quoting from my message. In the
browser (firefox), the color difference is very subtle here, and there's
no blank line between your and my content. In my mail agent (notmuch),
there's no (visible) difference at all between my content and your reply
in either the text or the html representations of your MIME message.

Offhand, I'd guess there's some typical way (or ways) to handle reply
quoting in the gmail reply interface, but I don't really know.

In plain text emails, of course, the standard way to quote things is to
prefix the lines with "> ", i.e.

> This is a quote

(I suspect there's also some way to get gmail to send plain-text replies
if/when you want to, but that shouldn't be required.)

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> I'm a bit surprised that anything else on that cifs connection works
> after this, really. I wonder what wireshark would show...

Hmm, ok -- thanks for investigating.

Winsoz, I forget, did you say this happens all the time on the same
path(s), or is it less predictable?

Thanks

[Rob Browning]

Winsoz <win...@gmail.com> writes:

> How do I know if xattr is needed to bup for backup without lost
> features or something else? I can modify the metadata.py to ignore
> xattr module or whatever but I don't if this breaks something in the
> bup workflow.

I don't know enough about the situation to say for sure, but if it were
just linux, and say ext4, losing (some) xattrs might or might not be
"fine". i.e. if having the "normal" linux permissions
(uid/gid/timestamps/mode/etc.) were good enough, you'd have those. You
just wouldn't have any extended attributes, which might or might not be
important.

https://en.wikipedia.org/wiki/Extended_file_attributes#Linux

For example, they might well be important if you were relying on ACLs,
SELinux, etc.

[Rob Browning]

Nix <n...@esperi.org.uk> writes:

> It looks like this happens if the TCP connection to the server is
> somehow interrupted and an attempt to bring it back with
> cifs_construct_tcon() fails (alas, the actual error gets eaten by
> cifs_sb_tlink()).

So it didn't look to you like this could be caused by say trying to read
xattrs that you currently didn't have the right to read? i.e. I just
wondered if this could be some "not running as root and the mount point
is restricted" kind of situation, or something...

And more broadly, if it's possible for a user to be disallowed xattrs
via EACCES, say if the attempt wasn't running as root[1], then I suppose
we really might want to defer this error.

[1] ...similar to chmod 600 wrt the actual data, etc. -- though I'm not
positive that would provoke EACCESS, specifically.

And hmm, I just noticed this from a while back:

commit 915db99286ee22b7f513b20dfa0ae6c51340880a
Author: Rob Browning <r...@defaultvalue.org>
Date: Sat May 21 09:11:01 2016 -0500

Treat EACCES like EPERM when removing xattrs

Thanks to Brian Minton for reporting the problem on Fedora 23.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>

Via note/0.28-from-0.27.1.md:

* `bup restore` should no longer crash when an attempt to remove an
xattr returns EACCES.

Not sure it's germane, though.

Winsoz, may not be relevant, but were you running bup save as root, or
as some other user?

Thanks

[Nix]

On 25 Feb 2023, Rob Browning outgrape:

> Nix <n...@esperi.org.uk> writes:
>
>> It looks like this happens if the TCP connection to the server is
>> somehow interrupted and an attempt to bring it back with
>> cifs_construct_tcon() fails (alas, the actual error gets eaten by
>> cifs_sb_tlink()).
>
> So it didn't look to you like this could be caused by say trying to read
> xattrs that you currently didn't have the right to read? i.e. I just
> wondered if this could be some "not running as root and the mount point
> is restricted" kind of situation, or something...

It's not immediately obvious that that's even a possibility from the
kernel code... but I'd expect it to be possible, intuitively, so I
probably just didn't dig deep enough. (I forgot that getxattr() etc take
paths, not fds...)

> Winsoz, may not be relevant, but were you running bup save as root, or
> as some other user?

Even if you are running as root, that doesn't mean the remote system
wants to let you see everything: a permission denial is perfectly
possible.

--
NULL && (void)

[Winsoz]

I'm sorry for this behavior, I'll put more attention to the text that I post. Maybe something is wrong with google groups interface or I wrong with it. :)

[Winsoz]

Yes, same path. I set up another plan in kup, different folder but below the same mount point of the network share.

The error is always on the root mount point, below that is ok.

And today the error is different: 9 EBADF  (Bad file descriptor). This make me mad, why the error is different every day?!

[Winsoz]

Sorry but last days were busy days at work, now I'll give a try with the exclusion of the xattr usage.

I think that can work and Samba seems not to use the extended attribute (at least the folders that I tested).

[Winsoz]

The error is raised as user as well as root.

In bup save and with my simple program that call listxattr().

[Winsoz]

Il giorno lunedì 20 febbraio 2023 alle 19:03:20 UTC+1 r...@defaultvalue.org ha scritto:

Winsoz <win...@gmail.com> writes:

> about this mail I want to specify that the error shows up during the bup
> save stage. The previously bup index stage goes well.

OK, ideally I'd like to understand the problem better before attempting
any solution, i.e. I'd like to know why that's an error, does it depend
on the server, the mount options, the local kernel or library...

Higher level, is it a bug we're working around somewhere else, or is
this the intended behavior for this arrangement? (Not knowing much
about cifs makes that a bit more difficult.)

I did look through some of the previous reports of trouble with cifs,
and I didn't see any issues with xattrs.

In any case, it might be useful (now or later) to know a bit more about
the arrangement, i.e.

- what kind of machine is running bup (linux, a bsd, macos, ...)
- what's the version of python-pyxattr
- what's the cifs server like, etc.
- if the server is samba, what's the remote filesystem like (ext4?)

Oh, and as a local hack, if this is the only problem, I suppose you
could consider just adding

xattr = None

at the top-level, after the block at the top of metadata.py that checks
for xattr support, i.e. the block guarded by

if sys.platform.startswith('linux'):

That's it. As I suppose the xattr = None at https://github.com/bup/bup/blob/0.32.x/lib/bup/metadata.py#L42

is a sufficient workaround. If this doesn't deteriorate my backups, for now is enough.

[Rob Browning]

Winsoz <win...@gmail.com> writes:

> Yes, same path. I set up another plan in kup, different folder but below
> the same mount point of the network share.
> The error is always on the root mount point, below that is ok.
> And today the error is different: 9 EBADF (Bad file descriptor). This make
> me mad, why the error is different every day?!

Was that via bup or your test program?

And no idea if it's relevant, but I noticed:
https://serverfault.com/questions/948625/samba-cifs-mount-error9-bad-file-descriptor

[Winsoz]

Same error code for bup and my test program.