[WIP/PATCH v2] Expose metadata through bup-fuse

[Gabriel Filion]

There is still one problem with the current code:

* Directories return a hard-coded value (4096 bytes) for their size.

Signed-off-by: Gabriel Filion <lel...@gmail.com>
---
I've dropped the part about overriding a method in FakeSymlink, since the right
fix was pushed to tmp/pending/meta.

this is still very slow, though, and I wouldn't be comfortable for it to be
accepted as-is. But I'm sending it to the list for comments.

it's based on the latest rebase of tmp/pending/meta which happened today.

cmd/fuse-cmd.py | 22 +++++++++++++++++-----
lib/bup/vfs.py | 3 +++
2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/cmd/fuse-cmd.py b/cmd/fuse-cmd.py
index 9253a18..0ee800d 100755
--- a/cmd/fuse-cmd.py
+++ b/cmd/fuse-cmd.py
@@ -56,18 +56,30 @@ class BupFs(fuse.Fuse):
def __init__(self, top):
fuse.Fuse.__init__(self)
self.top = top
-
+
+ ns_in_sec = 10**9
+
def getattr(self, path):
log('--getattr(%r)\n' % path)
try:
node = cache_get(self.top, path)
st = Stat()
- st.st_mode = node.mode
st.st_nlink = node.nlinks()
st.st_size = node.size()
- st.st_mtime = node.mtime
- st.st_ctime = node.ctime
- st.st_atime = node.atime
+ meta = node.metadata()
+ if meta:
+ st.st_uid = meta.uid
+ st.st_gid = meta.gid
+ st.st_mode = meta.mode
+ st.st_mtime = meta.mtime / self.ns_in_sec
+ st.st_ctime = meta.ctime / self.ns_in_sec
+ st.st_atime = meta.atime / self.ns_in_sec
+ st.st_rdev = meta.rdev
+ else:
+ st.st_mode = node.mode
+ st.st_mtime = node.mtime
+ st.st_ctime = node.ctime
+ st.st_atime = node.atime
return st
except vfs.NoSuchFile:
return -errno.ENOENT
diff --git a/lib/bup/vfs.py b/lib/bup/vfs.py
index 9d6267e..f090339 100644
--- a/lib/bup/vfs.py
+++ b/lib/bup/vfs.py
@@ -435,6 +435,9 @@ class Dir(Node):
else:
self._subs[name] = File(self, name, mode, sha, bupmode)

+ def size(self):
+ return 4096 # Very unscientific :\
+
def metadata(self):
"""Return this Dir's Metadata() object, if any."""
self._populate_metadata()
--
1.7.10

[Oei, Yung-Chin]

On 24 September 2012 05:47, Gabriel Filion <lel...@gmail.com> wrote:
> There is still one problem with the current code:
>
> * Directories return a hard-coded value (4096 bytes) for their size.

What kind of problems do you get from that? I think I've seen other
VFSes just return 0 size for directories. I have some cifs-mounts that
do that, and have not had practical problems with that so far.

I would perhaps say that I'd prefer 0 for a hard-coded value, because
that makes it clear it's a bogus value, whereas 4k could possibly be a
real value. But if you want something closer to the behaviour of
"proper" FSes, would it make sense instead to return the size of the
.bupm file?

> this is still very slow, though, and I wouldn't be comfortable for it to be
> accepted as-is. But I'm sending it to the list for comments.

Thanks, I'll give it a spin!

Btw, I think you mentioned you were looking at xattr/acl support too?
Would you have a use for an extra pair of hands (well, brains, but
that sounds pretentious) for that?

YC

[Gabriel Filion]

On 2012-09-24 07:43, Oei, Yung-Chin wrote:
> On 24 September 2012 05:47, Gabriel Filion <lel...@gmail.com> wrote:
>> There is still one problem with the current code:
>>
>> * Directories return a hard-coded value (4096 bytes) for their size.
>
> What kind of problems do you get from that?

the actual issue here was that I didn't know how to approach calculating
that number in a more reasonable way. so mentioning that here helped to
stimulate discussion ;)

> I think I've seen other
> VFSes just return 0 size for directories. I have some cifs-mounts that
> do that, and have not had practical problems with that so far.
>
> I would perhaps say that I'd prefer 0 for a hard-coded value, because
> that makes it clear it's a bogus value, whereas 4k could possibly be a
> real value.

right. that's a good point.

> But if you want something closer to the behaviour of
> "proper" FSes, would it make sense instead to return the size of the
> .bupm file?

could be an interesting implementation. is that ok with others? if it
is, I'll add this for my next rerun of the patch.

>> this is still very slow, though, and I wouldn't be comfortable for it to be
>> accepted as-is. But I'm sending it to the list for comments.
>
> Thanks, I'll give it a spin!
>
> Btw, I think you mentioned you were looking at xattr/acl support too?

yes I was/am. I've found methods of the fuse.Fuse class (getxattr(),
setxattr()) that we could implement to expose xattrs, but I'm not sure
whether acls are possible via FUSE.

I've had a very tough time finding any documentation about FUSE that
made any sense :( there are some examples out there that show how to
implement an FS and some very rare that show how to implement
get/setxattr, but I can't find proper documentation for the different
possible methods.

normally adding xattr support shouldn't be too tough and since the
Metadata objects are cached in each node (e.g. the damage is already
done for exposing simple metadata) it shouldn't have too much of a
negative impact on what's currently happening with my patch.

IIRC last time I had resorted to reading the python fuse layer to see
what methods were available, and that's how I saw there weren't any for
acls..

> Would you have a use for an extra pair of hands (well, brains, but
> that sounds pretentious) for that?

of course ;)

I'll try and get my hands on the xattr exposure soon.

I'm also trying to find a way to expose metadata through "bup ls"/ "bup
ftp ls" by reimplementing as less functionality as possible :)

the final step (could be done in parallel actually... it's final because
I think it has the lowest priority in my mind) would be to expose
metadata via "bup web", too. For this one, we'll probably simply re-use
the stuff from this patch and simply add some markup to hold the new
information.

I think that would make all front ends expose metadata, I hope I'm not
forgetting one :)

if you're able to implement any of the above points before me, then
we'll be advancing :)

we can also coordinate either here or on IRC.

--
Gabriel Filion

[Oei, Yung-Chin]

On 25 September 2012 23:41, Gabriel Filion <lel...@gmail.com> wrote:
>> Thanks, I'll give it a spin!
>>
>> Btw, I think you mentioned you were looking at xattr/acl support too?
>
> yes I was/am. I've found methods of the fuse.Fuse class (getxattr(),
> setxattr()) that we could implement to expose xattrs, but I'm not sure
> whether acls are possible via FUSE.

Sorry, I got sufficiently side-tracked looking at other bits that I
still haven't played with this.

You're right, there's really incredibly little documentation for fuse,
considering that it's such a widely used project.
Just in case it's useful, I've come across some clues about acl's in
the ntfs-3g docs. Specifically, there's a line saying "This has been
fixed in cooperation with the fuse project." But that's _all_ I can
find about acl's at all, without digging into the code...
http://b.andre.pagesperso-orange.fr/permissions.html#posixacls

More to follow...
YC

[Gabriel Filion]

On 2012-10-04 10:30, Oei, Yung-Chin wrote:
> On 25 September 2012 23:41, Gabriel Filion <lel...@gmail.com> wrote:
>>> Thanks, I'll give it a spin!
>>>
>>> Btw, I think you mentioned you were looking at xattr/acl support too?
>>
>> yes I was/am. I've found methods of the fuse.Fuse class (getxattr(),
>> setxattr()) that we could implement to expose xattrs, but I'm not sure
>> whether acls are possible via FUSE.
>

> You're right, there's really incredibly little documentation for fuse,
> considering that it's such a widely used project.
> Just in case it's useful, I've come across some clues about acl's in
> the ntfs-3g docs. Specifically, there's a line saying "This has been
> fixed in cooperation with the fuse project." But that's _all_ I can
> find about acl's at all, without digging into the code...
> http://b.andre.pagesperso-orange.fr/permissions.html#posixacls

ah interesting. I knew there must be an implementation out there, but
didn't find one :)

hmm, in src/ntfs-3g.c, I see:

107 /* ACLS may be checked by kernel (requires a fuse patch) or
here */
108 #define KERNELACLS ((HPERMSCONFIG > 6) & (HPERMSCONFIG < 10))

If I'm reading ntfs-3g's code right (which might not be the case since
my experience with C is limited), then ACLs are implemented as checks in
calls to file and dir access (so the fuse layer ensures ACLs are
respected without exposing them):

3249 static struct fuse_operations ntfs_3g_ops = {
[...]
3287 #if !KERNELPERMS | (POSIXACLS & !KERNELACLS)
3288 .access = ntfs_fuse_access,
3289 .opendir = ntfs_fuse_opendir,
3290 #endif

I'd have to test ntfs3g with ACLs to see if what I'm saying about ACLs
not being exposed is right or not.

--
Gabriel Filion