Serious Problem with Extended Attributes on Snow Leopard

[Gravity]

Hello,

I'm Martin one of the developers of Gravity Applications.

I want to start a discussion about a serious problem we discovered
regarding extended attributes on Snow Leopard.

In Leopard, when an application wrote a file, the OS preserved any
extended attributes of that particular file.
We recently found out that this is no longer the case on Snow Leopard.
In Snow Leopard it seems to be the responsibility of the application
to preserve extended attributes.
The problem is that most application don't preserve them.

Apple seems to have changed some internal bits of Spotlight and any
tags that have been applied before
are preserved when the index information for that file is updated.
But now, only the Spotlight index knows about the associated tags. The
extended file attributes are no longer present.

This leads to a couple of serious problems:

1. Inconsistent behavior
If the user copies a file with associated tags, those tags are only
copied when the extended attributes are still present.

2. Potential data loss
If the Spotlight index dies completely and the user reindexes the hard
drive than tagging information for files with lost extended attributes
are also lost.

The OpenMeta backup mechanism helps for the potential data loss as
long as the user hasn't moved files around.

Our proposed solution:

Today we released the first beta of Tags 2 which is backed by OpenMeta
(as that was our intention for the 2.0 release).
But in order to address the problems above we adopted our code to
additionally write tagging information in Spotlight Comments.

We found out that searching for Spotlight Comments is pretty fast in
Snow Leopard. Looks like Apple is doing a better job with indexing
Spotlight Comments now.
The first beta of Tags 2 is now running completely on Spotlight
Comments (but for now still writes tagging information in extended
attributes as well - but not using them).
So far we are pretty happy with the performance results and I
encourage you to give it a try for yourself.

Our goal is to find a common solution to provide the user with the
best possible experience and a cross-application tagging solution.
We are open for any serious solution but right now we think that using
Spotlight Comments is probably the best solution on Snow Leopard.

best regards,
Martin

Download Link of Tags 2 Beta: http://gravityapps.com/downloads/Tags2_Beta.dmg

[Johannes Hoffart]

Hi Martin,

I have just tested this, both with TextMate and vi, and the extended
attributes are preserved - I do not believe vi was designed to do
this, so it seems your information is only partly corret :). Could you
please specify which applications do not preserve them (excpet the
known Adobe ones?)

Thanks,
Johannes

[Gravity]

Hi Johannes,

You are right. TextMate is doing the right thing and command line
applications as well.
But TextEdit, Pages, Keynote, XCode, Coda, ... don't.

So the low level APIs are doing the right thing but higher level APIs
probably not.

best,
Martin

[Johannes Hoffart]

Hi Martin,

you are right. I believe this is a bug in the OS, we should definitely
report this. It's not like OpenMeta is the only application that is
affected by extended attributes going missing ... they are meant to be
permanent after all!

Did you already open a bug report regarding this in radar?

Regards,
Johannes

[Tom Andersen]

Thanks for the additional info. 

I filed a bug, a tech support incident. Both were addressed. But it looks like this issue will not be fixed (but maybe if enough people complain?)

I include here some more facts:

DTS wrote back to me:

"

At this point I can see no indication that 7238215 will be fixed in a Snow Leopard software update.  That's not to say it won't happen, just that it looks unlikely given the current state of affairs.

"

7238215 is the bug where I pointed out that xattrs are erased when using cocoa apps like Preview, Pages, TextEdit, etc.  I hope the AppKit team re thinks this. 

Detailed technical digging: (From DTS who were very helpful):

You can see from the stacks below that any app that uses the standard Cocoa NSDocument class will start erasing xattrs:

Now, back to TextEdit, whose change of behaviour is quite

interesting.  I ran TextEdit under both 10.5.x and 10.6 and looked

at the backtrace of a save.  Here's the results for 10.5.x:

#0  0x970b0a60 in rename ()

#1  0x95c95526 in VolFSMount::rename ()

#2  0x95c9f163 in VolFSMount::moveandrename ()

#3  0x95c9ed18 in FSReplaceObject ()

#4  0x95c9df45 in FSPathReplaceObject ()

#5  0x930ba8c1 in +[NSDocument

_finishWritingFileAtPath:byTakingContentsFromFileAtPath:addingAttributes:usingTemporaryDirectoryAtPath:keepingBackupFile:error

:] ()

#6  0x9307c5d4 in -[NSDocument

_writeSafelyToURL:ofType:forSaveOperation:error:] ()

#7  0x9307c2d6 in -[NSDocument

writeSafelyToURL:ofType:forSaveOperation:error:] ()

[...]

and here's the result for 10.6:

#0  0x921f095b in rename ()

#1  0x9695d593 in _RenameURL ()

#2  0x9695ed7a in _URLReplaceObject ()

#3  0x98205066 in -[NSFileManager

replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error

:] ()

#4  0x96fd979a in +[NSFileWrapper(NSInternal)

_finishWritingToURL:byTakingContentsFromItemAtURL:addingAttributes:usingTemporaryDirectoryAtURL:keepingBackupFile:error

:] ()

#5  0x96fb0d71 in -[NSDocument

_writeSafelyToURL:ofType:forSaveOperation:error:] ()

#6  0x96fafdbe in -[NSDocument

writeSafelyToURL:ofType:forSaveOperation:error:] ()

[...]

As you can see, TextEdit uses -[NSDocument

writeSafelyToURL:ofType:forSaveOperation:error:] to save its

documents, and in 10.6 that method was rewritten to /not/ call the

File Manager routine FSPathReplaceObject.  This is the reason why

extended attributes are not being preserved on 10.6:

FSPathReplaceObject was specifically designed to support this sort

of thing, and it seems that the AppKit equivalent code was not.

I would definitely classify this as a bug but, just to be sure, I'm

going to discuss this with AppKit engineering.

I think that the person working the tech support incident for me inquired with the Appkit team, and was told that they are not changing the code for 10.6. 

What to do? I don't know. For OpenMeta I have a solution that will help quite a bit, in that mds seems to replace erased xattrs if they are in the right format in the spotlight DB: So that's what I am doing for OpenMeta. Tags will be filed under kMDItemOMUserTags instead of kOMUserTags. I don't like it, but I could not figure any way around it. 

More from dts:

And finally, let's look at the different behaviour for different

attributes.  You wrote:

I have found that all skim xattrs,

de.codingmonkeys.seestate:

org.openmetainfo:kOMManaged:

anyKeyYouMakeUp

etc, etc

get erased, while

com.apple.metadata:kMDItemStarRating

com.apple.metadata:kMDItemWhereFroms

and I assume resource forks, (although I did not test that) and

some other apple ones are saved.

First up, the resource fork is definitely not saved.

1. Start with a file with a resource fork.

$ ls -l@ file.txt

-rw-r--r--@ 1 xxxx  xxxx 16 22 Sep 16:20 file.txt

com.apple.ResourceFork 368

com.apple.TextEncoding 11

com.apple.metadata:kMDItemStarRating 43

2. Open the file in TextEdit, modify it, save the changes, and close

it again.

3. The resource fork is now gone.

$ ls -l@ file.txt

-rw-r--r--@ 1xxxx  xxxx 17 22 Sep 16:21 file.txt

com.apple.TextEncoding 11

com.apple.metadata:kMDItemStarRating 43

Secondly, it's not actually Cocoa that's preserving the Spotlight

attributes.  Check out the following:

1. Start with a file with a star rating:

$ ls -l@ file.txt

-rw-r--r--@ 1xxxx  xxxx 16 22 Sep 16:25 file.txt

com.apple.TextEncoding 11

com.apple.metadata:kMDItemStarRating 43

2. Run the SetXAttr.d DTrace script:

$ cat SetXAttr.d

#! /usr/sbin/dtrace -q -s

syscall::setxattr:entry

/ ! self->trace /

{

 self->trace = 1;

 self->path = arg0;

 self->attr = arg1;

}

syscall::setxattr:return

/ self->trace /

{

 path = copyinstr(self->path);

 attr = copyinstr(self->attr);

 printf("%s: setxattr('%s', '%s') -> %d\n", execname, path, attr,

errno);

 self->trace = 0;

}

syscall::fsetxattr:entry

/ ! self->trace /

{

 self->trace = 1;

 self->attr = arg1;

}

syscall::fsetxattr:return

/ self->trace /

{

 attr = copyinstr(self->attr);

 printf("%s: fsetxattr(?, '%s') -> %d\n", execname, attr, errno);

 self->trace = 0;

}

$ sudo ./SetXAttr.d

3. Open the file in TextEdit, modify it, save the changes, and close

it again.

4. Look at the output of the script:

TextEdit: setxattr('/private/var/folders/Ie/[...]/file.txt',

'com.apple.TextEncoding') -> 0

mds: fsetxattr(?, 'com.apple.metadata:kMDItemStarRating') -> 0

^C

As you can see, TextEdit only saves one extended attribute to the

file, namely "com.apple.TextEncoding".  The

"com.apple.metadata:kMDItemStarRating" is being set by <x-man-page://8/mds

, aka the Spotlight daemon process.  So it seems that, rather than

Cocoa preserving these extended attributes, Spotlight is handling

them as a special case.

I investigate this facility in depth as well.  As you're probably

aware, Spotlight watches the file system for events of interest to

it, and uses those events to update its index.  In this case

Spotlight is specifically looking for 'safe save' operations (that

is, renames which replace a file at the new path).  When it sees

one, it copies its attributes from its internal records of the

original file to the replacement file.

You can verify that this has nothing to do with Cocoa by simply

performing the rename from Terminal:

1. Start with a file with a star rating.

$ ls -l@ file.txt

-rw-r--r--@ 1xxxx  xxxx 13 22 Sep 16:56 file.txt

com.apple.metadata:kMDItemStarRating 43

2. Create a replacement file.

$ cat > file-bu.txt

this is a test

^D

3. Verify that the attributes are the way we expect.

$ ls -l@ file*.txt

-rw-r--r--  1xxxx  xxxx 15 22 Sep 17:28 file-bu.txt

-rw-r--r--@ 1xxxx  xxxx 13 22 Sep 16:56 file.txt

com.apple.metadata:kMDItemStarRating 43

4. Do the rename.

$ mv file-bu.txt file.txt

5. Check the attributes of the final; somehow it has retained its

star rating.

$ ls -l@ file*.txt

-rw-r--r--@ 1xxxx  xxxx 15 22 Sep 17:28 file.txt

com.apple.metadata:kMDItemStarRating 43

--Tom

[Michael Tsai]

On Oct 9, 2009, at 11:06 AM, Tom Andersen wrote:

> I filed a bug, a tech support incident.

I've filed a bug as well. I think NSFileManager should behave the same
way that FSReplaceObject does.

> For OpenMeta I have a solution that will help quite a bit, in that
> mds seems to replace erased xattrs if they are in the right format
> in the spotlight DB: So that's what I am doing for OpenMeta. Tags
> will be filed under kMDItemOMUserTags instead of kOMUserTags.

kMDItemOMUserTags? Is that a typo?

Your solution seems to address searchability but not data exchange. It
seems like this change will break compatibility with other
applications that are using "com.apple.metadata:kOMUserTags".

My plan for EagleFiler is to continue using
"com.apple.metadata:kOMUserTags" and "org.openmetainfo:kOMUserTags",
using the timestamp to see which is newer, and to restore the xattrs
if they get deleted.

--Michael

[Tom Andersen]

Michael,

Filing another bug should help, but I believe dts when they tell me
that its unlikely to change for 10.6.

kMDItem* items get preserved when saving files - so really I have no
choice.
It seems that it works pretty well for compatibility - I keep a look
out for files that have older tag formats and update them, I also
write out tags in the old format.

--Tom

[Michael Tsai]

On Oct 12, 2009, at 11:10 AM, Tom Andersen wrote:

> kMDItem* items get preserved when saving files - so really I have no
> choice.
> It seems that it works pretty well for compatibility - I keep a look
> out for files that have older tag formats and update them, I also
> write out tags in the old format.

So when you said "instead of" you meant "in addition to"? How are you
resolving conflicts, i.e. when the attributes have different values?

--Michael

[Gravity]

@Michael: You can't check the xattr "com.apple.metadata:kOMUserTags"
and "org.openmetainfo:kOMUserTags" for which is newer. Both will be
removed when files are being saved.

I can confirm that "kMDItemOMUserTags" works with my tests so far.
Great find!

But I'm a bit afraid of relying on the Spotlight process to recreate
tagging information.
Sounds like using something undocumented for something undocumented to
work.

You haven't responded to the Spotlight comment thing.
I don't say that it is a perfect solution as it has some drawbacks as
well (losing information in terminal apps for example).
But at least it is an official solution.

Martin

[Michael Tsai]

On Oct 12, 2009, at 11:26 AM, Gravity wrote:

> @Michael: You can't check the xattr "com.apple.metadata:kOMUserTags"
> and "org.openmetainfo:kOMUserTags" for which is newer. Both will be
> removed when files are being saved.

Yes, and both may be added back by other applications. So unless you
want to overwrite possibly newer data, you should to check all three
locations now. (Is there a timestamp for kMDItemOMUserTags?)

> You haven't responded to the Spotlight comment thing.
> I don't say that it is a perfect solution as it has some drawbacks as
> well (losing information in terminal apps for example).
> But at least it is an official solution.

Spotlight comments are still official, but I think they're still
really slow to read and write.

--Michael

[Michael Grant]

On Oct 12, 2009, at 11:39 AM, Michael Tsai wrote:

> Spotlight comments are still official, but I think they're still
> really slow to read and write.

Plus many users were using "Finder comments" years before Spotlight
was even a twinkle in Steve Jobs' eye, and would prefer to keep them
clear for user-level purposes.

Michael

--
<http://www.linkedin.com/in/migrant>
<http://twitter.com/MiGrant>
<http://globalocal.blogspot.com>
<http://eurobubba.com>

I never metadata I didn't like.

[tom.andersen]

I uploaded the OpenMeta code that I am using in Yep 2

It deals with updating things two ways:
1) On launch, it always does a search for items that have kOMuserTags
but no kMDItemOMUserTags, and fixes that. (this is more for initial
upgrade).
2) When getUserTags: is called, I check the dates, and use the newest
tags.

To use the onLaunch thing for updating to kMDItemOMUserTags, you need
to call [OpenMetaBackup upgradeOpenMetaTokMDItemOM].

This seems to work pretty well.

On the comment thing. - I have documented on my computer where setting
a few hundred comments will kill the Finder. Also comments are stored
in the .DS_Store file, then replicated out to the xattrs, then
replicated into the Spotlight DB, when they get out of sync, they tend
to stay out of wack - it is easy to get a situation where the three
places disagree on what the comment is. I think this is because there
is no easy way to tell when to rescan the .DS_Store file into the
xattrs (just a guess). Really the whole spotlight comments (which ARE
really Finder comments) is a mess.

--Tom

[tom.andersen]

In the longer term:
---------------------------
Apple knows that this is a problem with 10.6, but for some reason
(performance or security?) decided to backtrack. Metadata on file
systems is a hot topic, and right now things are quite messy on
Apple's side. HFS+ is getting really really old, and I think that a
replacement for it may come along soon, perhaps even before 10.7? Any
new file system will likely have first class handling of meta data,
including user added metadata. At least that's my best guess at what
they might do next. When I say metadata is messy on Apple's side -
just think of the following places where metadata for a single file is
stored:

File name, file mod date, permissions, and a few other things are
stored 'with the file' on the file system.
Spotlight comments are stored in the .DS_Store sidecar file, along
with other details.
Some things like Finder labels are stored in some HFS+ specific
location.
QuickLook images - definitely metadata in my opinion - available with
an API call.
Quarantine services - buried in some database somewhere.
Launch Services (the stuff that keeps track of what application to
open what files with) - some other database hidden away somewhere.

There are at least 5, and likely more places where they store metadata
about a single file. The whole system is basically a pile of
afterthoughts. A shiny new file system could change all of that. With
the current system, a simple rename of a file may result in many
updates to many systems on the computer, which is error prone and
complicated. A unified metadata system that runs at the file system
level would make things run better and faster.

Looking forward to that - if it happens,

--Tom

[Michael Tsai]

On Oct 13, 2009, at 12:28 PM, tom.andersen wrote:

> I uploaded the OpenMeta code that I am using in Yep 2

Why did you change the keys in org.openmetainfo? We should be able to
keep the format in our domain constant, to maximize compatibility.

--Michael

[Michael Tsai]

On Oct 13, 2009, at 12:29 PM, tom.andersen wrote:

> HFS+ is getting really really old, and I think that a
> replacement for it may come along soon, perhaps even before 10.7?

Apple is not going to replace their filesystem between releases.

> Any
> new file system will likely have first class handling of meta data,
> including user added metadata.

HFS+ already has first-class handling of metadata. The problems we're
encountering are above the filesystem level, in the application
frameworks.

> Quarantine services - buried in some database somewhere.

This info is stored in the extended attributes.

> Launch Services (the stuff that keeps track of what application to
> open what files with) - some other database hidden away somewhere.

The file-specific bindings are stored in the resource fork. The
general bindings are not file metadata.

> The whole system is basically a pile of
> afterthoughts. A shiny new file system could change all of that.

It could also all be changed with HFS+, since it supports extended
attributes.

--Michael

[Rolf Schmolling M.A.]

Hi,
to add my perspective some thoughts and questions: (Note that I am not
a programmer and my understanding is limited)

• For me openmeta is a solution I have been looking for: file-based
tagging and indexing vs. database systems (where one has to put all
stuff into to run into backup-problems for a start…)
• it has been possible to mix different things: ways and places to
look for files, keep them together etc. – basically a very broad
compatibility across no a growing number of applications. So I tag
files while saving, create smart folders (keep things together) in
Finder or within applications like Leap or others. And, after while it
just works…

Now we face the problem of a necessary change/adjustment. Will this
mean compatibility is lost? Need all apps pick up the new code to
work? What about those people not (yet) on SL?

What do others besides ironic think and commit themselves to?

regards,
Rolf

[Neuronical]

As an IT professional for 21 years I have personally had trouble with
the approach of the openMeta implementation since its inception. For a
third party to use another party's namespace (whether it be Apple's or
anyone else's) to accomplish their own objectives, no matter how much
I as a user want those objectives to succeed, is a kludgy approach
that leaves the door wide open for unforeseen problems such as the one
introduced by SL. This concern has already been brought forward by
others but I feel was dismissed almost out-of-hand.

That being said, I applaud the efforts made by those at Ironic and
others who have been trying to bring this extremely useful
functionality to OS X. However, in its current implementation, it's
unreliable, it's just now starting to become convoluted and "patchy",
and any future development going down the same path as has thus far
been taken will, imho, continue to be problematic.

My thoughts on this would not be so negative were it not for the fact
that this effort is attempting to set a standard, meaning that for
cross-program consistency all development houses and independents
should adhere to openMeta when incorporating functionality allowing
users to define their own file-specific metadata. But a standard
should not have a foundation so shaky that the implementation is
continually being modified in reaction to completely uncontrollable
and unforeseeable events. Even if the standard is only *at risk* of
these events occurring it renders the standard unreliable for anyone
to actually come to depend on it to fulfill its intended purpose. As
such, in it's current design, I feel that it's a novelty at best, and
although I dearly wish to, I can't possibly rely on it in any
meaningful way.

I'm not intending to flame anyone's work, or snipe at honest people
making a concerted effort to enrich hundreds of thousands, perhaps
millions of people's experience using OS X. I wish for the effort to
succeed as much as anyone else does. I just wanted to express my
concern regarding the current implementation. Because I want to trust
it to make my life easier, but I just can't.

Dan

[Tom Andersen]

The thing is, xattrs are designed to hold user defined metadata.

It is Apple's bug to erase some metadata like xattrs and not others
like file creation date, finder flags etc.

This big also erases the entire resource fork.

OpenMeta does not use reserved namespaces to store data, but rather
merely mirrors data into spotlight using the only available route.

The reason it's open source is that before it there were about four or
so ways to store tags that were all incompatible, and all used the
Finder comments to store them.

I have been doing the mac programming thing for a while too. People
are basically demanding this kind of functionality. Using comments to
store the tags does not work, and trying to track them in some kind of
db has huge problems. So do the best we can.

--Tom