Account Options

  1. Sign in
The old Google Groups will be going away soon.
Switch to the new Google Groups.
Google Groups Home
« Groups Home
Serious Problem with Extended Attributes on Snow Leopard
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  18 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Gravity  
View profile  
 More options Oct 9 2009, 10:01 am
From: Gravity <martink...@gmail.com>
Date: Fri, 9 Oct 2009 07:01:46 -0700 (PDT)
Local: Fri, Oct 9 2009 10:01 am
Subject: Serious Problem with Extended Attributes on Snow Leopard
Hello,

I'm Martin one of the developers of Gravity Applications.

I want to start a discussion about a serious problem we discovered
regarding extended attributes on Snow Leopard.

In Leopard, when an application wrote a file, the OS preserved any
extended attributes of that particular file.
We recently found out that this is no longer the case on Snow Leopard.
In Snow Leopard it seems to be the responsibility of the application
to preserve extended attributes.
The problem is that most application don't preserve them.

Apple seems to have changed some internal bits of Spotlight and any
tags that have been applied before
are preserved when the index information for that file is updated.
But now, only the Spotlight index knows about the associated tags. The
extended file attributes are no longer present.

This leads to a couple of serious problems:

1. Inconsistent behavior
If the user copies a file with associated tags, those tags are only
copied when the extended attributes are still present.

2. Potential data loss
If the Spotlight index dies completely and the user reindexes the hard
drive than tagging information for files with lost extended attributes
are also lost.

The OpenMeta backup mechanism helps for the potential data loss as
long as the user hasn't moved files around.

Our proposed solution:

Today we released the first beta of Tags 2 which is backed by OpenMeta
(as that was our intention for the 2.0 release).
But in order to address the problems above we adopted our code to
additionally write tagging information in Spotlight Comments.

We found out that searching for Spotlight Comments is pretty fast in
Snow Leopard. Looks like Apple is doing a better job with indexing
Spotlight Comments now.
The first beta of Tags 2 is now running completely on Spotlight
Comments (but for now still writes tagging information in extended
attributes as well - but not using them).
So far we are pretty happy with the performance results and I
encourage you to give it a try for yourself.

Our goal is to find a common solution to provide the user with the
best possible experience and a cross-application tagging solution.
We are open for any serious solution but right now we think that using
Spotlight Comments is probably the best solution on Snow Leopard.

best regards,
Martin

Download Link of Tags 2 Beta: http://gravityapps.com/downloads/Tags2_Beta.dmg


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Johannes Hoffart  
View profile  
 More options Oct 9 2009, 10:31 am
From: Johannes Hoffart <johan...@nudgenudge.eu>
Date: Fri, 9 Oct 2009 16:31:52 +0200
Local: Fri, Oct 9 2009 10:31 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
Hi Martin,

I have just tested this, both with TextMate and vi, and the extended  
attributes are preserved - I do not believe vi was designed to do  
this, so it seems your information is only partly corret :). Could you  
please specify which applications do not preserve them (excpet the  
known Adobe ones?)

Thanks,
Johannes

Am 09.10.2009 um 16:01 schrieb Gravity:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gravity  
View profile  
 More options Oct 9 2009, 10:35 am
From: Gravity <martink...@gmail.com>
Date: Fri, 9 Oct 2009 07:35:23 -0700 (PDT)
Local: Fri, Oct 9 2009 10:35 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
Hi Johannes,

You are right. TextMate is doing the right thing and command line
applications as well.
But TextEdit, Pages, Keynote, XCode, Coda, ... don't.

So the low level APIs are doing the right thing but higher level APIs
probably not.

best,
Martin

On Oct 9, 4:31 pm, Johannes Hoffart <johan...@nudgenudge.eu> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Johannes Hoffart  
View profile  
 More options Oct 9 2009, 10:40 am
From: Johannes Hoffart <johan...@nudgenudge.eu>
Date: Fri, 9 Oct 2009 16:40:07 +0200
Local: Fri, Oct 9 2009 10:40 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
Hi Martin,

you are right. I believe this is a bug in the OS, we should definitely  
report this. It's not like OpenMeta is the only application that is  
affected by extended attributes going missing ... they are meant to be  
permanent after all!

Did you already open a bug report regarding this in radar?

Regards,
Johannes

Am 09.10.2009 um 16:35 schrieb Gravity:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Andersen  
View profile  
 More options Oct 9 2009, 11:06 am
From: Tom Andersen <tom.ander...@gmail.com>
Date: Fri, 9 Oct 2009 11:06:25 -0400
Local: Fri, Oct 9 2009 11:06 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard

Thanks for the additional info.

I filed a bug, a tech support incident. Both were addressed. But it  
looks like this issue will not be fixed (but maybe if enough people  
complain?)

I include here some more facts:
DTS wrote back to me:
"
At this point I can see no indication that 7238215 will be fixed in a  
Snow Leopard software update.  That's not to say it won't happen, just  
that it looks unlikely given the current state of affairs.
"
7238215 is the bug where I pointed out that xattrs are erased when  
using cocoa apps like Preview, Pages, TextEdit, etc.  I hope the  
AppKit team re thinks this.

Detailed technical digging: (From DTS who were very helpful):

You can see from the stacks below that any app that uses the standard  
Cocoa NSDocument class will start erasing xattrs:

I think that the person working the tech support incident for me  
inquired with the Appkit team, and was told that they are not changing  
the code for 10.6.

What to do? I don't know. For OpenMeta I have a solution that will  
help quite a bit, in that mds seems to replace erased xattrs if they  
are in the right format in the spotlight DB: So that's what I am doing  
for OpenMeta. Tags will be filed under kMDItemOMUserTags instead of  
kOMUserTags. I don't like it, but I could not figure any way around it.

More from dts:

--Tom

On 2009-10-09, at 10:40 AM, Johannes Hoffart wrote:

...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Tsai  
View profile  
 More options Oct 11 2009, 2:45 pm
From: Michael Tsai <li...@mjtsai.com>
Date: Sun, 11 Oct 2009 14:45:37 -0400
Local: Sun, Oct 11 2009 2:45 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 9, 2009, at 11:06 AM, Tom Andersen wrote:

> I filed a bug, a tech support incident.

I've filed a bug as well. I think NSFileManager should behave the same  
way that FSReplaceObject does.

> For OpenMeta I have a solution that will help quite a bit, in that  
> mds seems to replace erased xattrs if they are in the right format  
> in the spotlight DB: So that's what I am doing for OpenMeta. Tags  
> will be filed under kMDItemOMUserTags instead of kOMUserTags.

kMDItemOMUserTags? Is that a typo?

Your solution seems to address searchability but not data exchange. It  
seems like this change will break compatibility with other  
applications that are using "com.apple.metadata:kOMUserTags".

My plan for EagleFiler is to continue using  
"com.apple.metadata:kOMUserTags" and "org.openmetainfo:kOMUserTags",  
using the timestamp to see which is newer, and to restore the xattrs  
if they get deleted.

--Michael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Andersen  
View profile  
 More options Oct 12 2009, 11:10 am
From: Tom Andersen <tom.ander...@gmail.com>
Date: Mon, 12 Oct 2009 11:10:32 -0400
Local: Mon, Oct 12 2009 11:10 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
Michael,

Filing another bug should help, but I believe dts when they tell me  
that its unlikely to change for 10.6.

kMDItem* items get preserved when saving files - so really I have no  
choice.
It seems that it works pretty well for compatibility - I keep a look  
out for files that have older tag formats and update them, I also  
write out tags in the old format.

--Tom

On 2009-10-11, at 2:45 PM, Michael Tsai wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Tsai  
View profile  
 More options Oct 12 2009, 11:15 am
From: Michael Tsai <li...@mjtsai.com>
Date: Mon, 12 Oct 2009 11:15:49 -0400
Local: Mon, Oct 12 2009 11:15 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 12, 2009, at 11:10 AM, Tom Andersen wrote:

> kMDItem* items get preserved when saving files - so really I have no
> choice.
> It seems that it works pretty well for compatibility - I keep a look
> out for files that have older tag formats and update them, I also
> write out tags in the old format.

So when you said "instead of" you meant "in addition to"? How are you  
resolving conflicts, i.e. when the attributes have different values?

--Michael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gravity  
View profile  
 More options Oct 12 2009, 11:26 am
From: Gravity <martink...@gmail.com>
Date: Mon, 12 Oct 2009 08:26:49 -0700 (PDT)
Local: Mon, Oct 12 2009 11:26 am
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
@Michael: You can't check the xattr "com.apple.metadata:kOMUserTags"
and "org.openmetainfo:kOMUserTags" for which is newer. Both will be
removed when files are being saved.

I can confirm that "kMDItemOMUserTags" works with my tests so far.
Great find!

But I'm a bit afraid of relying on the Spotlight process to recreate
tagging information.
Sounds like using something undocumented for something undocumented to
work.

You haven't responded to the Spotlight comment thing.
I don't say that it is a perfect solution as it has some drawbacks as
well (losing information in terminal apps for example).
But at least it is an official solution.

Martin


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Tsai  
View profile  
 More options Oct 12 2009, 12:39 pm
From: Michael Tsai <li...@mjtsai.com>
Date: Mon, 12 Oct 2009 12:39:07 -0400
Local: Mon, Oct 12 2009 12:39 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 12, 2009, at 11:26 AM, Gravity wrote:

> @Michael: You can't check the xattr "com.apple.metadata:kOMUserTags"
> and "org.openmetainfo:kOMUserTags" for which is newer. Both will be
> removed when files are being saved.

Yes, and both may be added back by other applications. So unless you  
want to overwrite possibly newer data, you should to check all three  
locations now. (Is there a timestamp for kMDItemOMUserTags?)

> You haven't responded to the Spotlight comment thing.
> I don't say that it is a perfect solution as it has some drawbacks as
> well (losing information in terminal apps for example).
> But at least it is an official solution.

Spotlight comments are still official, but I think they're still  
really slow to read and write.

--Michael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Grant  
View profile  
 More options Oct 12 2009, 12:51 pm
From: Michael Grant <mgr...@gmail.com>
Date: Mon, 12 Oct 2009 11:51:24 -0500
Local: Mon, Oct 12 2009 12:51 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 12, 2009, at 11:39 AM, Michael Tsai wrote:

> Spotlight comments are still official, but I think they're still
> really slow to read and write.

Plus many users were using "Finder comments" years before Spotlight  
was even a twinkle in Steve Jobs' eye, and would prefer to keep them  
clear for user-level purposes.

Michael

--
<http://www.linkedin.com/in/migrant>
<http://twitter.com/MiGrant>
<http://globalocal.blogspot.com>
<http://eurobubba.com>

I never metadata I didn't like.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
tom.andersen  
View profile  
 More options Oct 13 2009, 12:28 pm
From: "tom.andersen" <tom.ander...@gmail.com>
Date: Tue, 13 Oct 2009 09:28:21 -0700 (PDT)
Local: Tues, Oct 13 2009 12:28 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
I uploaded the OpenMeta code that I am using in Yep 2

It deals with updating things two ways:
1) On launch, it always does a search for items that have kOMuserTags
but no kMDItemOMUserTags, and fixes that. (this is more for initial
upgrade).
2) When getUserTags: is called, I check the dates, and use the newest
tags.

To use the onLaunch thing for updating to kMDItemOMUserTags, you need
to call [OpenMetaBackup upgradeOpenMetaTokMDItemOM].

This seems to work pretty well.

On the comment thing. - I have documented on my computer where setting
a few hundred comments will kill the Finder. Also comments are stored
in the .DS_Store file, then replicated out to the xattrs, then
replicated into the Spotlight DB, when they get out of sync, they tend
to stay out of wack - it is easy to get a situation where the three
places disagree on what the comment is. I think this is because there
is no easy way to tell when to rescan the .DS_Store file into the
xattrs (just a guess). Really the whole spotlight comments (which ARE
really Finder comments) is a mess.

--Tom

On Oct 12, 12:51 pm, Michael Grant <mgr...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
tom.andersen  
View profile  
 More options Oct 13 2009, 12:29 pm
From: "tom.andersen" <tom.ander...@gmail.com>
Date: Tue, 13 Oct 2009 09:29:42 -0700 (PDT)
Local: Tues, Oct 13 2009 12:29 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
In the longer term:
---------------------------
Apple knows that this is a problem with 10.6, but for some reason
(performance or security?) decided to backtrack. Metadata on file
systems is a hot topic, and right now things are quite messy on
Apple's side. HFS+ is getting really really old, and I think that a
replacement for it may come along soon, perhaps even before 10.7? Any
new file system will likely have first class handling of meta data,
including user added metadata. At least that's my best guess at what
they might do next. When I say metadata is messy on Apple's side -
just think of the following places where metadata for a single file is
stored:

File name, file mod date, permissions, and a few other things are
stored 'with the file' on the file system.
Spotlight comments are stored in the .DS_Store sidecar file, along
with other details.
Some things like Finder labels are stored in some HFS+ specific
location.
QuickLook images - definitely metadata in my opinion - available with
an API call.
Quarantine services - buried in some database somewhere.
Launch Services (the stuff that keeps track of what application to
open what files with) - some other database hidden away somewhere.

There are at least 5, and likely more places where they store metadata
about a single file. The whole system is basically a pile of
afterthoughts. A shiny new file system could change all of that. With
the current system, a simple rename of a file may result in many
updates to many systems on the computer, which is error prone and
complicated. A unified metadata system that runs at the file system
level would make things run better and faster.

Looking forward to that - if it happens,

--Tom


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Tsai  
View profile  
 More options Oct 13 2009, 2:50 pm
From: Michael Tsai <li...@mjtsai.com>
Date: Tue, 13 Oct 2009 14:50:24 -0400
Local: Tues, Oct 13 2009 2:50 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 13, 2009, at 12:28 PM, tom.andersen wrote:

> I uploaded the OpenMeta code that I am using in Yep 2

Why did you change the keys in org.openmetainfo? We should be able to  
keep the format in our domain constant, to maximize compatibility.

--Michael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Tsai  
View profile  
 More options Oct 13 2009, 2:57 pm
From: Michael Tsai <li...@mjtsai.com>
Date: Tue, 13 Oct 2009 14:57:30 -0400
Local: Tues, Oct 13 2009 2:57 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
On Oct 13, 2009, at 12:29 PM, tom.andersen wrote:

> HFS+ is getting really really old, and I think that a
> replacement for it may come along soon, perhaps even before 10.7?

Apple is not going to replace their filesystem between releases.

> Any
> new file system will likely have first class handling of meta data,
> including user added metadata.

HFS+ already has first-class handling of metadata. The problems we're  
encountering are above the filesystem level, in the application  
frameworks.

> Quarantine services - buried in some database somewhere.

This info is stored in the extended attributes.

> Launch Services (the stuff that keeps track of what application to
> open what files with) - some other database hidden away somewhere.

The file-specific bindings are stored in the resource fork. The  
general bindings are not file metadata.

> The whole system is basically a pile of
> afterthoughts. A shiny new file system could change all of that.

It could also all be changed with HFS+, since it supports extended  
attributes.

--Michael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Rolf Schmolling M.A.  
View profile  
 More options Oct 13 2009, 4:09 pm
From: "Rolf Schmolling M.A." <rolf.schmoll...@googlemail.com>
Date: Tue, 13 Oct 2009 22:09:24 +0200
Local: Tues, Oct 13 2009 4:09 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
Hi,
to add my perspective some thoughts and questions: (Note that I am not  
a programmer and my understanding is limited)

• For me openmeta is a solution I have been looking for: file-based  
tagging and indexing vs. database systems (where one has to put all  
stuff into to run into backup-problems for a start…)
• it has been possible to mix different things: ways and places to  
look for files, keep them together etc. – basically a very broad  
compatibility across no a growing number of applications. So I tag  
files while saving, create smart folders (keep things together) in  
Finder or within applications like Leap or others. And, after while it  
just works…

Now we face the problem of a necessary change/adjustment. Will this  
mean compatibility is lost? Need all apps pick up the new code to  
work? What about those people not (yet) on SL?

What do others besides ironic think and commit themselves to?

regards,
Rolf

Am 12.10.2009 um 17:10 schrieb Tom Andersen:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Neuronical  
View profile  
 More options Oct 13 2009, 7:50 pm
From: Neuronical <dan.dob...@gmail.com>
Date: Tue, 13 Oct 2009 16:50:49 -0700 (PDT)
Local: Tues, Oct 13 2009 7:50 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
As an IT professional for 21 years I have personally had trouble with
the approach of the openMeta implementation since its inception. For a
third party to use another party's namespace (whether it be Apple's or
anyone else's) to accomplish their own objectives, no matter how much
I as a user want those objectives to succeed, is a kludgy approach
that leaves the door wide open for unforeseen problems such as the one
introduced by SL. This concern has already been brought forward by
others but I feel was dismissed almost out-of-hand.

That being said, I applaud the efforts made by those at Ironic and
others who have been trying to bring this extremely useful
functionality to OS X. However, in its current implementation, it's
unreliable, it's just now starting to become convoluted and "patchy",
and any future development going down the same path as has thus far
been taken will, imho, continue to be problematic.

My thoughts on this would not be so negative were it not for the fact
that this effort is attempting to set a standard, meaning that for
cross-program consistency all development houses and independents
should adhere to openMeta when incorporating functionality allowing
users to define their own file-specific metadata. But a standard
should not have a foundation so shaky that the implementation is
continually being modified in reaction to completely uncontrollable
and unforeseeable events. Even if the standard is only *at risk* of
these events occurring it renders the standard unreliable for anyone
to actually come to depend on it to fulfill its intended purpose. As
such, in it's current design, I feel that it's a novelty at best, and
although I dearly wish to, I can't possibly rely on it in any
meaningful way.

I'm not intending to flame anyone's work, or snipe at honest people
making a concerted effort to enrich hundreds of thousands, perhaps
millions of people's experience using OS X. I wish for the effort to
succeed as much as anyone else does. I just wanted to express my
concern regarding the current implementation. Because I want to trust
it to make my life easier, but I just can't.

Dan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Andersen  
View profile  
 More options Oct 13 2009, 8:42 pm
From: Tom Andersen <tom.ander...@gmail.com>
Date: Tue, 13 Oct 2009 20:42:20 -0400
Local: Tues, Oct 13 2009 8:42 pm
Subject: Re: Serious Problem with Extended Attributes on Snow Leopard
The thing is, xattrs are designed to hold user defined metadata.

It is Apple's bug to erase some metadata like xattrs and not others  
like file creation date, finder flags etc.

This big also erases the entire resource fork.

OpenMeta does not use reserved namespaces to store data, but rather  
merely mirrors data into spotlight using the only available route.

The reason it's open source is that before it there were about four or  
so ways to store tags that were all incompatible, and all used the  
Finder comments to store them.

I have been doing the mac programming thing for a while too. People  
are basically demanding this kind of functionality. Using comments to  
store the tags does not work, and trying to track them in some kind of  
db has huge problems. So do the best we can.

--Tom

On 2009-10-13, at 7:50 PM, Neuronical <dan.dob...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »