com.apple.metadata and org.openmeta etc
tom.andersen
do:
1) It seems that everyone but me is in some sort of agreement that we
should not only store tags, etc with com.apple.metadata* attribute
names.
2) Mr Kim suggested a 'search order' - which does clarify where the
definitive data lies. With the search order that he proposes, keeping
the apple domain as primary - sounds ok to me. Then when you write say
tags to a file - you write them to three places:
com.apple.metadata:kOMUserTags <--primary - since users see this in
spotlight, etc
org.metadata - in case the com.apple.metadata should ever be deleted
by Apple, etc.
central xattr database/file - in case Adobe CS4, or other agents wipe
out org.metadata and com.apple.metadata
I guess if OpenMeta finds a file that only has
com.apple.metadata:kOMUserTags, then it could write out the
org.openmeta and backup systems.
P Kim
org.openmeta (or some other acceptable) domain?
Tom Andersen
way to do it.
--Tom
P Kim
so any idea of a timetable would be helpful. Looking at the code, this
specific feature requires just a few lines of code which I can send
over if you want but I think it's straightforward enough that you can
do it in yourself. In any case, any news would be helpful as I'm
delaying my release until I can get resolution on the whole OM
integration.
Thanks,
paul
Tom Andersen
I have been working away here on Leap - and fixing some bugs and
performance things in OpenMeta.
I was waiting until I got Leap 2 out to release the code.
I will hopefully get it up next week. If anyone is biting at the bit,
I can send them what I have now.
There are some important optimizations. Plus a few bugs, plus the
prefs code is in its own .m,h files.
--Tom
Jon Gotow
>Paul,
>
>I have been working away here on Leap - and fixing some bugs and
>performance things in OpenMeta.
>
>I was waiting until I got Leap 2 out to release the code.
Wouldn't it be better to release the code before you ship Leap, in
case we find bugs or have issues with something about the
implementation?
- Jon
--
________________________________________________________________________
Jon Gotow go...@stclairsoft.com
St. Clair Software http://www.stclairsoft.com/
Fax (540)552-5898 ftp://ftp.stclairsoft.com/
P Kim
I've also noticed checkins to OM. Is the codebase in a good place
right now to integrate?
P Kim
tags in the xattrs is commented out. If I am looking in the wrong
place or am missing it entirely, please let me know. Otherwise, I'm
finding this a bit frustrating as it was my understanding that we
agreed to add it in. There wasn't any warning that this code would not
be included. I have already delayed my beta for a month waiting for
this and I find it disappointing that it was left out without so much
as a discussion about it.
Again, if I'm missing something then show me where the relevant code
is.
Paul Kim
with this but I'm going to take one last stab at it.
If you will not implement the tag mirroring, how do you feel about a
branch or fork where I add that functionality? If I do this, can I
still be deemed OpenMeta compliant/compatible?
Also, what is the status of any domains that can be used for this
purpose? I see that OM currently uses openmeta.com for preferences but
the whois record for that domain indicates someone in South Korea owns
it. I don't know if this is someone affiliated with you so any
clarification on this would be appreciated.
It would be very easy for me to drop all this now as it seems like you
are doing everything possible to make sure I quit in frustration. If
you want OM to be seriously considered, I ask that please you stop
evading the questions.
paul
Tom Andersen
I guess I have been working up a response here for the last few days.
I have worked a lot on the OpenMeta code, and made a lot of changes,
tried to simplify things a little, and added support for setting tags
with authentication. I took out the concept of backup stamps, and
optimized the restore process, etc.
Then I got to the place where I was supposed to add the 'mirroring'
code. I just couldn't bring myself to do it. I guess my stomach turned
at the thought of TRIPLY mirrored data and all the problems it entails
- like for instance how to handle gravity tags, etc.
Here is the comment from the code.
This is from the 'get' call. Changed a bit to clear things up.
// Mirroring:
// it might seem prudent to look for an attribute with a key of
com.apple.metadata:kOMUserTags
// and if that fails, to look to the mirror data which will be
under . But if someone sets a tag using an OpenMeta app,
// then removes it with another, possibly old openmeta app, or some
other application that uses
// its own method to call setxattr() directly, the mirrored tags will
be old and wrong.
// So it seems that we would only want to restore from the mirrored
tags with a rewrite of how open meta works,
// after some catastrophic apple erasure of tags. I don't see
a way to tell automatically that the mirrored tags should be used.
// The results will still not be accurate (as the mirrored tags could
be old, etc)
// then why not just use the backup files, and restore from them to
this hypothetical 'new' system?
// id obj = [self getXAttr:[self spotlightKey:metaDataKey] path:path
error:error];
// if (obj == nil)
// obj = [self getXAttr:metaDataKey path:path error:error];
In other words - since I know I need backup for Adobe applications,
and other eventualities, - lets make sure the backup system works.
Simplicity is everything in code and data. There will be enough
problems to solve without throwing in another dimension of data.
Also to say it again: how would we tell when to use the mirrored tags?
I don't see a way. if some Apple update comes down the pipe erasing
all com.apple.metadata, then something would have to be done - we
would need to write quite a bit of code to get it all working again.
(like an OpenMeta database). I also think that there would be some
time in which to get this done in.
So I'm not putting it in. I guess if you believe that it won't work
and that apple will come and blow all your users data away, then
either don't use it, or create your own routine to set tags, which
also sets tags on some mirror domain. The obvious way to do it would
be to also set the tags with out the "com.apple.metadata:" prefix.
--Tom
Michael Tsai
> In other words - since I know I need backup for Adobe applications,
> and other eventualities, - lets make sure the backup system works.
> Simplicity is everything in code and data.
What do you plan to do to make sure it works? The current backup
system does not seem to be simple (especially with the aggressive new
directory caching) and has many shortcomings (previously discussed).
It's also no more accurate in the face of non-conformant applications
than the mirrored xattrs--less so, really, because it would be much
easier for a second codebase to look in the xattrs than to re-
implement your backup system.
I can see different apps possibly rolling their own backup systems,
e.g. to handle security differently. Simplicity is exactly the reason
to define xattrs as the official storage: they're simple to access and
stuck to the file so that there's no "domain mismatch." And, unlike
with a backup system, the data itself will be so simple as to not
change between versions.
> Also to say it again: how would we tell when to use the mirrored tags?
> I don't see a way.
I thought we already discussed this and determined that it was
possible by looking at the tag time--which Gravity's Tags does set.
> So I'm not putting it in.
This is disappointing. First, you agreed to put it in. Paul offered to
write the code, but you said you were going to do it. He delayed his
beta waiting for you. Then you said you were working on it, but that
you wouldn't let us see it until after Leap shipped. Now, Leap has
shipped, the code is not written (what you wrote and commented out is
not what we discussed), and you apparently won't accept a patch, either.
--Michael
Paul Kim
cover them all so bear with me.
First off, as usual, not all my questions have been answered. Maybe
they are moot at this point but it is just confirming a pattern of
evading straightforward questions. It doesn't help build confidence
when you gloss over the weak spots.
Secondly, I have a hard time believing you just came up with this
decision on the day you released the code. If you had changed your
mind (or had no intention of really doing it), it would have saved me
a lot of time if you had told me then. The lack of communication on
this front is pretty appalling.
I don't understand your bending over backward for an app whose
developers, last I heard, have no interest in support OpenMeta. A
standard is something for people to adhere to, not a formalization of
someone else's bad practices.
As for not doing mirroring, you should look at it the other way. To
solve the Adobe problem, you have cooked up a pretty complicated
backup system, one that, by your comments in the code itself, you
aren't even sure works properly. Most of your arguments you levy
against mirroring can apply to your backup system. You talk about
simplicity but looking at the code for your backup system, I do not
see it. In the very least, I do not consider the current
implementation ready for prime time. But no one is going to convince
you on this point. I don't get the sense you are actually open to any
feedback that may differ at all from what you already had in mind.
Also, to quote:
> The obvious way to do it would be to also set the tags with out
the "com.apple.metadata:" prefix.
I can't believe you just said this, and your commented out code also
follows this. After all the debate over domains, you suggest we (the
proponents of observing proper ownership rules with regards to xattr
namespaces) use an unprefixed xattr. If you disagree with us, that's
one thing, but things like this seem to indicate a real
misunderstanding over what was being discussed.
In the end, I can't officially support OpenMeta based on flaws in both
the design and implementation as well a strong resistance towards
addressing them (or even giving straight answers). Right now, I am
considering a hidden default that certain users can set that will
point to an already installed 'omtool' binary. This would be
unsupported and, this way, I won't have to link against this code.
Also, users would have to read up on the risks involved before being
able to learn about this default. Note that this support would be
unofficial and unsupported so do not use my company or product in
conjunction with OpenMeta for any promotional purposes.
Good luck.
sherkaner
that seems likely to lead to a combination of slighty-different but
somewhat-incompatible tagging systems and apps that simply don't
support good tagging.
I understand that you have good reasons for feeling the way you do
about this, Tom, but I guess I'd urge you to consider that to most of
us users, the primary feature of OpenMeta is pervasive support;
without that, none of the technical details matter. I have been
waiting for years for a system that encourages more developers to
support it such that I can use tags the way I want. If pervasive
support is about to fall apart, then I'm a lot less likely to even
convert my old spotlight comment tags over, or buy new apps from
Ironic that are OM-only -- at least I have a few existing apps that
support my existing tags well enough for my basic needs.
Please consider giving in to a different solution, even if you find it
inferior, for the sake of getting the other esteemed developers here
on-board and make working with them (and others) the primary goal of
OpenMeta.
Gmen
I have spent the last several weeks watching the development of the
new Leap (already a user of Leap and Yep), anxiously awaiting its
release.
Only some very intense last-minute reading led me to the posts of Tom,
Jon, Paul and Michael, among others. I was happy to read the evolution
of their thought process, and proud of them (even though I don't know
them) when they started moving forward on what seemed to me a solid
foundation into which all could invest. I agree with the initial
application of considerable effort to create a bedrock for such an
ambitious concept, one with such easily-foreseeable promise and
utility. Many posts boiled it down to: 1. Security and 2.
Searchability
Unfortunately, there was a last-minute 90 degree turn, and I feel the
agreement and compromise (well thought-out, to my eye) is now being
threatened.
I would encourage each of you to take a single deep breath, and get
back on the tracks. You are obviously in agreement on the main
principles, and though you each might be motivated by time, effort or
pride, you are obviously very, very close to a sparkling solution for
all!
Here are my two cents' worth on a few details that should not
overwhelm the big picture:
a. Use the org.openmeta domain as a primary. Mirror to com.apple, but
IMO the future will be stronger without relying on Apple. Stake our
your space.
b. Forget the 'secret' vs 'non-secret' argument. It doesn't matter to
very many users, IMO
c. Be considerate! If another developer is waiting on you, help
provide some mileposts, so he or she can make their own decisions as
events unfold. Asking someone to wait without explanation can make
them feel as if their needs are a very low priority compared to your
needs, which will destroy consensus, rather than building it.
d. Be considerate! To your end-users. We all want reliability. I am
not going to spend a lot of time and effort tagging my files if this
metadata might be lost down the road (especially due to reasonably
foreseeable consequences which were not addressed for minor reasons).
I have decided not to purchase Leap because i do not want to invest my
time, which is significantly more valuable than the reasonable price
of the software. I am sad to say this, because I was so looking
forward to it (I have been scanning in thousands of paper documents
and restructuring my life, and am disappointed not to have this great
tool...)
What are your objectives? Have you all agreed on these? If not, you
can make no real decisions. I strongly urge you to re-define the
objectives, then look at the means to get there.
Sorry in advance if I hurt anyone's feelings. I hear where you are all
coming from, and appreciate greatly the passion. BTW, I love DFX, and
downloaded the trial of Hazel, which I would also love to use in my
life restructuring process. But I usually plan out my path before i
invest a lot of time, and I cannot use the tagging path as it stands
now.
Sincere best wishes to resolving this and creating a new paradigm.
Gentlemen? Can you do it?
Johannes Hoffart
I totally agree with Michael and his view. One thing that is bothering
me, though: would the mirrored xattr tags survive the Adobe apps? If
this is really the case, then scrap the additional backup system.
Even if this is not the case, though, I would still propose to go
forward with the mirrored tags as a primary store (mirrored in the
public namespace that is, not in Apple's).
As a developer of Punakea, where we have recently switched to using
OpenMeta, I was very glad of a good solution finally available for
tagging, but, as others here, I'm concerned with the tag safety and
future-proofness of the whole system. If Tags has a good solution, why
doesn't OpenMeta go the same way?
So, while we will not hold back on the Punakea release waiting for the
final decision (we have migrated our tags once alreay ;)), I still
encourage the OpenMeta developers to address this issue quickly, if you
truly want to create an open standard.
Regards,
Johannes Hoffart
Michael Tsai
> One thing that is bothering
> me, though: would the mirrored xattr tags survive the Adobe apps?
No. Basically, there is a tension between designing a backup system to
handle the situation with the Adobe apps, and designing a simple,
reliable, and future-proof system for storing tags under our own
namespace. Neither design solves both problems well. Mirrored xattrs
do not address the former at all, but they are great for the latter.
Tom's plist backup partially solves the Adobe problem, but it is poor
at the latter.
I think the mirroring is easy to do now and clearly the right choice
for the future. It can be supplemented with another system to work
around the Adobe problem, until those applications are updated to work
with xattrs.
> As a developer of Punakea, where we have recently switched to using
> OpenMeta, I was very glad of a good solution finally available for
> tagging, but, as others here, I'm concerned with the tag safety and
> future-proofness of the whole system. If Tags has a good solution, why
> doesn't OpenMeta go the same way?
As far as I know, Tags does not currently have *any* solution.
--Michael
Johannes Hoffart
> I think the mirroring is easy to do now and clearly the right choice
> for the future. It can be supplemented with another system to work
> around the Adobe problem, until those applications are updated to work
> with xattrs.
I agree with you there.
> As far as I know, Tags does not currently have *any* solution.
I was referring to this part of your last mail:
"I thought we already discussed this and determined that it was
possible by looking at the tag time--which Gravity's Tags does set."
Which would make the mirroring between to xattrs feasible, as far as I
understand. It would be a solution to keeping the xattr tags in sync
properly, right?
I have followed the discussion over the last few weeks, about using the
apple.com namespace, and I haven't been commenting on it. It is also my
opinion, especially for an "open standard", which OpenMeta should be,
and also tries to be if I understand the whitepaper correctly, a
namespace that isn't owned by the standard creators is not a solution in
my eyes. I know this has all been said before, but the more, the
merrier, right? ;)
Regards,
Johannes Hoffart
Michael Tsai
>> As far as I know, Tags does not currently have *any* solution.
>
> I was referring to this part of your last mail:
>
> "I thought we already discussed this and determined that it was
> possible by looking at the tag time--which Gravity's Tags does set."
>
> Which would make the mirroring between to xattrs feasible, as far as I
> understand. It would be a solution to keeping the xattr tags in sync
> properly, right?
What I meant was that Tags does not currently do xattr mirroring or
any other sort of backup. However, because it does set the tag time
(as does OpenMeta), we can use that to make sure the xattrs are in sync.
--Michael
Jon Gotow
>I totally agree with Michael and his view. One thing that is bothering
>me, though: would the mirrored xattr tags survive the Adobe apps? If
>this is really the case, then scrap the additional backup system.
The problem with Adobe apps will continue for quite a while - Adobe's
apps use a method of saving files that destroys xattrs - we have to
deal with that out of practical necessity. Even if they've fixed it
in CS4 (have they?) there are still piles of CS3 users that will
continue using that for the foreseeable future. I'm one of those,
because I've yet to see a compelling reason to shell out for CS4.
>Even if this is not the case, though, I would still propose to go
>forward with the mirrored tags as a primary store (mirrored in the
>public namespace that is, not in Apple's).
The mirrored tags were agreed upon in the last flurry of activity on
OpenMeta, and I was under the impression that everyone, including
Tom, saw them as a reasonable solution to future-proofing us against
any changes that impact the use of the com.apple.metadata naming
convention. Even without a built-in recovery mechanism, just
mirroring the data to org.openmeta (or whatever) xattrs will
safeguard the user's data.
I think several developers made it clear that this was crucial for
their buy-in. Tom, I'm not understanding your approach - if we want
this effort to succeed, we have to come to some consensus with other
developers and put code in place that makes the community happy, not
just you. And communication is necessary too. I understand being
busy - I've been off coding and running my business too - but if you
spearhead a project like this, it's imperative that you at least
answer other developers' questions.
I'm feeling pretty stupid for adopting OpenMeta in Default Folder X
if it's going to fall flat now - pulling OpenMeta out of DFX would be
messy now, and there will be unhappy users. I'd certainly rather see
the mirroring added to the code so everyone can move forward. It's a
good idea and generally works well - why is it so hard to be
pragmatic and get everything working?
The backup solution is practical for dealing with apps that strip
xattrs, but it's not a substitute for mirroring at-risk data in
another store.
>So, while we will not hold back on the Punakea release waiting for the
>final decision (we have migrated our tags once alreay ;)), I still
>encourage the OpenMeta developers to address this issue quickly, if you
>truly want to create an open standard.
That means you, Tom. You have control over the code, so we
ultimately need your input. Please respond.
tom.andersen
are not discussing the 'basics' of how tagging should work, but rather
some of the finer points of how to deploy it, so that your data will
last and be useful.
What we have agreed on: (at least from my viewpoint)
-------------------------------------------------
xattrs are here to stay, Linux has them, Apple has had them since
10.4, and other operating systems have them too. I think that its
obvious that this is where tags, workflows and other meta data belong.
Given that tags should be an xattr - how do we store them? There are
dozens of ways. There are hierarchical tags, case sensitive,
lowercase, ordered (alphabetical, entry order, etc), unordered. Then
there is the format - as an array, a single comma delimited string,
using @ &, etc. Then there is the binary layout of the data.... the
list goes on and on.
Even without spotlight interoperability, I think that writing the tags
as an NSArray plist is an obvious pick - its allows any characters in
the tags, is unicode, etc. Having the array ordered, and case
insensitive case preserving seems to be ok with everyone too.
Apple should endorse and recognize OpenMeta as an example of how to
properly interact with the metadata store for applications writing
metadata to any file type. I have in fact asked for an
What we disagree on: the mirror
________________________
There is the 'why' - why we need to mirror. In my view the need to
mirror is based on fear uncertainty and doubt (FUD).
Fear: Apple will erase tags: (I just don't see why or how.)
A) There has to be a prefix of some sort as a hint to apple to index
the metadata:
If the spotlight DB is to mirror some attributes in the xattrs (it
can't do them all, because there are no format rules for xattrs) then
some prefix (or other mechanism) will be needed: So how do you name
this prefix? The data will be reflected in Spotlight metadata - and it
is definitely an apple generated database: so com.apple.metadata makes
sense - it does not mean that apple owns the data any more than them
owing all rtfd files on your computer.
B) What's the problem?
Apple also owns the "com.apple.iwork.pages.pages" but are they going
to start erasing peoples pages documents? Similarly, if you make an
application that writes rtfd files (com.apple.rtfd) will apple then
remove them from the computer ever? Or change the format without
notice? So is the key name of an attribute like the extension on a
file? - It seems that is is to me. The keyname gives a hint about the
format and content of the data in the attribute. Just like an
extension. In our case the extension is com.apple.metadata:kOMUserTags
- this means that the data is in metadata ready format, and that it
should be stored in the spotlight store under kOMUserTags.
C) Where is the complaint from Apple?
I have contacted apple about this on more than one occasion. I have
filed bug reports on this - the reply was to ask for an enhancement
request to the documentation. I have emailed people at apple about
this. They just don't seem to get excited about it.
What we disagree on: if we did use a mirror - which data would be
king?
________________________
If you make org,openmeta 'king' then any disagreement between
org.openmeta* and com.apple.metadata* will look an awful lot like a
bug to the user. They set a tag using say any currently released
application, then do a spotlight search, which confirms that the tags
are what they set. Then they open up a newer app that uses
org.openmeta* as king and that application will erase the recently
added tags. Or show a different set of tags, or at least its not clear
- the newer app could try to guess what is going on based on time
stamps, etc. Or the app could ask the user - great.
If you make com.apple.metadata* king, then things work more smoothly
from a users perspective. Basically org.openmeta* tags are ignored by
all apps for reading, (but newer applications would write them). To me
it seems clear that this is the way to go.
See for example this:
http://sage.ucsc.edu/~wgscott/xtal/wiki/index.php/Extended_Attributes
This person has no idea about OpenMeta, (it would seem) and would
likely set kOMUserTags directly with their script.
What we disagree on: Writing a mirror of tags: when would we possibly
read from them?
--------------------------------------------------------------------------------
Assume for a moment all the mirrored data is written. Under what
circumstance do you use it? Simple precedence won't work, as
attributes can be erased easily by anyone, on purpose. It would be
really frustrating to erase tags in one app only to have them reappear
again. So if there are no com.apple.metadata kOMUserTags but there are
some openmeta.org ones - do we use those? Likely not - it is far more
likely that the com.apple.metadata ones are correct. A tagtime on the
file gives little or no more information: is that the date that the
tags were removed? or the date that the tags were set, and not the
mysterious erase time.
So I think that the only reason to ever use the mirror would be if
Apple were to start erasing user data from the xattrs, based on
partial key names only (say for 10.7) and we would have plenty of
time. (I personally think this is about as likely as Apple launching
a mission to the moon. ) To get searching working after such an
unlikely event, there would have to be databases written, an mds like
query service written, etc etc. If this were to happen, then it could
very well be that there are other changes - like xattrs being dumped
for some hypothetical new system, or other new problems. Writing out
and managing a mirroring system for this unlikely event is in my
opinion a waste of time. There are real issues to deal with _now_ with
10.5 and certain applications, etc.
Question:
Can you write some code that would automatically switch to using
org.openmeta* tags? I sat there for a while and tried to figure it
out. It seems like I will never be able to use them, unless I know
that a change is coming down the pipe - but in that case much other
work would have to be done - so do something at that point.
eg:
if (com.apple.metadata:kOMUserTags don't exist but
org.openmeta:kOMuserTags do)
{
// what to do?
// most likely, some old app or script, or command line tool has
removed them,
// so replacing them will usually be the wrong choice.
}
--Tom
Max
perspective".
The "only" thing a customer wants is a good, stable product, which can
be used within the whole system.
OpenMeta has great potential but if you aren't able to work together
or to agree on a compromise it will be cluttered and limited to only
some products.
And if this happens - OpenMeta will fail (in my opinion)!!
It can only become something like a "standard" if as much as possible
developer and products use it.
The more, the better!
Nevertheless, I hope you will find a solution for all your "technical"
problems, but I would suggest, that you try a more "professional"
communication.
- At the moment you discuss things at different places (google groups,
ironic forum, ...)
- There is no clear process, no roadmap.
- There are no lists, what is missing, what is unclear, where you have
to work on.
etc. etc.
For all customers I hope, you get in contact more frequently to
discuss things (this thread was started on April 13th... :/ ) - until
everyone is satisfied and until the customers get "one system for
tagging in every OS X application". :)
Good luck! ;)
Michael Tsai
> What we have agreed on: (at least from my viewpoint)
> [...]
> Apple should endorse and recognize OpenMeta as an example of how to
> properly interact with the metadata store for applications writing
> metadata to any file type.
What I'd like is for Apple to provide a documented and supported means
for third-party developers to attach Spotlight-indexable metadata to
files. I don't think relying on a a "com.apple.metadata" prefix is
necessarily the best design, because it's not very flexible and
imposes an unnecessary constraint on the data. Stuffing two kinds of
information (indexability and ownership/name) into one field is a hack.
It might be better for applications to declare (via an API or
Info.plist key) which xattrs should be indexed, or to extend the
Spotlight importer system to allow multiple plug-ins to operate on the
same file, or to use some other design entirely. So I hope that Apple
will add official support for this type of functionality, but I also
hope they will come up with something better than using a prefix.
> What we disagree on: the mirror
> [...]
> In my view the need to mirror is based on fear uncertainty and doubt
> (FUD).
In my view, it's good engineering to rely on the specifications of the
parts you are building with. If you have a part that's rated for
between 50-80 degrees and you try use it at 20 degrees, it might not
work as intended. You want to use it anyway, on the grounds that you
don't foresee any problems. Maybe it really did pass the tests at 20
degrees. Just to be sure, we ask the vendor whether they support the
use of the part at 20. They say no. At this point, you still decide to
use the part.
I don't want to ride in a vehicle that was built this way. I certainly
don't want to sell one and be responsible for all the passengers. I
don't think this is FUD. FUD would be if the part actually *was* rated
for 20 degrees, but I told people that you were using shoddy parts
that weren't up to spec.
> A) There has to be a prefix of some sort as a hint to apple to index
> the metadata:
Prefixing is simply the technique that Apple is currently using within
the OS and built-in apps. There's no reason it has to remain that way.
It is also ambiguous what "com.apple.metadata" *means*. Is it metadata
assigned by Apple? Or is it metadata that's indexed by Spotlight?
Since it is an undocumented prefix, we only know how it *currently
behaves*. I think it's unwise to infer meaning and a contract from
what may be a temporary implementation detail.
> com.apple.metadata makes sense - it does not mean that apple owns
> the data
Yes, this is as clear as can be. I think it's obvious that "com.apple"
means Apple ownership; you don't.
> B) What's the problem?
> Apple also owns the "com.apple.iwork.pages.pages" but are they going
> to start erasing peoples pages documents?
I don't think this is a good analogy because
"com.apple.iwork.pages.pages" is a UTI (an abstract property that a
file may or may not conform to) whereas the xattr is a concrete
storage location. Another concrete storage location is the /System/
Library/CoreServices/ folder. If you store Pages documents there, I
would not expect the OS to preserve them.
> Similarly, if you make an
> application that writes rtfd files (com.apple.rtfd) will apple then
> remove them from the computer ever? Or change the format without
> notice?
Well, they actually did change the Pages format without notice. It
used to be a package, and now it's a Zip file.
RTFD is different because Apple provides an API for accessing it.
Apple would be free to make changes to RTFD that do not violate those
contracts.
Note that if you use the QuickLook API to view a Pages document, the
change is invisible to your application.
> So is the key name of an attribute like the extension on a
> file?
No.
> C) Where is the complaint from Apple?
First, the prefix is not documented as a supported technique. So
assuming that it does what you want and will continue to do so is
guess-work. Second, as previously discussed, I filed a DTS incident
asking Apple whether developers could rely on "com.apple.metadata",
and they said no. In other words, the lack of documentation is not an
accidental omission; they do not support this.
> Assume for a moment all the mirrored data is written. Under what
> circumstance do you use it?
You would use the data with the most recent tag time. Eventually,
perhaps, the transition is over and you simply define org.openmeta to
be the one official location.
> Simple precedence won't work, as
> attributes can be erased easily by anyone, on purpose. It would be
> really frustrating to erase tags in one app only to have them reappear
> again. So if there are no com.apple.metadata kOMUserTags but there are
> some openmeta.org ones - do we use those? Likely not - it is far more
> likely that the com.apple.metadata ones are correct.
I don't think it makes sense to take the absence of com.apple.metadata
to mean that the *user* deleted the tags. To indicate deleted tags, I
would expect to see an empty array and a recent tag time. As with
anything, a user who runs a script that modifies the data in a non-
conforming way may see unexpected results. By defining a standard and
a providing an open-source reference implementation, we make it easy
for people to access the data in a reliable way.
> So I think that the only reason to ever use the mirror would be if
> Apple were to start erasing user data from the xattrs, based on
> partial key names only (say for 10.7) and we would have plenty of
> time.
I just don't think it makes sense to store data in an unsure location,
with promises to migrate later if necessary. It's easy to get it right
now, so why not do it? Any sort of transition would only be more
difficult later, when there's more user data, more applications and
versions of them, and more versions of the OS to contend with.
> Can you write some code that would automatically switch to using
> org.openmeta* tags?
Yes: you use the most recent non-absent value.
--Michael
Gmen
Great, detailed responses from both Tom and Michael. Its obvious you
are both very passionate. A pleasure to read, because of that passion.
Could I ask a dumb question? Are you (and the other major
participants) agreed on the objectives?
I ask because it seems you are both a bit over-focused on the means
rather than the end, which makes it harder to find common ground.
Perhaps when you get to a difficult patch it might help to step back
and review/agree upon objectives, then concretize them.
Then, you can start to list the intermediate steps, and the obstacles
you will need to address individually. I know this all sounds rather
basic, but it sounds to me like you really do have a framework in
place to create an awesome system-wide tool with widespread
applications, from which you all could benefit.
If you keep the channels open like the discussion above, I am sure you
will find the solutions.
Cheers!
Tom Andersen
I have recently tried to get more Apple feedback on this manner. I
have posted in the spotlight - dev list and sent another email out to
Apple. I really think some sort of direction would be good on this. http://lists.apple.com/archives/Spotlight-dev/2009/Jun/index.html
I think that your question is a good one:
I view the objectives as taking metadata to the next level, so that
users can organize and track the data they care about on increasingly
file stuffed computers. (OK that was totally off the cuff - if I
thought about it some more I am sure something better would come out).
--Tom
Tom Andersen
On 1-Jun-09, at 4:30 PM, Michael Tsai wrote:
>
> On Jun 1, 2009, at 12:54 PM, tom.andersen wrote:
>
>> What we have agreed on: (at least from my viewpoint)
>
>> [...]
>> Apple should endorse and recognize OpenMeta as an example of how to
>> properly interact with the metadata store for applications writing
>> metadata to any file type.
>
> What I'd like is for Apple to provide a documented and supported means
> for third-party developers to attach Spotlight-indexable metadata to
> files. I don't think relying on a a "com.apple.metadata" prefix is
> necessarily the best design, because it's not very flexible and
> imposes an unnecessary constraint on the data. Stuffing two kinds of
> information (indexability and ownership/name) into one field is a
> hack.
>
> It might be better for applications to declare (via an API or
> Info.plist key) which xattrs should be indexed, or to extend the
> Spotlight importer system to allow multiple plug-ins to operate on the
> same file, or to use some other design entirely. So I hope that Apple
> will add official support for this type of functionality, but I also
> hope they will come up with something better than using a prefix.
compared to merely documenting an existing technology. Its obviously
not going to happen for Leopard, and I doubt anything could even be
done at this late date for 10.6. I have asked at every WWDC for the
past few years for something like this to be made available. With the
com.apple.metadata* thing, Apple can gain reliability on their
software by documenting how to properly set metadata on a file.
As for a method to extend metadata - there are many ways to imagine
how to do it. But the way that Apple uses now has the large advantage
of it being simple to implement and understand.
Since the keys in the Spotlight database are simple strings, it seems
that whatever method they use, it will have to use straightforward key
strings. And 3 or 6 letters at the start of a keyname is plenty to
define a namespace.
>
>> B) What's the problem?
>> Apple also owns the "com.apple.iwork.pages.pages" but are they going
>> to start erasing peoples pages documents?
>
> I don't think this is a good analogy because
> "com.apple.iwork.pages.pages" is a UTI (an abstract property that a
> file may or may not conform to) whereas the xattr is a concrete
> storage location. Another concrete storage location is the /System/
> Library/CoreServices/ folder. If you store Pages documents there, I
> would not expect the OS to preserve them.
>
>> Similarly, if you make an
>> application that writes rtfd files (com.apple.rtfd) will apple then
>> remove them from the computer ever? Or change the format without
>> notice?
>
> Well, they actually did change the Pages format without notice. It
> used to be a package, and now it's a Zip file.
>
> RTFD is different because Apple provides an API for accessing it.
> Apple would be free to make changes to RTFD that do not violate those
> contracts.
>
> Note that if you use the QuickLook API to view a Pages document, the
> change is invisible to your application.
>
>> So is the key name of an attribute like the extension on a
>> file?
>
> No.
is the name of a storage location.
>
>> C) Where is the complaint from Apple?
>
> First, the prefix is not documented as a supported technique. So
> assuming that it does what you want and will continue to do so is
> guess-work. Second, as previously discussed, I filed a DTS incident
> asking Apple whether developers could rely on "com.apple.metadata",
> and they said no. In other words, the lack of documentation is not an
> accidental omission; they do not support this.
documentation.
>
>> Assume for a moment all the mirrored data is written. Under what
>> circumstance do you use it?
>
> You would use the data with the most recent tag time. Eventually,
> perhaps, the transition is over and you simply define org.openmeta to
> be the one official location.
will simply not work, once a new mirror capable application ships and
has mirrored all the tags. There will be applications that directly
set only what they need to - user scripts, etc.
Then, if you define org.openmeta as the 'real' location - using any
old application will change the spotlight DB so searches will see one
thing, while the tag editor will see another.
>
>> Simple precedence won't work, as
>> attributes can be erased easily by anyone, on purpose. It would be
>> really frustrating to erase tags in one app only to have them
>> reappear
>> again. So if there are no com.apple.metadata kOMUserTags but there
>> are
>> some openmeta.org ones - do we use those? Likely not - it is far more
>> likely that the com.apple.metadata ones are correct.
>
> I don't think it makes sense to take the absence of com.apple.metadata
> to mean that the *user* deleted the tags. To indicate deleted tags, I
> would expect to see an empty array and a recent tag time. As with
> anything, a user who runs a script that modifies the data in a non-
> conforming way may see unexpected results. By defining a standard and
> a providing an open-source reference implementation, we make it easy
> for people to access the data in a reliable way.
everything with no bugs but that is not the case.
Even data that is correctly written to documented Apple APIs can
easily get corrupted. Just try setting the Finder comment on 1000
files in a row. It does not matter if you use Apple events,
AppleScript, or whatever - eventually the Finder locks up and dies,
leaving some Finder comments properly mirrored into
com.apple.metadata:kMDItemSpotlightComment and some not. Just a mess.
But that's a system that uses a triple store - .DS_Store,
com.apple.metadata and the spotlight db. This is in fact about as
close as I can find to a mirrored system with the 'real' data not
indexable as I have found. I have no doubt though that mirroring in
xattrs would work better (but still not good enough).
--Tom
Gmen
Thanks for taking the extra steps. Hopefully you get some useful
feedback.
Off the cuff, a good start! Could you think about it (all of you) and
be a bit more specific? Do you all agree on specific objectives?
Imagine a combo of Hazel, DFX, Leap, etc with all the power of real,
system-wide tags. Any useful data on your computer at your fingertips
almost instantly...
BTW, I am being a bit selfish. I love these programs, and would love
to use such capability myself.
Scott
On Jun 1, 2:28 pm, Tom Andersen <tom.ander...@gmail.com> wrote:
> Thanks for the encouragement!
>
> I have recently tried to get more Apple feedback on this manner. I
> have posted in the spotlight - dev list and sent another email out to
>
> - Show quoted text -
Michael Tsai
>>> So is the key name of an attribute like the extension on a
>>> file?
>>
>> No.
> The xattr is the name of a storage location in the same way that .rtfd
> is the name of a storage location.
Prefixes and suffixes have (by convention) different meanings. Google
owns all the URLs beginning with google.com, but no one owns all the
URLs ending with .html.
>> You would use the data with the most recent tag time. Eventually,
>> perhaps, the transition is over and you simply define org.openmeta to
>> be the one official location.
>
> That will not work, as removing tags with any shipping application
> will simply not work, once a new mirror capable application ships and
> has mirrored all the tags.
Please give a specific example, because I have no idea what you're
referring to.
> There will be applications that directly
> set only what they need to - user scripts, etc.
Really? How do you even construct a binary plist from AppleScript or sh?
> Then, if you define org.openmeta as the 'real' location - using any
> old application will change the spotlight DB so searches will see one
> thing, while the tag editor will see another.
That's why I said "eventually, perhaps." At some point, people won't
even be able to run the old applications, just like I can't run
FrameMaker in Classic on my MacBook Pro.
>> I don't think it makes sense to take the absence of
>> com.apple.metadata
>> to mean that the *user* deleted the tags. To indicate deleted tags, I
>> would expect to see an empty array and a recent tag time. As with
>> anything, a user who runs a script that modifies the data in a non-
>> conforming way may see unexpected results. By defining a standard and
>> a providing an open-source reference implementation, we make it easy
>> for people to access the data in a reliable way.
> It would be great if all software was written by and conformed to
> everything with no bugs but that is not the case.
Do you have an example of an application that doesn't conform? Or is
this just a hypothetical that someone might write code that actively
ignores the standard? I just tested this with Leap 2 and with Tags,
and both remove the tags by setting an empty array.
Also, if there were an application that deleted the xattr instead of
setting an empty array, this would create equal problems for the xattr
mirroring and for your existing plist backup system.
--Michael
Gmen
Is there a way to suggest solutions, then test them, instead of taking
people to task or confronting them?
My two cents...
sherkaner
I'd like to add something to your suggested design goals for OM that I
think is critical to this discussion. Not only does it need to
include the technical goal of letting users organize and track data,
it needs to let users do that across as many applications as
possible. I think this really gets to the crux of the argument;
there is a lot of agreement about the technical goals, but the
disagreement about the details of implementation toward those goals is
pushing the goal of making this an adopted cross-app standard to the
wayside.
I think what some of us users are trying to highlight is that we don't
care as much about what seem to be relatively minor differences in
opinion about the technical implementation as we do about ensuring the
cross-platform nature of OM. And ensuring that doesn't end when the
standard is published by a lead developer. That developer must
actively encourage and work with developers to address their needs,
design cycles, and requests. It's good that the discussion here has
been renewed, but I'm afraid that it's still going to come back to a
difference of opinion that ends in a stalemate that benefits nobody.
I'll state is as clearly as I can: If OM's implementation cannot
convince the developers engaged here to include it in their products
(which include some of the most relevant current applications to us
users), then its value as a standard is nearly nil; it will merely be
Ironic's chosen tagging system. Job 1 here needs to be finding the
agreement between all developers that will get the standard released
and implemented, not winning a technical argument.
Johannes Hoffart
stick with the com.apple prefix:
Just think of the way Spotlight works at the moment. Spotlight importers
overwrite any changes made to the Spotlight DB directly, effectively
only caching the documents for efficient search. What if Apple would, in
the future, adopt a system that handles metadata the same way? They
might give developers an API to write their metadata, and then, upon
_really_ writing the metadata to the file, flush the com.apple.metadata
stuff, rebuilding it from the original sources, just to avoid
synchronization issues with apps that do not use the Apple API?
Even if this is a remote possibility , it highlights the basic sense of
a namespace, and I think that there should be no doubt about what
namespaces are used for. Namespaces were introduced to have your own
area in a shared environment, precisely to avoid conflicts.
Tom, I share your concern about irritating users, but do you really
think any serious developer would use the xattrs, but not know about
OpenMeta? Most probably he would get to know of the method to add user
generated metadata to files through OpenMeta, and in the course learn of
the very simple API. Why should a potential developer not use OpenMeta then?
I think that mirroring the attributes to "our own" namespace, but
keeping the com.apple.metadata one as a primary for the time being
should be a good comprise, and very easy to implement, right?
Performance issues should be negligible I believe.
I really hope we can resolve this issue, because I love the idea and I'm
very thankful that Ironic made OpenMeta available in the first place!
Regards,
Johannes Hoffart
Tom Andersen
How about (hate to partial quote Johannes)
> I think that mirroring the attributes to "our own" namespace, but
> keeping the com.apple.metadata one as a primary for the time being
> should be a good comprise, and very easy to implement, right?
> Performance issues should be negligible I believe.
I have registered openmetainfo.org - should we use that for a mirror -
anyone feel free to reply with a better domain - i think that it makes
sense to use a domain thats not already taken.
If you do find a good free one, don't post it in public - just email
it around a bit. A public post will likely result in the domain being
swiped by someone.
I registered it in my name - there is no 'openmeta' company or
organization to register it under.
What I coud do is post a letter to code.google.com/p/openmeta that
says that the openmetainfo.org domain will not be held for ransom or
sold for any money be me when the initial 2 year period is up. Would
that be ok?
Perhaps there is someone with more knowledge of how to handle the
domain ownership?
--Tom
Michael Tsai
> I think that mirroring the attributes to "our own" namespace, but
> keeping the com.apple.metadata one as a primary for the time being
> should be a good comprise, and very easy to implement, right?
I would rather figure out something that makes sense technically,
before acting. This compromise seems to postpone that, and add more
cases to consider, with a possible second code transition down the line.
We were agreed before that we could migrate to org.openmeta while
maintaining compatibility by using the tag time to determine which
domain was primary. Tom now says, "that will not work...with any
shipping application." I think he's wrong and that it will be
compatible with the currently shipping applications. The only
potential problem is someone writing a script that modifies one domain
but not the other *and* doesn't set the tag time. But if Tom is right,
then obviously we need to re-think this. So, Tom, please explain why
you think this won't work.
--Michael
Michael Tsai
> I have registered openmetainfo.org - should we use that for a mirror -
> anyone feel free to reply with a better domain - i think that it makes
> sense to use a domain thats not already taken.
Agreed. There are also a few variants of openmeta.* available.
Whichever domain we pick, the preferences should be stored there as
well.
> I registered it in my name - there is no 'openmeta' company or
> organization to register it under.
>
> What I coud do is post a letter to code.google.com/p/openmeta that
> says that the openmetainfo.org domain will not be held for ransom or
> sold for any money be me when the initial 2 year period is up. Would
> that be ok?
Yes.
--Michael
sherkaner
essentially "resolved", from Tom's point of view and that of the other
developers here (such as Default Folder X, Hazel, etc.)?
I guess I'm hoping that there is more discussion behind the scenes
going on to get these developers back on board. For example, I know
Hazel's developer had to "give up" on OM for a release, and I'm
wondering if things have been resolved such that it can be included
again officially in the next version.
Jon Gotow
I've been off at WWDC for the last week, so am just catching up now.
Tom and I were discussing this via email before I left, so I'll pick
that up and see where we are.
Michael Tsai
> This thread has gone a bit cold, and I'm curious: is this issue
> essentially "resolved", from Tom's point of view and that of the other
> developers here (such as Default Folder X, Hazel, etc.)?
As I see it, nothing has changed and we're waiting for a response from
Tom.
--Michael
Martin
inject a bit of user perspective into the ongoing discussion. I
understand the urge by developers to find a robust and future proof
solution to the tagging issue on OS X, and the arguments that have
been exchanged here show a high level of both commitment and thought
about the issue. In this respect, I find everything going on here
admirable. This being said, I am aghast, and I think any "just user"
reading this should be.
The way the current discussion is going several important applications
will not support openmeta (Hazel comes to my mind first); others wich
do by now are more or less considering discontinuing support
(DefaultFolderX). All because people cannot agree about "the best way
to do it". Oh really, folks, seriously?
From an user perspective, what is needed is a tagging system that just
works. Apple has provided no ability for OS X to have one, so any
attempt to create one on system level is by necessity a hack (by
system level I mean here things not confined to some proprietary
pseudo-DB system, e.g. XMP). Using Spotlight comments was no less one
than injecting xattr metadata into com.apple namespace (I'm not going
to repeat everything that has been said against using the spotlight
comments here – see http://mac.synthesisstudios.com/filespot/support/viewtopic.php?f=6&t=14
if a synopsis is needed). We can all either sit on our haunches until
Apple delivers (probably right after efax stops sucking and FileVault
gets fixed, say, around OS X 10.32 in, oh, 2049. OK, maybe 2048), or
we agree to hack the system for the time being, with the option of
dropping this once Apple delivers.
xattr tagging in the com.apple namespace is the closest thing to such
a "just works" system I have seen yet. Extended attributes get
maintained by any transfer that respects the filesystem, without
needing any special knowledge of a proprietary format (be it OS X
sidecar files, aka .DS_Store, or XMP, or whatever). Just to give you
an example: I use SugarSync to sync my files across several Macs, and
xattr tags carry over just fine despite the fact SugarSync is entirely
oblivious to OS X peculiarities like resource forks, Spotlight
comments etc. That and seamless Spotlight access is exactly what I am
talking about, as close to set and forget from the user POV as any
system will ever get until Apple releases a tagging API.
Michael, Jon, everybody who objected: don't deny your users a feature
because its future proofness is not set in stone. No computer
technology is, everything developed is only ever done so "for the time
being". openmeta is no 100 % solution, but it is further up that path
that anything developed before. Tom, if people hesitate to jump on the
openmeta bandwagon because of the com.apple issue, do provide
mirroring, or let someone implement it for you, whatever your personal
feelings about it. The last thing any user wants is to say: "Oh, we
could have had decent and robust, easily accessible tagging, but we
won't because the devs squabbled about how future proof it is". Yes, I
said "squabbled", and I rest my case, because unless you all redirect
your energies into developing a constructive compromise that is all
this argument, admirable as the thought involved in it may be, will
remembered as.
No harm meant, I highly respect everybody involved here, but as I
said, I am aghast. Thank you for listening.
Michael Tsai
> All because people cannot agree about "the best way
> to do it". Oh really, folks, seriously?
>
> From an user perspective, what is needed is a tagging system that just
> works.
I think we're all agreed that finer details like the backup system can
be polished up later.
For the basic tag storage, it's not so much a question of agreeing on
the "best way" as that some feel that the current code is
unacceptable. I think "just works" means more than working on a
particular Mac with a particular version of the OS. Users expect that
their data will stay intact and that tagging applications will
continue to interoperate correctly (given reasonable assumptions about
the future).
> Using Spotlight comments was no less one
> than injecting xattr metadata into com.apple namespace
I don't like Spotlight comments, either, but that statement simply
isn't true. Apple has provided developer access to the comments since
System 7.
> Michael, Jon, everybody who objected: don't deny your users a feature
> because its future proofness is not set in stone. No computer
> technology is, everything developed is only ever done so "for the time
> being".
Not knowing what the future may bring is one thing. Unnecessarily
relying on something that we know ahead of time to be out-of-bounds
would be negligent.
> openmeta is no 100 % solution, but it is further up that path
> that anything developed before.
The thing is, in April we agreed on (what we thought was) a 100%
solution. So I see no reason to go backwards from that unless someone
can point to a reason why some other way would be better.
As I see it, the current situation is that:
(a) Tom rejects the April solution but has not identified a specific
problem with it.
(b) Tom controls the code.
So I would like to move forward, but I don't think the ball is in my
court.
--Michael
Martin
thank you for taking the time to reply to my little rant. I was not
hinting the ball was in your court, or advocating everybody move
forward adopting openmeta without discussion on the issue at hand or
compromise on it (that's why I called on Tom to implement mirroring).
But I do actually think the problem here is the discussion has taken a
turn that is lacking user perspective. Take the statement
> > Using Spotlight comments was no less one
> > than injecting xattr metadata into com.apple namespace
>
> I don't like Spotlight comments, either, but that statement simply
> isn't true. Apple has provided developer access to the comments since
> System 7.
supported / approved is the specific way of doing this this exact way
by the API manufacturer"). I would however argue that from a user's
point of view, Spotlight comments are far more hackish than what
openmeta does:
* Because tag data is ambiguous as its identity is implicit in its
syntax, the user needs to remember a non-standard way of indicating
what is searched for (prefixing "&", for instance, instead of the
Spotlight field indicator "<name>:") and might get imprecise results
(a search for "&todo" will also turn up non-tags like any mention of
"&todo" in the body of a document indexed by Spotlight).
* Because tag data is not encapsulated it can be overwritten by any
application using the API for Spotlight comments at any time. Also, as
it is stored in sidecar file on folder level, it can be disconnected
from the file it applies to, and it does not carry over easily and
robustly to other systems (my Sugarsync usage case being in
point: .DS_Store tagging simply does not sync that way), leaving the
user to possibly tag the same file several times both over time and
systems, depending on usage scenario and history. *Actual* robustness
form a user perspective is thus much lower than *potential* technical
robustness (API support).
I'd argue both factors contribute greatly to the lukewarm adoption of
tagging systems on the Mac until now, and any system bent on changing
this needs to be generic, seamless, free of interface paradigm breaks,
and both portable (across machines) and robust.
Now I do understand that robustness over time is exactly the issue at
hand: does the *potentially* lower robustness over OS incarnations of
openmeta outweigh the *actual* user facing advantages so much that it
justifies not implementing it? Judging the potential for breakage in
future OS versions boils down to second guessing Apple's development
path, something nobody has ever been any good at except for one Jobs,
S., and which I certainly won't try to do, but judging the advantage
to users is something I will allow myself to do.
In my opinion, for users, openmeta is too big a step forward to pass
by, so I can only clamor for it to be implemented widely. I can accept
it needs a maximum of insurance against Apple barring the shortcut it
takes, but not to the point of discarding the system entirely. Meaning
either Tom implements mirroring, or someone fork the code and create
openmeta+ or such (doing so in a way backwards compatible with the
current incarnation, please), but either way, and unless someone has
an even better solution up their sleeve, it should be implemented by
anybody thinking tagging is the way to go for OS X users, ASAP. That's
all from a user's standpoint, really. Thank you for reading.
Tom Andersen
All attributes written to com.apple.metadata:* are also written to
org.openmetainfo:*
To read from these requires code that I don't think can be written at
this point, as the only way that they can be out of sync is through an
application not knowing about org.openmetainfo writing tags or other
data to only com.apple.metadata. In that case the com.apple.metadata
stuff should be taken as the 'truth' - it is what shows in generic
Spotlight searches, mdls, spotlight searches for tags, etc.. - its
what the person set.
No one has ever offered one line of code for me to include in
OpenMeta. If there is someone who wants to make a change please take
the time to do so.
Just to be clear:
I feel that this whole discussion compares well to discussing 'how
many angels can dance on the head of a pin'. Or what if Apple changes
its disk format to ext4 without telling the developers or users ahead
of time. ?
The chances of Apple changing this technology are both distant in time
and unlikely. I can't find a hint of a change in Snow Leopard, nor can
I understand how Apple would or even _could_ remove the
com.apple.metadata mechanism. Things like Time Machine backups use
them - backups that need to be viable years down the road.
Furthermore: Say we wrote a 'internet downloader' application: would
it not be 'right' to set com.apple.quarantine on disk images
downloaded with it? Or set the com.apple.kMDItemWhereFroms? I just
don't get it.
--Tom
Michael Tsai
> Now I do understand that robustness over time is exactly the issue at
> hand: does the *potentially* lower robustness over OS incarnations of
> openmeta outweigh the *actual* user facing advantages so much that it
> justifies not implementing it?
That is a false dichotomy because there are ways to have both
robustness and the user-facing advantages.
--Michael
sherkaner
> many angels can dance on the head of a pin'. Or what if Apple changes
> its disk format to ext4 without telling the developers or users ahead
> of time. ?
You may feel that this is something that makes no difference (and
indeed from my perspective, I don't feel strongly about either
direction on this), but the primary problem here is not technical, it
is social: the developers you need the support of are not
implementing OpenMeta because of this issue. That is enormous
problem!
Please consider actively (ie. not just waiting for code) trying to win
these developers back by addressing their concerns. I think us users
have repeatedly expressed that winning their support is the most
important thing here. I have now stopped using Leap because I'm not
sure how to reinstall the pre-OpenMeta version, and I'm not interested
in paying for Leap2 (nor converting my tags to OpenMeta) if I'm not
actually gaining the cross-app support that it is meant to deliver. I
would rather stick with my spotlight comment tags that at least have a
bit of support in other apps, even if it's technically inferior. That
will change the minute you do what it takes to get these other
developers reengaged and excited about OpenMeta.
Michael Tsai
> OpenMeta code has had mirroring since June 4, 2009.
Cool. I think things would move along more quickly if you could tell
the group when you make a change like this.
> All attributes written to com.apple.metadata:* are also written to
> org.openmetainfo:*
What's the status of openmetainfo.org? It looks like it's parked at
GoDaddy.
> To read from these requires code that I don't think can be written at
> this point
Why can't the code be written? Didn't we discuss using the
kOMUserTagTime in April?
> No one has ever offered one line of code for me to include in
> OpenMeta.
I offered on January 30:
<http://ironicsoftware.com/community/comments.php?DiscussionID=632>
Paul offered on May 8:
<http://groups.google.com/group/openmeta/msg/71263063ec54a05a?hl=en>
You never actually accepted or rejected the offers.
> I feel that this whole discussion compares well to discussing 'how
> many angels can dance on the head of a pin'. Or what if Apple changes
> its disk format to ext4 without telling the developers or users ahead
> of time. ?
Do you have an example of an unsupported HFS+ feature that would not
work on ext4? Otherwise, I don't see the analogy.
> The chances of Apple changing this technology are both distant in time
> and unlikely.
As far as I can tell, that's just your hunch. The official word from
Apple is that it's not supported (i.e. subject to change).
> Things like Time Machine backups use
> them - backups that need to be viable years down the road.
My understanding is that most of the metadata is actually stored
elsewhere. Apple mirrors it into com.apple.metadata for the benefit of
Spotlight. I don't believe com.apple.metadata is the primary data
store for any user-entered data, except for OpenMeta. It is the
primary store for kMDItemWhereFroms, but Apple considered that so
unimportant that in 10.4 it was only "stored" in the Spotlight index.
> Furthermore: Say we wrote a 'internet downloader' application: would
> it not be 'right' to set com.apple.quarantine on disk images
> downloaded with it? Or set the com.apple.kMDItemWhereFroms?
That would be incorrect because you should use the LSQuarantine API to
do this.
--Michael
Tom Andersen
Thanks for the input.
openmetainfo.org is likely parked I bought the name - but that's it. I
guess its not too hard to point it to google code openmeta.
On 24-Jun-09, at 8:21 PM, Michael Tsai wrote:
>
> On Jun 24, 2009, at 7:34 PM, Tom Andersen wrote:
>
>> OpenMeta code has had mirroring since June 4, 2009.
>
> Cool. I think things would move along more quickly if you could tell
> the group when you make a change like this.
>
>> All attributes written to com.apple.metadata:* are also written to
>> org.openmetainfo:*
>
> What's the status of openmetainfo.org? It looks like it's parked at
> GoDaddy.
>
>> To read from these requires code that I don't think can be written at
>> this point
>
> Why can't the code be written? Didn't we discuss using the
> kOMUserTagTime in April?
>
>> No one has ever offered one line of code for me to include in
>> OpenMeta.
>
> I offered on January 30:
>
> <http://ironicsoftware.com/community/comments.php?DiscussionID=632>
>
> Paul offered on May 8:
>
> <http://groups.google.com/group/openmeta/msg/71263063ec54a05a?hl=en>
>
> You never actually accepted or rejected the offers.
suggested. I don't know if he wants reading or not. Most people here
don't seem to want to use org.openmetainfo as the primary store.
>
>> I feel that this whole discussion compares well to discussing 'how
>> many angels can dance on the head of a pin'. Or what if Apple changes
>> its disk format to ext4 without telling the developers or users ahead
>> of time. ?
>
> Do you have an example of an unsupported HFS+ feature that would not
> work on ext4? Otherwise, I don't see the analogy
their data.
Michael Tsai
> openmetainfo.org is likely parked I bought the name - but that's it. I
> guess its not too hard to point it to google code openmeta.
Sounds good.
> I added some lines of code to write the org.openmetainfo as paul
> suggested. I don't know if he wants reading or not. Most people here
> don't seem to want to use org.openmetainfo as the primary store.
Is that true? I thought Jon and Paul both supported moving in the
direction of using org.openmetainfo as the primary store. The only
benefit to having com.apple.metadata as primary is that's what the
currently deployed apps do. But by using the kOMUserTagTime we can
transition to the non-Apple namespace while maintaining compatibility.
>> Do you have an example of an unsupported HFS+ feature that would not
>> work on ext4? Otherwise, I don't see the analogy
> I was trying to say that Apple will tell people before they erase
> their data.
OK. My point is that Apple is only responsible for telling you when
they break a contract. They don't tell people when they remove a
private API or change a private file format. You can't expect a
warning if you weren't supposed to put your data there in the first
place.
--Michael
Jon Gotow
>Is that true? I thought Jon and Paul both supported moving in the
>direction of using org.openmetainfo as the primary store. The only
>benefit to having com.apple.metadata as primary is that's what the
>currently deployed apps do. But by using the kOMUserTagTime we can
>transition to the non-Apple namespace while maintaining compatibility.
I want to keep com.apple.metadata as the primary store and mirror
data to org.openmetainfo. We can switch this at a later date if
that's desirable to you and others (I'm comfortable just having a
backup of the data in org.openmetainfo that can be salvaged if
necessary, though I personally don't think that will be necessary).
Delaying the switch would give Gravity some time to incorporate
OpenMeta so that users are all using apps based on OpenMeta before we
transition to org.openmetainfo as the primary store. If we do this,
then all apps will be writing to both stores by the time the
transition occurs and there will be far fewer instances of OpenMeta
having to "fix" things by shuffling tags and deciding which source is
the correct one in a particular instance.
Jon Gotow
>Cool. I think things would move along more quickly if you could tell
>the group when you make a change like this.
Yes! I wasn't aware it'd been done either. I was just wondering if
I'd missed something while I was away at WWDC...
Tom Andersen
of it was in the checkin comment.
Michael Tsai
> I want to keep com.apple.metadata as the primary store and mirror
> data to org.openmetainfo. We can switch this at a later date if
> that's desirable to you and others (I'm comfortable just having a
> backup of the data in org.openmetainfo that can be salvaged if
> necessary, though I personally don't think that will be necessary).
>
> Delaying the switch would give Gravity some time to incorporate
> OpenMeta so that users are all using apps based on OpenMeta before we
> transition to org.openmetainfo as the primary store. If we do this,
> then all apps will be writing to both stores by the time the
> transition occurs and there will be far fewer instances of OpenMeta
> having to "fix" things by shuffling tags and deciding which source is
> the correct one in a particular instance.
I'm not sure I understand your reasoning. You seem to agree that we
*can* figure out which tags are correct, but you're reluctant to do
this. From my perspective:
1. Doing the transition now (in OpenMeta) means that even apps that
use com.apple.metadata will get automatic mirroring in
org.openmetainfo, as soon as the tags are read by any OpenMeta
application. So I see having to "fix things" as a pro, not a con.
2. If we switch now, deployed apps would "just work" if anything
happened to com.apple.metadata. There would be no need to issue
updates that salvage the data in org.openmetainfo.
3. The code will be the same whether or not we write it before Gravity
incorporates OpenMeta.
4. Even after Gravity switches, we would still want to "decide which
source is the correct one" in order to remain compatible with any
legacy apps.
So I see several advantages to switching earlier, and no
disadvantages. Am I missing something?
--Michael
Jon Gotow
>I'm not sure I understand your reasoning. You seem to agree that we
>*can* figure out which tags are correct, but you're reluctant to do
>this. From my perspective:
>
>1. Doing the transition now (in OpenMeta) means that even apps that
>use com.apple.metadata will get automatic mirroring in
>org.openmetainfo, as soon as the tags are read by any OpenMeta
>application. So I see having to "fix things" as a pro, not a con.
>
>2. If we switch now, deployed apps would "just work" if anything
>happened to com.apple.metadata. There would be no need to issue
>updates that salvage the data in org.openmetainfo.
>
>3. The code will be the same whether or not we write it before Gravity
>incorporates OpenMeta.
>
>4. Even after Gravity switches, we would still want to "decide which
>source is the correct one" in order to remain compatible with any
>legacy apps.
>
>So I see several advantages to switching earlier, and no
>disadvantages. Am I missing something?
The disadvantage I see of doing it now is that you're taking on the
responsibility for data synchronization for Tags and old apps that
don't support org.openmetainfo themselves. Yes, the code for this
will work in nearly all cases, but there will always be the isolated
users that, for whatever reason have some bizarre system problem,
disk corruption, or other unanticipated circumstance that will result
in the OpenMeta code doing the wrong thing, or things getting blown
apart somewhere.
Yes, that's ambiguous, but in my experience such problems crop up
every so often and serve to make some poor user's life hell for a few
days while they try and unwind the damage (I do code very
defensively, but still get blindsided by weird stuff like last week,
when a user's disk developed a bad block underneath a cache file and
would lock whenever anything tried to delete the cache. I wasn't
sure that was it until she emailed me a week later to tell me the
whole drive went went south and was being replaced under AppleCare).
By waiting 6 months, we significantly reduce the number of cases in
which OpenMeta needs to make critical decisions about which data is
"correct" (org.openmetainfo vs. com.apple.metadata) and thus reduce
the chance for error. By that time, most apps will be writing to
both org.openmetainfo and com.apple.metadata themselves because
they'll all be using OpenMeta.
Again, this is all predicated on my take that exception cases on
individual Macs are more of a present concern than com.apple.metadata
going away. I guess it's a more simplistic view - don't add more
logic than you have to until it's needed (where we obviously differ
on the term 'needed'). In the end, I honestly think we're arguing
awfully small details. If it makes you happy to write the code now
and include it, I'm not going to pitch a big fuss.
Michael Tsai
> The disadvantage I see of doing it now is that you're taking on the
> responsibility for data synchronization for Tags and old apps that
> don't support org.openmetainfo themselves. Yes, the code for this
> will work in nearly all cases, but there will always be the isolated
> users that, for whatever reason have some bizarre system problem,
> disk corruption, or other unanticipated circumstance that will result
> in the OpenMeta code doing the wrong thing, or things getting blown
> apart somewhere.
OK. I'm happy with a two-phase transition (first add mirroring, then
switch the primary) if that will make people more comfortable.
Tom, are you working on the reading code, or would it help if I
submitted a patch? I think we discussed before the idea of making a
preference. Developers could set it when initializing OpenMeta, or it
could kick in automatically after 6 months or so.
--Michael
Michael Tsai
> I have registered openmetainfo.org
Could you set this up to forward to code.google.com/p/openmeta?
--Michael
Tom Andersen
Thanks,
--Tom
Michael Tsai
> Done that,
Thanks!
--Michael