org.openmetainfo:kOMUserTags -> org.openmetainfo:kMDItemOMUserTags

[Michael Tsai]

What is the reason for renaming this key and the corresponding
timestamp? To me, it looks like this reduces performance (because
there are now more xattrs to read and write in order to remain
compatible) without providing any benefit.

--Michael

[Tom Andersen]

I wanted to make things simple, - eventually I will pull all mention
of kOM* out of the code. I started coding with only adding a mirror to
kMDItemOMUserTags, but that got a little complicated.

Also - when an older program sets tags - I have the time stamps for
that, so I can compare that to the time stamp on kMDItemOMUserTags,
and then take the newer one.

I am thinking that the code could be changed to ignore the older ones
in about a year. (Its also easy to tell if a user is using old
software, and tell them to upgrade, etc). But perhaps I won't be able
to drop the old ever.

--Tom

[Michael Tsai]

On Oct 16, 2009, at 11:57 AM, Tom Andersen wrote:

> I wanted to make things simple, - eventually I will pull all mention
> of kOM* out of the code.

I think this makes things more complicated. In the short term, the
code is more complicated because you're migrating data that doesn't
really need to be migrated. Longer term, you'll have to break
compatibility (bad) or continue checking and writing multiple
redundant locations forever (slow). If 10.7 makes another change, are
you going to rename the keys again? I think it would be much better to
have a fixed location (org.openmetainfo:kOMUserTags) that apps can
rely on remaining the same.

--Michael

[Tom Andersen]

I will look at it more.

--Tom

[Michael Tsai]

On Oct 16, 2009, at 5:38 PM, Tom Andersen wrote:

> I will look at it more.

Hi Tom,

Did you change this for the final version of Yep 2.0?

--Michael

[Tom Andersen]

Michael,

I decided to keep it using kMDItemOMUserTags.

Since saving files in NSDocument based apps now wipes out the xattrs on a file, with kMDItem* restored, it seems that it is now only a useful system for data that is stored in kMDItem* keys. Which is ok, but I had wanted to store more in the xattrs, like workflows and other data that could not fit into Spotlight. Now it seems that OpenMeta should only be used for data that is mirrored into spotlight. Having keys with the kOM names suggests otherwise.

--Tom

[Michael Tsai]

On Oct 28, 2009, at 12:36 PM, Tom Andersen wrote:

> I decided to keep it using kMDItemOMUserTags.
>
> Since saving files in NSDocument based apps now wipes out the xattrs
> on a file, with kMDItem* restored, it seems that it is now only a
> useful system for data that is stored in kMDItem* keys.

That seems irrelevant to me. Whether the system is deleting or
restoring in the com.apple.metadata domain is unrelated to
org.openmetainfo. In fact, one of the reasons for creating
org.openmetainfo was to isolate us from changes that Apple might make
to the OS.

> Now it seems that OpenMeta should only be used for data that is
> mirrored into spotlight. Having keys with the kOM names suggests
> otherwise.

Suggests to whom? Why does it matter? As far as I can tell, users gain
nothing from this change. But having an unstable specification reduces
interoperability and performance.

--Michael

[travisn]

Greetings,

I am all for debate and the creation of the "best" solution to this
problem. We should not be surprised considering the unsupported state
of the tagging methods used by OM. That said, OM does represent the
best practical standard that can be reasonably achieved at this
time.

As a user of many of the apps developed by the great coders here, I
beseech the developers here to feel the leeway to take some time to
address this issue if it results in a more robust or compatible
solution.

FInally, in the end, it seems obvious that tagging is in flux on OS X
and I am ok with patches and changes as long as moderate backward
compatibility is observed, with the understanding that it may take
time (and Apple) for a stable method to be found. So even each
developer implements slightly different solutions, as long as they are
compatible, I am ok with that.

Thanks.

-Travis N

[Johannes Hoffart]

Hi Tom,

as a developer using OpenMeta I am very grateful for the effort you
put into developing it and working around upcoming issues!

As a standard, though, I think there is a bit of a problem with
changing stuff frequently. I read through this thread and still do not
understand why the "main" location of the xattrs was changed (again?).
I know about the problem with document-based applications, and I'd
like to know how you solved it by migrating to kMDItem*? Are
attributes in this domain preserved by Apple?

On a side node, I think it would be great to have a specification of
the standard - if we really want to have a standard! Or is there any
that I do not know of? I can only find the AboutOpenMeta.pdf, and this
seems more like a PR thing :)

Cheers,
Johannes

[Tom Andersen]

I tried not changing the attribute names used to store the data - and things were more complicated, at least to me.

There are two things that i would like to see:

1) A 'lite' API without restore, and a lot of other stuff that would be simpler for dev who want to just edit a few tags to use.

2) As you say, a spec would be nice.

--Tom

[Jon Gotow]

At 12:36 PM -0400 10/28/09, Tom Andersen wrote:
>I decided to keep it using kMDItemOMUserTags.

So you're using [OpenMetaBackup upgradeOpenMetaTokMDItemOM] in Leap 2
to upgrade the user's tags when Leap 2 runs?

- Jon

--
________________________________________________________________________
Jon Gotow go...@stclairsoft.com
St. Clair Software http://www.stclairsoft.com/
Fax (540)552-5898 http://twitter.com/stclairsoft

[Tom Andersen]

Jon,

Sorry for the delay here.

I am running upgradeOpenMetaTokMDItemOM in Leap 2.5 and Yep 2.

I run the call 2 minutes after program launch.

[OpenMetaBackup performSelector:@selector(upgradeOpenMetaTokMDItemOM) withObject:nil afterDelay:2.0*60.0];

My plan is to take this out of the typical startup in about April. After that I will run it when a user picks a restore metadata command. 

I have right now 2 copies of Leap 2.5 running, plus Yep 2, and mds is idling at 0 - 0.2 %. I was under the impression that mds was always running on every system? 

I have noticed that it uses memory - but I think it is of the cache variety - it keeps things in memory for faster searching. 

I think that mds has a lot of work to do the first time  [OpenMetaBackup upgradeOpenMetaTokMDItemOM]  is called. 

--Tom

--

You received this message because you are subscribed to the Google Groups "OpenMeta" group.
To post to this group, send email to open...@googlegroups.com.
To unsubscribe from this group, send email to openmeta+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openmeta?hl=en.

[Tom Andersen]

Jon,

I have just updated the source in the repository to match what I actually used in in Yep 2.0.4 and Leap 2.5.

Its all back in my head now:

Essentially, with the code that you used there was a bug - a giant, slow loop that runs through mds - i was never able to recreate it on my computer, it only happenend with certain files that were tagged weirdly or like that - I had a user with this problem, and basically figured out that something like this was happening:

1) Run search for files to update, (the search is for files with kOMUserTags and NO kMDItemOMUserTags)
2) update the files found

Problem: some of the files that were attempted to be updated are not actually updated - no kMDItemOMUserTags are written.
So these files are still in the search results from 1, and since we tried to update them, now come in as a 'changed' event in the search.
The routine notes that some files have changed, attempts to update the tags on them, which results in the same thing happening, no kMDItemOMUserTags are written, but spotlight counts it as a change.
So the files again show up in the search results from 1.

That's the loop.

Thanks for writing about this - somehow I forgot to update the openmeta repository over the last few weeks.

--Tom


On 2009-12-02, at 1:05 PM, Jon Gotow wrote:

[Jon Gotow]

At 2:07 PM -0500 12/2/09, Tom Andersen wrote:
>I am running upgradeOpenMetaTokMDItemOM in Leap 2.5 and Yep 2.
>
>I run the call 2 minutes after program launch.

Ahh - good call - I'm running it as soon as my spotlight tag window
gets activated, which could bog things down at the wrong time.

>My plan is to take this out of the typical startup in about April.
>After that I will run it when a user picks a restore metadata
>command.

If you stop running it by default, you're going to have a few users
who return to using your app that aren't going to be able to access
their old tags. Also, AFAIK Tags and MailTags aren't using the new
kMDItemOM* attributes, are they? So to maintain compatibility,
doesn't the upgrade process have to be run for the forseeable future?

>I have right now 2 copies of Leap 2.5 running, plus Yep 2, and mds
>is idling at 0 - 0.2 %. I was under the impression that mds was
>always running on every system?
>I have noticed that it uses memory - but I think it is of the cache
>variety - it keeps things in memory for faster searching.

Yes, it's always running and doesn't present much overhead in normal
operation. The big hangup is that the initial run of
upgradeOpenMetaTokMDItemOM kicks off a very long process. This is
especially true if the person is using MailTags, which seems to add
tags to _every_ email in their mailbox. I added debug logging, and
one user sent me a multi-megabyte console log, showing that every one
of his thousands of email messages was getting upgraded with the new
kMDItemOM* tags. This is not a good thing for these users - he said
mds was consuming 50-100% of one cpu for hours.

>I think that mds has a lot of work to do the first
>time [OpenMetaBackup upgradeOpenMetaTokMDItemOM] is called.

Yes, that's the issue. And because Default Folder X is so popular,
I'm getting complaints from quite a few people, many of whom don't
really care about OpenMeta, but happen to be using MailTags (where
Default Folder X will never be asked to look at metadata tags).

[Jon Gotow]

At 2:22 PM -0500 12/2/09, Tom Andersen wrote:
>I have just updated the source in the repository to match what I
>actually used in in Yep 2.0.4 and Leap 2.5.

Tom, I have some pretty choice words for you that I can't say in
public. That was irresponsible - you've just inconvenienced tens of
thousands of MY customers, wasted days of my time, and put another
project way behind schedule. If I could yank OpenMeta support easily
right now, I definitely would. This project has been a huge pain in
the neck for relatively little benefit to me or my customers. This
is a peripheral function in Default Folder X, yet seems to be
consuming an inordinate amount of my time.

>Its all back in my head now:

Gee, I'm glad you've remembered now. It would've been a little more
convenient for those of us who have stupidly become reliant on you if
you'd check in the good code when you finish it.

>Essentially, with the code that you used there was a bug - a giant,
>slow loop that runs through mds - i was never able to recreate it on
>my computer, it only happenend with certain files that were tagged
>weirdly or like that - I had a user with this problem, and basically
>figured out that something like this was happening:
>
>1) Run search for files to update, (the search is for files with
>kOMUserTags and NO kMDItemOMUserTags)
>2) update the files found
>
>Problem: some of the files that were attempted to be updated are not
>actually updated - no kMDItemOMUserTags are written.
>So these files are still in the search results from 1, and since we
>tried to update them, now come in as a 'changed' event in the search.
>The routine notes that some files have changed, attempts to update
>the tags on them, which results in the same thing happening, no
>kMDItemOMUserTags are written, but spotlight counts it as a change.
>So the files again show up in the search results from 1.
>
>That's the loop.
>
>Thanks for writing about this - somehow I forgot to update the
>openmeta repository over the last few weeks.

This is ridiculous.

[sherkaner]

Oh no, I *just* finished converting my files and workflow over to a
system relying on Default Folder X for initial tagging (that's
actually all I use DFX for, despite that OM is understandably a
peripheral function from Jon's perspective) and Leap 2.5 for searching
and tag modification (and Tags for some other ad hoc tagging). I'm
now paranoid that I'm going to regret this decision... Is this going
to resolve itself, or do I need to start backing away from an OM-based
system? I need all of these tools to work together, and to be able to
rely on them continuing to do so for years or I need to just start
sorting everything into folders again while I can.

[Rolf Schmolling M.A.]

Hi,

I want to add that the ability to apply (openmeta-)tags while saving a file via DFX has become an indispensable part of the way I work!

Regards, Rolf

[indevsoftware]

On 2009-12-02, at 3:17 PM, Jon Gotow wrote:

If you stop running it by default, you're going to have a few users
who return to using your app that aren't going to be able to access
their old tags.  Also, AFAIK Tags and MailTags aren't using the new
kMDItemOM* attributes, are they?  So to maintain compatibility,
doesn't the upgrade process have to be run for the forseeable future?

I am starting to look at the new kMDItemOM* attributes now.  Keeping mind that my use of them are to mirror my keywords to kMDItemOMUserTags -- I am not using the xattrs as the main storage of tags. 

I have right now 2 copies of Leap 2.5 running, plus Yep 2, and mds

is idling at 0 - 0.2 %. I was under the impression that mds was

always running on every system?

I have noticed that it uses memory - but I think it is of the cache

variety - it keeps things in memory for faster searching.

Yes, it's always running and doesn't present much overhead in normal
operation.  The big hangup is that the initial run of
upgradeOpenMetaTokMDItemOM kicks off a very long process.  This is
especially true if the person is using MailTags, which seems to add
tags to _every_ email in their mailbox.  

I am not certain how you are coming to this conclusion.   MailTags stores data only on the messages that are specifically tagged. Now it may be some users have a inbox rule that tags every message that comes in -- in which case, it would be specific to certain users but as a general rule, a message has tag data only if it has been tagged by actions of the user.

--Scott

Scott Morrison

sm...@indev.ca

Indev Software

Productivity Enhancements for Mac OS X Mail.

[Tom Andersen]

Jon,

I'm not trying to hide anything here. If I was I would not be posting on this forum. When I made the changes - which was only a few weeks ago - I needed feedback - since the problem was not happening to me. I did not want to check in code that was not tested - so I ran it in the field first. I should have checked it in a week or so ago. I'm sorry about that. I spent a long time tracking this bug down.

--Tom

On 2009-12-02, at 3:17 PM, Jon Gotow wrote:

[Tom Andersen]

I think that the problem - (its NOT a bug in your software ( I think)), was that somehow you would set kOMUserTags in the Spotlight DB, but it would not be set in the xattrs on the mail files? As I said - it never happened to me, even when I tried to get MailTags to cause the problem. Its just that the user with the issue sent me some samples or similar that convinced me that it was something to do with emails, and he was tagging emails with MailTags. 

Sorry I can't offer any more information. 

--Tom

[Tom Andersen]

I tend to use tagging as an addition to having a reasonable filing system. Since I was never personally able to actually do that with anything other than code, I made that Filed Documents thing to auto file. I actually don't think its a good idea to throw all files in one folder, and just use tags.

One of these days some OS vendor will wake up and start directly supporting some sort of extensible metadata system. But the current system of folders for organization will be here for a while. Why they came up with a system where a file can only be found in one way is a mystery to me. But it was probably hard to think that far out of the box in like 1965, or where ever the 'folders within folders' paradigm was thought out.

I think that s3 has a good idea - essentially a flat filing system. One giant NSDictionary of key ==> file. (it also lacks good metadata support, which i really can't understand).

--Tom

[sherkaner]

Oh, I do still use folders to an extent. I was considering using your
Filed Documents feature, but in the end I didn't like that the date-
based structure was using when the document was filed, not the files'
creation date (such as the iPhone library) which would have some
meaning to me on its own. As it is, when dumping all my files in for
the first time, they would all just be in the single folder for that
one date as I understand it! Personal preference I guess.

That said, certainly one of the goals of having robust metadata is the
ability to relieve the user or having to try to represent metadata in
a folder structure. So currently I compromise by organizing by year
of creation and beneath that the name of the creator (I'm dealing with
a lot of files from different sources for my work). So every year,
I'll just save off an "archive" and start a new "current files" folder
with the creator-based folder structure beneath. But if I were to
lose the enormous about of metadata that I've poured into my OM tags,
or I were no longer able to rely on those tags or the workflow I've
built up for them, I would have to try to recreate at least some of
this metadata in a folder structure once more. Otherwise my files
would be hopelessly unfindable.

Sorry for the off-topic post in the thread. Just trying to again make
it clear how important reliability and cross-functionality of OM is to
real users.

> >> For more options, visit this group athttp://groups.google.com/group/openmeta?hl=en.

[Jon Gotow]

At 4:30 PM -0500 12/2/09, indevsoftware wrote:
>I am starting to look at the new kMDItemOM* attributes now. Keeping
>mind that my use of them are to mirror my keywords to
>kMDItemOMUserTags -- I am not using the xattrs as the main storage
>of tags.

So there are instances when a message will be in the Spotlight
database but won't have the tags set as xattrs?

>I am not certain how you are coming to this conclusion. MailTags
>stores data only on the messages that are specifically tagged. Now
>it may be some users have a inbox rule that tags every message that
>comes in -- in which case, it would be specific to certain users but
>as a general rule, a message has tag data only if it has been tagged
>by actions of the user.

Sorry - my misunderstanding. I've just gotten system logs from a few
users in the last 24 hours which show kOMUserTags on what appears to
be every single mail message (the files are named sequentially). I
assumed you were somehow doing this by default, but I'm sure they've
just got a rule set up to do it. I wasn't trying to lay any sort of
blame on you - my point was that there are users with thousands or
tens of thousands of tagged files - via rules involving MailTags or
whatever - and upgrading those tags needs to be done in some
bulletproof fashion.

[indevsoftware]

On 2009-12-02, at 5:59 PM, Jon Gotow wrote:

> At 4:30 PM -0500 12/2/09, indevsoftware wrote:
>> I am starting to look at the new kMDItemOM* attributes now. Keeping
>> mind that my use of them are to mirror my keywords to
>> kMDItemOMUserTags -- I am not using the xattrs as the main storage
>> of tags.
>
> So there are instances when a message will be in the Spotlight
> database but won't have the tags set as xattrs?

When I write out messages with the MailTags data I do call the OpenMeta class methods to mirror the tags.
However, if a user move the messages from folder to folder, It may be possible the xattrs are lost -- Mail doesn't move the cache file, it rewrites it in the new location so xattrs would be lost -- I write out the MailTags in this process but now that I think about it, the tags are not written to the xattrs in this process. Something to fix.

>
>> I am not certain how you are coming to this conclusion. MailTags
>> stores data only on the messages that are specifically tagged. Now
>> it may be some users have a inbox rule that tags every message that
>> comes in -- in which case, it would be specific to certain users but
>> as a general rule, a message has tag data only if it has been tagged
>> by actions of the user.
>
> Sorry - my misunderstanding. I've just gotten system logs from a few
> users in the last 24 hours which show kOMUserTags on what appears to
> be every single mail message (the files are named sequentially). I
> assumed you were somehow doing this by default, but I'm sure they've
> just got a rule set up to do it. I wasn't trying to lay any sort of
> blame on you - my point was that there are users with thousands or
> tens of thousands of tagged files - via rules involving MailTags or
> whatever - and upgrading those tags needs to be done in some
> bulletproof fashion.

no worries on that. and sorry if I came across as defensive . -and as you say -- it nevertheless underscores the need for efficient update mechanism.
Scott

>
> - Jon
> --
> ________________________________________________________________________
> Jon Gotow go...@stclairsoft.com
> St. Clair Software http://www.stclairsoft.com/
> Fax (540)552-5898 http://twitter.com/stclairsoft
>

> --
>
> You received this message because you are subscribed to the Google Groups "OpenMeta" group.
> To post to this group, send email to open...@googlegroups.com.
> To unsubscribe from this group, send email to openmeta+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/openmeta?hl=en.
>
>

[Jon Gotow]

At 7:46 PM -0500 12/2/09, indevsoftware wrote:
>When I write out messages with the MailTags data I do call the
>OpenMeta class methods to mirror the tags.
>However, if a user move the messages from folder to folder, It may
>be possible the xattrs are lost -- Mail doesn't move the cache file,
>it rewrites it in the new location so xattrs would be lost -- I
>write out the MailTags in this process but now that I think about
>it, the tags are not written to the xattrs in this process.
>Something to fix.

OK, so it looks like that's the cause of the OpenMeta xattr upgrade
mechanism continuously looping. Any messages that users have moved
from one mailbox to another no longer have the original xattrs on
them. The upgrade code tries to copy the xattrs from the file, it
doesn't get them from Spotlight's metadata store (at least as far as
I can see from a quick look at the code).

To close this hole, the upgrade mechanism should get the old
kOMUserTags tags from the Spotlight metadata rather than reading them
from the xattr, then write them to the new kMDItemOMUserTags xattrs.
This is more self-consistent, since the metadata query is finding the
items based on the metadata store, not the xattrs.

>no worries on that. and sorry if I came across as defensive . -and
>as you say -- it nevertheless underscores the need for efficient
>update mechanism.

Yes, the update mechanism needs to be there, and will need to be used
long-term to avoid having 'lost' xattrs if users happen to run old
code or pull out files that were tagged with the old xattr names and
then stashed away somewhere.