Experimental MetaData TiddlyWiki released

[Udo Borkowski]

Find an experimental adaption of the TiddlyWiki 2.0.8 at

http://tiddlywiki.abego-software.de/metadata/

that supports MetaData.

Based on discussions with ClintChecketts and JeremyRuston I implemented
some kind of "minimal" interface to cover most of the ideas of "meta
data" that were discussed in this forum. It may serve as the starting
point for the MetaData feature planned for TiddlyWiki 2.1.

Feedback is highly appreciated.

Udo

P.S.: this is not a "supported extension" or so, but just an
"experiment". It will be dropped as soon as the TiddlyWiki supports
MetaData (i.e with the release of 2.1).

P.P.S.: A unified diff of the JavaScript changes can be found here:
http://tiddlywiki.abego-software.de/patch/metadata1.diff.txt

[Simon Baird]

Very nice.

--
Simon Baird <simon...@gmail.com>

[Clint Checketts]

On 4/17/06, Udo Borkowski <Udo.Bo...@gmx.de> wrote:

Find an experimental adaption of the TiddlyWiki 2.0.8 at

        http://tiddlywiki.abego-software.de/metadata/

that supports MetaData.

I couldn't resist... I had to write an addition to this macro:
=====================================================================================
version.extensions.metaValue = {major: 0, minor: 1, revision: 0, date: new Date(2006,4,17)};
config.macros.metaValue = {tooltip: "Show tiddlers tagged with",noTags: "There are no tags to display"};

config.macros.metaValue.handler = function(place,macroName,params,wikifier,paramString,tiddler){
  createTiddlyElement(place,"span",null,params[1],store.getValue(tiddler.title, params[0]));
}
=====================================================================================

Pardon the line breaks, I'm sneaking away from homework as it is ;) It's simple, it just grabs a tiddler's metadata and prints it in the tiddler. It can be used in a tiddler directly or in a template. I'm sure we could make it more robust in the future.

Now, I have to get back to my final projects. Thanks for the work Udo!

-Clint

[Daniel Baird]

After thinking about it a bit, I reckon it could go further.

I'd definitely like to see nesting, or we'll have terrible namespace clashes; but I also reckon it'd be cool to just start treating a tiddler as "just" a data object, and put literally everything into metadata -- modified dates, titles, tags, content.

So if you had a tiddler:

tiddler.getValue("core.title") gets the tiddler title.
tiddler.getValue("core.content") gets the text content.
tiddler.getValue("core.type") could be plugin, if it's a plugin tiddler
tiddler.getValue("reminder.nextReminder") gets the date of the next occurrence of this reminder
tiddler.getValue("gtd.context") gets the context of this gtd action
etc etc

and you could have smart tiddler objects:

tiddler.getValue("core.tags") returns the tags string, and
tiddler.getValue("core.tagslist") returns an array of individual tags

You'd only need to save tags once though, so you would need to be able to maintain a list of datapoints that you don't save.  You could even have a rule like "nothing under "cache.*" gets saved to store".  Then you can do cool stuff like cache the rendered html for a tiddler:

tiddler.setValue("cache.wikifiedTiddlerContent", tiddlerHtml)

Then you can show it, hide it, show it etc without calling the wikifier every time.


There's a million more cool things that'd be possible with something like that.. server sides would be easier, etc.  The price is: as well as the cool new "data object div" tiddler, there'd have to be another "legacy div" tiddler written if we wanted to keep backwards compatiblity.


Wotcha all think?

;Daniel

--
Daniel Baird
http://danielbaird.com (TiddlyW;nks! :: Whiteboard Koala :: Blog :: Things That Suck)
[[My webhost uptime is ~ 92%.. if no answer pls call again later!]]

[Udo Borkowski]

I'd definitely like to see nesting, or we'll have terrible namespace clashes;

Me too. That's what I mean when I am talking about "nested" MetaData (like an address object that contains "street", "city" etc. sub-fields) in the "Possible Extensions" section of "http://tiddlywiki.abego-software.de/metadata/#%5B%5BMetaData%20Version%5D%5D". At the storage side I seem them implemented as nested HTML elements in the data DIV. I.e.

<div tiddler="JoeBrown" modifier="UdoBorkowski" modified="200604172252" created="200604172244" tags="">the tiddler text
    <div class="data">
        <p class="address">
            <p name="street">Washington St</p>
            <p name="zip">12345</p>
        </div>
        <p name="phone">555-1234</p>
    </div>
</div>

but I also reckon it'd be cool to just start treating a tiddler as "just" a data object, and put literally everything into metadata -- modified dates, titles, tags, content.

Absolutely yes. If foreward compatibility is not an issue (i.e. old TiddlyWiki core code must be able to work on the new format) the implementation is very straight forward: just move all attributes into subelements of the data DIV. This just needs minor changes in the saveToDiv and loadFromDiv functions. This would also make the store XHTML compliant, I think. For me foreward compatibility would not be that important in the longterm, as long old TiddlyWiki stored could be read. With the suggested approach this is possible, and the first save would write a "converted" store.

I guess foreward compatibility is only an issue for plugins that directly work on the store DIV (and for the serverside adaptions). Maybe we should discuss how important "foreward" compatiblity is for other people.

and you could have smart tiddler objects:

tiddler.getValue("core.tags") returns the tags string, and
tiddler.getValue("core.tagslist") returns an array of individual tags

... so you would need to be able to maintain a list of datapoints that you don't save.  You could even have a rule like "nothing under "cache.*" gets saved to store"

These are interesting ideas. I tried to come up with an "as small as possible" solution for the MetaData, following Jeremy's idea of keeping the core as small as possible but still providing as much commonly used functionality as possible. So we probably again get to the point to discuss what features (of MetaData) should go into the core and what should be implemented through extensions. For me nested meta data (/namespaces) is definitely a core feature, for smart tiddlers I am not sure, but the "cache" feature should also be in the core.



Udo

[Jules]

!!!!!! Wow

Very nice concepts Udo and Daniel.

I really like this idea. It is simple but very powerful and keeps
things nice and standard. I think that the nested data should be kept
as xhtml to make it future compatible with XML processing.

For me, backwards compatibility with existing plugins would not be an
issue. I think that we need to accept some disruption over time as TW
is still finding its way forwards and is a very "young" idea. And in
this case, it looks like the idea is so powerful and useful that it
would be worth the disruption.

--
Jules
www.knightnet.org.uk
http://knighjm.googlepages.com/knightnet-default-tw.html

[Daniel Baird]

On 18/04/06, Udo Borkowski <Udo.Bo...@gmx.de > wrote:

I'd definitely like to see nesting, or we'll have terrible namespace clashes;

Me too. That's what I mean when I am talking about "nested" MetaData (like an address object that contains "street", "city" etc. sub-fields) in the "Possible Extensions" section [...]

Yes -- I'd seen it in your suggestions Udo, just wanted to vote a resounding "yes" for it :)

but I also reckon it'd be cool to just start treating a tiddler as "just" a data object, and put literally everything into metadata -- modified dates, titles, tags, content.

Absolutely yes. If foreward compatibility is not an issue (i.e. old TiddlyWiki core code must be able to work on the new format) the implementation is very straight forward: just move all attributes into subelements of the data DIV.

I think an important implication of this for me is that the tiddler *content* should be just another data element, rather than a sibling of the data div.  Which means we wouldn't need a data div anymore.. just the tiddler div.

This would also make the store XHTML compliant, I think.

Yes -- nice little side benefit isn't it :)

and you could have smart tiddler objects:

tiddler.getValue("core.tags") returns the tags string, and
tiddler.getValue("core.tagslist") returns an array of individual tags

... so you would need to be able to maintain a list of datapoints that you don't save.  You could even have a rule like "nothing under "cache.*" gets saved to store"

These are interesting ideas. I tried to come up with an "as small as possible" solution for the MetaData [..snip..]  For me nested meta data (/namespaces) is definitely a core feature, for smart tiddlers I am not sure, but the "cache" feature should also be in the core.

Yes I agree that nesting is absolutely necessary.

I think having a list<->string converter like in my example wouldn't be too difficult (eg Array.join() and String.split() would go a long way toward making it work :) and I'd love to have a no-save-list, for things that require once-off parsing, like Jeremy Sheeley's reminders and whatnot.

I picked those couple of things because they seemed really useful to have for the cost of maybe a dozen lines of code each.  I can't actually think of anything else that I'd consider as useful, although I'm sure that stuff is there :)

Udo, I think you are I are thinking on very similar lines for this.  What does everyone else think?

[Saq Imtiaz]

Another vote for the nested metadata here, it would be a horrible mess otherwise....

And great work Udo!
Cheers,

Saq

[Saq Imtiaz]

PS: if compatibility with existing plugins is a concern, we could always finalize the ETM in a beta, and then give plugin writers fair time to adapt their plugins and make them compatible.

Overall I feel the ETM will be more than beneficial enough in the long run to warrant disrupting existing code and plugins....
Cheers,

Saq

[Jeremy Ruston]

Excellent stuff, Udo, as usual. Thanks for sharing.

There's a couple of points to pull out of the discussion:

First backwards compatibility: if we're going to pursue XHTML
compliance for 2.1.0 then I think that pretty likely means that we'll
have broken backwards compatiblity (meaning that a plugin written
pre-2.1.0 won't be able to understand the format of a post-2.1.0
tiddlywiki file). So, I'd be happy to see the suggested integration of
old-style and new style metadata. I agree that it's going to require
some way of handling different data types.

Secondly, nesting. I don't think that any code is actually needed to
do this; can't we just adopt a convention to use 'dot' as a separator
in metadata field names? I don't see much advantage in nesting the
underlying DIV storage.

Thirdly, the cache idea. I can see where this idea is coming from, but
I wonder whether it isn't introducing unnecessary complexity. Surely,
if a plugin wants to cache some information related to the tiddler
"HelloThere", a reasonable way to do it is for the plugin itself to
own a cache object that's private:

myCache["HelloThere"].["my field name"] = "blah blah blah";

Fourthly, there's another a related feature to consider. As I've
mentioned in previous threads, I favour the idea of what I'm calling
EmbeddedFields for tiddlers that contain structured data. There was
some discussion towards the end of this thread (see my post at Apr 12
2006 3:33 pm):

http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/6ebda0957ea0c891/99de69ed697a5674?q=tomo&

Now, I know that about 99% of the people reading this are going to go
"ugh?" surely that's solving exactly the same as the custom metadata
we're discussing here... I'm not so sure; I think it's analogous to
the difference between file metadata (like it's created date) and the
file content. Anyway, I'd welcome any thoughts on this.

Cheers

Jeremy

--
Jeremy Ruston
mailto:jer...@osmosoft.com
http://www.tiddlywiki.com

[Udo Borkowski]

So, I'd be happy to see the suggested integration of
old-style and new style metadata. I agree that it's going to require
some way of handling different data types.

Here my idea:

When writing the DIV we put the "attribute" metadata into the data Div. I would keep the "tiddler" name as an attribute, as the "name" attribute. This should keep the storage XHMTL compliant but seems to be nicer when the storage is later processed with XML tools. The tiddler content (text) I would keep as a text node in the DIV. This should still keep the storage XHMTL compliant.

So instead of

<div tiddler="JoeBrown" modifier="UdoBorkowski" modified="200604172252" created="200604172244" tags="">the tiddler text
    <div class="data">
        <p name="street">Washington St</p>
        <p name="zip">12345</p>
        <p name="phone">555-1234</p>
    </div>
</div>

we could write

<div name="JoeBrown">the tiddler text
    <div class="data">
        <p name="modifier">UdoBorkowski</p>
        <p name="modified">200604172252</p>
        <p name="created">200604172244</p>
        <p name="tags"></p>
        <p name="street">Washington St</p>
        <p name="zip">12345</p>
        <p name="phone">555-1234</p>
    </div>
</div>

(added whitespaces for clarity)


When reading the DIV we first look for the legacy metadata (modifier, created,...) in the data DIV (and the "name" attribute). Only if this is not found the old attributes are checked and used. This way we can read the old format, and save with the new format.

(Of cause the old Tiddler object properties "text", "title", "tags", "modifier" etc. still exist, just filled from the data DIV)

Secondly, nesting. I don't think that any code is actually needed to
do this; can't we just adopt a convention to use 'dot' as a separator
in metadata field names? I don't see much advantage in nesting the
underlying DIV storage.

You " 'dot' as a separator " convention approach will definitely work.

On the other hand I could imagine that having a "real" nested DIV structure would make it easier when processing the storage with XML tools, e.g. when translating the original storage into some other format. Some path expressions to select e.g. just the "core" attributes may be simpler and more efficient when working on a more structured storage format.

Nevertheless it is OK to use the suggested "dot as separator " approach without loosing too much.
* For a novice user this "looks like a nested structure" (even though this is "only" a convention).
* We may later decide to use the "." in the name as a "real" separator and store the data in a nested DIV storage. This will not affect the clients that are just accessing the getValue/setValue function.

Thirdly, the cache idea. I can see where this idea is coming from, but
I wonder whether it isn't introducing unnecessary complexity.

I actually like the idea to use one set of functions to attach volatile and persistent (meta) data to tiddlers. I don't see too much complexity involved here.

Eg. assume we define that all fields starting with "temp." are temporary fields that are not stored this just requires an extra test against the regular expression /^temp\./ for the field names before their values are stored.

(BTW: when we are using a "real" nested structure this would just add a check for the top level names).

Fourthly, there's another a related feature to consider. As I've
mentioned in previous threads, I favour the idea of what I'm calling
EmbeddedFields for tiddlers that contain structured data. 

To me the embedded fields and the metadata are two features that allow the user to attached "structured" data to a tiddler (and the DataTiddlerPlugin is a third). Both features are independent of each other and have very different consequences to the users. The most obvious ones are visibility to the users (good with embedded fields) and performance (better with metadata). But an analogous to file metadata and and file content I don't see here. The actual meaning of the data stored in the (meta) data fields depends on the usage by the client code: this could be "content" or "meta" data. A field may even contain both Data and Meta Data (e.g. like currently the Tags). I even expect that most of the developers that are planning to use the meta data feature will probably put "pure" data into the fields, not "MetaData".

Since both features are independent I would prefer to first get the metadata stuff into TiddlyWiki. In a second step embedded fields may be added.


Udo

[Jeremy Ruston]

> <div name="JoeBrown">the tiddler text
> <div class="data">
> <p name="modifier">UdoBorkowski</p>
> <p name="modified">200604172252</p>
> <p name="created">200604172244</p>
> <p name="tags"></p>
> <p name="street">Washington St</p>
> <p name="zip">12345</p>
> <p name="phone">555-1234</p>
> </div>
> </div>

I'm really no expert in XML styling, but might it be more useful to
put the tiddler text in it's own sub-DIV? That way it could be more
simply independently addressed with an XPATH-type query, wouldn't it?
Actually, picking up another common request, it could even be a <PRE>
so that we can finally handle whitespace properly?

> When reading the DIV we first look for the legacy metadata (modifier,
> created,...) in the data DIV (and the "name" attribute). Only if this is not
> found the old attributes are checked and used. This way we can read the old
> format, and save with the new format.

Makes sense, but how would the code know that, say, "modified" and
"created" are dates and not strings? The current approach of having
special knowledge baked into the core seems a tad primitive.

> Eg. assume we define that all fields starting with "temp." are temporary
> fields that are not stored this just requires an extra test against the
> regular expression /^temp\./ for the field names before their values are
> stored.

OK, I'm happy with that. and "temp." seems like a good prefix.

> Since both features are independent I would prefer to first get the
> metadata stuff into TiddlyWiki. In a second step embedded fields may be
> added.

I'm actually advocating doing embedded fields for 2.1.0 as well. The
motivation is to support more robust handling of plugins as part of
the new ImportTiddlers code.

Cheers

Jeremy.

[Saq Imtiaz]

Secondly, nesting. I don't think that any code is actually needed to
do this; can't we just adopt a convention to use 'dot' as a separator
in metadata field names? I don't see much advantage in nesting the

underlying DIV storage.

You " 'dot' as a separator " convention approach will definitely work.

Most of the technical aspects of this discussion are beyond my comprehension, but as someone who is eagerly looking forward to using ETM, I have an observation that I'd like to share.

One of the advantages that I see of proper nesting of metadata, is that it will be easier to remove, and check for, metadata 'groups'.

Let's say you are uninstalling a plugin called AddressBook that created nested metadata like this:
AddressBook

Street
Zip
City
Country

Then you could easily run a script like: tiddler.removeValue(AddressBook), and thus remove all nested metadata content as well.

Similarly, if for a tiddler you wanted to check to see if AddressBook metadata had already been saved, you could check for: if (getValue(tidder.AddressBook)), instead of having to check for each individual metadata entry.

Now I have very little programming experience, and my apologies in advance if I'm missing something, but I'm imagining that the above two examples would be a little more convoluted and less efficient to implement if the metadata is not properly nested.

Just some thoughts, as I said before I'm new at this, so if I'm completely wrong here, just pretend you never read this. :)

Saq

[Jeremy Ruston]

> Just some thoughts, as I said before I'm new at this, so if I'm completely
> wrong here, just pretend you never read this. :)

Not at all, these are sound observations.The behaviour you describe is
indeed desirable. You could imagine something similar being done with
wildcards (tiddler.removeValue("AddressBook.*").

Your comments made me think also that it might be useful to
incorporate a means of enumerating through all the items of metadata
attachments on a particular tiddler.

Cheers

Jeremy

[Udo Borkowski]

I'm really no expert in XML styling, but might it be more useful to
put the tiddler text in it's own sub-DIV? That way it could be more
simply independently addressed with an XPATH-type query, wouldn't it? 

getting the tiddler text is simple. It just requires this XPath expression (the meta data will not be included)

    div/text()

Actually, picking up another common request, it could even be a <PRE>
so that we can finally handle whitespace properly?

I think we are fairly free in choosing the tags. I though of making all tags "p" tags, since "p" is shorter than "div" (or "pre"), so we save some space. I assume that the "<pre>-like" whitespace handling could also be done through a proper CSS definition of the subelements of the store. We should ask our CSS guru.

Makes sense, but how would the code know that, say, "modified" and
"created" are dates and not strings? The current approach of having
special knowledge baked into the core seems a tad primitive.

Therefore I planned to use conditional code that accesses the "class" attribute (not added in the sample, I forgot). I.e. it may look like this:

 <p name="modifier" class="String">UdoBorkowski</p>
 <p name="modified" class="Date">200604172252</p>
 <p name="created" class="Date">200604172244</p>
 <p name="tags" class="StringList"></p>

If you look into the code you will see that I currently only check for "String". Since I assume "String" is the most frequently used type I made this default, i.e. it does not need to appear in the store (i.e. a missing "class" indicates a String field).

Regarding the type names I am not sure if we should really use the "full" names or if a less speaking but shorter name (like "D" for date or so) would help to keep the store smaller. Also with an eye on the TiddlyWiki size we may consider "predefined types" for the standard core fields since explicitly stating the type with these fields (that exists with every tiddler) add at least 30 Bytes to every tiddler.

BTW: size may also be an argument why to use nested storage: When not using nested storage the common prefix in the name is repeated for every field, making the storage larger. E.g. if we choose "core." as the prefix for the core fields we save 2 bytes per tiddler when using an extra "core" level around the fields instead of using the long names "core.modifier", "core.modified",....

I'm actually advocating doing embedded fields for 2.1.0 as well. The
motivation is to support more robust handling of plugins as part of
the new ImportTiddlers code.

Makes sense.


Udo

[Saq Imtiaz]

On 4/18/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

> Just some thoughts, as I said before I'm new at this, so if I'm completely
> wrong here, just pretend you never read this. :)

Not at all, these are sound observations.The behaviour you describe is
indeed desirable.

Glad to help out when I can Jeremy.

You could imagine something similar being done with
wildcards (tiddler.removeValue("AddressBook.*").

Your comments made me think also that it might be useful to
incorporate a means of enumerating through all the items of metadata
attachments on a particular tiddler.

I think the important thing is that no matter what format we decide upon for ETM, we need to make sure it's easy to manipulate the metadata in an efficient manner. Also the core should offer some functions for this which cover the most frequent usage scenarios, so as to avoid the duplication that would occur if each plugin writer had to write their own.
Cheers,

Saq

[Udo Borkowski]

Your comments made me think also that it might be useful to
incorporate a means of enumerating through all the items of metadata
attachments on a particular tiddler.

Good point. This looks like asking for something like this:

    TiddlyWiki.prototype.forEachField = function(tiddler, callback) ...

with a callback looking like this:

    function myCallBack(store, tiddler, fullFieldName, value) ...


(if we are going to support "real" nested fields the callback may get an extra "parent" parameter)


Udo

[Jeremy Ruston]

It sounds like it's worth coding up the nested fields approach so that
we can assess what the impact/benefits are.

With reference to the <PRE> tag, earlier discussions arrived at the
conclusion that it's only <PRE> that does whitespace preservation and
works across all browsers. I wouldn't use it for the metadata, just
for the text, hence the suggestion of further wrapping the text.

For type indicators, yet another possibility is to use prefixes on the
names like 'txt' and 'dat', as with the option handling....

Cheers

Jeremy

On 18/04/06, Udo Borkowski <Udo.Bo...@gmx.de> wrote:
>

[Luke Blanshard]

Udo Borkowski wrote:
> ...

> >Actually, picking up another common request, it could even be a <PRE>
> >so that we can finally handle whitespace properly?
> >
> I think we are fairly free in choosing the tags. I though of making all
> tags "p" tags, since "p" is shorter than "div" (or "pre"), so we save
> some space. I assume that the "<pre>-like" whitespace handling could
> also be done through a proper CSS definition of the subelements of the
> store. We should ask our CSS guru.

Isn't the PRE issue an IE bug, where IE wrongly compresses whitespace
in the text nodes of non-PRE elements? If that is true, then I don't
think CSS will likely solve the problem.

Luke

[Clint Checketts]

On 4/18/06, Luke Blanshard <lu...@blanshard.us> wrote:

Udo Borkowski wrote:
I assume that the "<pre>-like" whitespace handling could
> also be done through a proper CSS definition of the subelements of the
> store. We should ask our CSS guru.

Isn't the PRE issue an IE bug, where IE wrongly compresses whitespace
in the text nodes of non-PRE elements?  If that is true, then I don't
think CSS will likely solve the problem.

Luke

I haven't seen any CSS gurus around, but Luke is correct in that IE plays dumb regarding ignoring whitespace.

Its funny because if you use pre{white-space: normal;} the pre will compress the whitespace as expected. But the reverse (white-space: pre;) only works on the PRE tag and then its pointless because thats its default setting!

-Clint

[Bob McElrath]

Jeremy Ruston [jeremy...@gmail.com] wrote:
> Thirdly, the cache idea. I can see where this idea is coming from, but
> I wonder whether it isn't introducing unnecessary complexity. Surely,
> if a plugin wants to cache some information related to the tiddler
> "HelloThere", a reasonable way to do it is for the plugin itself to
> own a cache object that's private:
>
> myCache["HelloThere"].["my field name"] = "blah blah blah";

This would not allow the plugin to save cached data to the tiddly.html
file.

For Plugin:jsMath I would like to cache rendered math. Right now I have
tiddlers that take upwards of 10 seconds to render. It turns out the
biggest bottleneck in jsMath is putting things into a hidden <span>,
measuring its size using offsetWidth and offsetHeight, and using that
info to set CSS sizes. (This is jsMath.BBoxFor if any CSS guru's want
to take a stab at speeding it up) So, caching an already-rendered
equation would be a big boon. Since the rendered text is a unique
function of the input string "$x$" this would be quite efficient.
People also often repeat $x$ in their discussion of $f(x)$ where $x$ is
an integer... no need to render it twice.

How would you suggest to do the above cache? This particular cache is
independent of the tiddler, since $x$ can appear in multiple tiddlers,
but you only need one cached value. I suppose I could always create a
tiddler jsMathCache and put everything in there. For example (sorry the
jsMath output is rather verbose):

<div class="jsMathCache">
<div name="$x$">
<SPAN class="typeset">
<NOBR>
<SPAN class="scale">
<SPAN style="position: relative; width: 0.687em; height: 1.062em;">
<SPAN style="position: absolute; top: 0em; left: 0em;">
<SPAN class="cmmi10"> x </SPAN>
&nbsp;
</SPAN>
<SPAN style="width: 0.687em; height: 1.062em; vertical-align: 1.062em;" class="blank"/>
</SPAN>
<SPAN style="margin-right: -1px;" class="spacer"/>
<SPAN style="width: 1px; height: 0.581em; vertical-align: 0.531em;" class="blank"/>
</SPAN>
</NOBR>
</SPAN>
</div>
<div name="\[\int_a^b x~dx\]">
etc....
</div>
</div>

But going farther, I'd like to cache the entire rendered tiddler as
HTML. Also I see no reason not to save this to the file. With a
last-rendered and last-modified attribute, it would be easy to keep the
two in sync, and you could create a tiddlywiki that only has to do the
heavy lifting of parsing and rendering right after an edit. Your
tiddlywiki file would be much larger -- but for local applications this
wouldn't be a problem. It could be enabled with a simple switch in the
OptionsPanel: "save rendered tiddlers".

--
Cheers,
Bob McElrath [Univ. of California at Davis, Department of Physics]

"It is almost universally felt that when we call a country
democratic we are praising it; consequently, the defenders of every
kind of regime claim that it is a democracy, and fear that they
might have to stop using the word if it were tied down to any one
meaning." -- George Orwell

[Jeremy Ruston]

> For Plugin:jsMath I would like to cache rendered math. Right now I have
> tiddlers that take upwards of 10 seconds to render. It turns out the
> biggest bottleneck in jsMath is putting things into a hidden <span>,
> measuring its size using offsetWidth and offsetHeight, and using that
> info to set CSS sizes. (This is jsMath.BBoxFor if any CSS guru's want
> to take a stab at speeding it up) So, caching an already-rendered
> equation would be a big boon. Since the rendered text is a unique
> function of the input string "$x$" this would be quite efficient.
> People also often repeat $x$ in their discussion of $f(x)$ where $x$ is
> an integer... no need to render it twice.

The discussion here has been about a *volatile* cache that doesn't get
saved to tiddly.html. For your purposes, Udo's extended metadata will
do what you need out of the tin, I think.

Cheers

Jeremy

[Bob McElrath]

Jeremy Ruston [jeremy...@gmail.com] wrote:
> The discussion here has been about a *volatile* cache that doesn't get
> saved to tiddly.html. For your purposes, Udo's extended metadata will
> do what you need out of the tin, I think.

Sorry for the confusion, I'll look at that.

For a volatile cache though, I think I agree with you, individual plugin
authors can do this willy nilly, I don't think the store or TW core
needs to know about it.

[Udo Borkowski]

>
>
>For your purposes, Udo's extended metadata will
>do what you need out of the tin, I think.
>

I think so, too, except that the stored content of the field "$x$" will
not contain any < or > etc. since the content is HTML escaped. But this
should be no problem, I guess.

And in the current prototype the name $x$ is not a valid field name.
Is there a need to make $ a valid (meta) data field name? Just let me
know and I will add it.

Udo

[Udo Borkowski]

Thanks Luke, Clint and Jeremy for that info, I didn't know that. That's again the difference between theory and real life ;-)

So it make perfectly sense to use the PRE tag to surround this IE bug.

Udo

[Bob McElrath]

I'm sorry I deleted some relevant pieces of this thread. Could you send
a URL for the plugin under discussion?

And it is a plugin, not part of the core?

I'll look at it more closely. Tex math can in general contain lots of
complicated punctuation $\alpha^{+}$ etc. that would probably have to be
escaped, but that can be dealt with.

Thanks,

[Udo Borkowski]

Here the link to the thread: http://groups.google.com/group/TiddlyWikiDev/browse_frm/thread/b4fa2553ece95e09

Actually this is not a plugin but an "experimental adaption of the TiddlyWiki 2.0.8 that supports MetaData.". It may go (in a modified way) into the core for TW 2.1. You find the adaped TiddlyWiki at  http://tiddlywiki.abego-software.de/metadata/



Regarding the identifier syntax I would like to restrict it first to have more option for extensions later and to avoid possible problems with XML etc. that may have restriction on "their identifiers" (e.g. the possible values of the "name" attribute in an HTML element).

Currently the "classic" identifier syntax
    [a-zA-Z_][a-zA-Z_0-9]*

is assumed (and a "." as a separator for namespaces/nested fields).



Udo

[Jeremy Ruston]

> Regarding the identifier syntax I would like to restrict it first to have
> more option for extensions later and to avoid possible problems with XML
> etc. that may have restriction on "their identifiers" (e.g. the possible
> values of the "name" attribute in an HTML element).
>
> Currently the "classic" identifier syntax
> [a-zA-Z_][a-zA-Z_0-9]*

Does this mean that one wouldn't be able to have tiddlers with Unicode
characters in their titles?

[Udo Borkowski]

>
>
> I wouldn't use it for the metadata, just for the text, ...
>

With my new knowledge of the IE whitspace handling: isn't there a
problem if we keep the "field data" in non-PRE tags?

This would mean that client code that set a value "A B" (i.e. A and B
separated by three spaces) would read an "A B" the next time the
TiddlyWiki is loaded in IE. Currently I write the string value as
htmlEncoded, but htmlEncode does not escape spaces, ie IE would merge
subsequent spaces. Or am I missing something?

Udo

[Bob McElrath]

Udo Borkowski [Udo.Bo...@gmx.de] wrote:
> Here the link to the thread:
> http://groups.google.com/group/TiddlyWikiDev/browse_frm/thread/b4fa2553ece95e09
>
> Actually this is not a plugin but an "experimental adaption of the
> TiddlyWiki 2.0.8 that supports MetaData.". It may go (in a modified way)
> into the core for TW 2.1. You find the adaped TiddlyWiki at
> http://tiddlywiki.abego-software.de/metadata/

Aaah excellent thanks.

> Regarding the identifier syntax I would like to restrict it first to
> have more option for extensions later and to avoid possible problems
> with XML etc. that may have restriction on "their identifiers" (e.g. the
> possible values of the "name" attribute in an HTML element).
>
> Currently the "classic" identifier syntax
> [a-zA-Z_][a-zA-Z_0-9]*
>
> is assumed (and a "." as a separator for namespaces/nested fields).

Why not just escape the 'fieldName' argument all the time? Then you
don't care what kind of line noise names the user tries to store things
with. I would replace the logic in isValidMetaDataFieldName,
isValidMetaDataValue, checkMetaDataFieldName, and
checkMetaDataFieldValue with a pair of escape/unescape functions.

I don't understand where the restrictions on the "name" attribute come
from, but one could also consider a slightly different structure:
<div tiddler...>
blah blah
<div name="data">
<div name="fieldName1">$x$
<div name="value">...</div>
</div>
<div name="fieldName2">$y$
<div name="value">...</div>
</div>
</div>
</div>

e.g. putting the fieldName as XML text rather than an attribute. Then
you can use the same escaping functions as are already present in
javascript and tiddlywiki since only <>& need to be escaped.

Of course, getting things out then requires a loop on all fields...but
actually you're doing that anyway...I would have thought in your current
code you could have used getElementById in some way to avoid the loop.

--
Cheers,

[Udo Borkowski]

Does this mean that one wouldn't be able to have tiddlers with Unicode
characters in their titles?

(this refers to the suggested "name" attribute in the Tiddler DIV, I guess)

I am not sure. I don't know if there are any restriction on the "name" attribute in an XHTML element. If this can be any Unicode text (I guess so, but I am not sure right now), using any kind of tiddler title in the name attribute of the XHMTL element should be no problem .


Regarding the syntax for the meta data field names I would still like keep some restrictions. E.g. the name should not contain chars like "[" "]" (we may introduce arrays later) and quotes should also be avoided. In general, starting with a restricted syntax gives us more options in the future without getting into compatibility issues. Nevertheless adding a "$" seems to be OK (this is typically also part of a "classic" identifier).


Udo

[Jeremy Ruston]

> I am not sure. I don't know if there are any restriction on the "name"
> attribute in an XHTML element. If this can be any Unicode text (I guess so,
> but I am not sure right now), using any kind of tiddler title in the name
> attribute of the XHMTL element should be no problem .

Ouch. It looks like the name attribute is deprecated in XHTML:

http://www.w3.org/TR/xhtml1/#h-4.10

> Regarding the syntax for the meta data field names I would still like keep
> some restrictions. E.g. the name should not contain chars like "[" "]" (we
> may introduce arrays later) and quotes should also be avoided. In general,
> starting with a restricted syntax gives us more options in the future
> without getting into compatibility issues. Nevertheless adding a "$" seems
> to be OK (this is typically also part of a "classic" identifier).

Quite agree, we should restrict the metadata field names tightly for the moment.

[Bob McElrath]

Udo Borkowski [Udo.Bo...@gmx.de] wrote:
> >
> >
> >Does this mean that one wouldn't be able to have tiddlers with Unicode
> >characters in their titles?
> >
>
> (this refers to the suggested "name" attribute in the Tiddler DIV, I guess)
>
> I am not sure. I don't know if there are any restriction on the "name"
> attribute in an XHTML element. If this can be any Unicode text (I guess
> so, but I am not sure right now), using any kind of tiddler title in the
> name attribute of the XHMTL element should be no problem .
>
>
> Regarding the syntax for the meta data field names I would still like
> keep some restrictions. E.g. the name should not contain chars like "["
> "]" (we may introduce arrays later) and quotes should also be avoided.
> In general, starting with a restricted syntax gives us more options in
> the future without getting into compatibility issues. Nevertheless
> adding a "$" seems to be OK (this is typically also part of a "classic"
> identifier).

This would not be nearly enough. Tex needs the full ASCII set,
including nasty things like <>&\{}. See my other message on escaping.

--
Cheers,

[Udo Borkowski]

Why not just escape the 'fieldName' argument all the time?  Then you
don't care what kind of line noise names the user tries to store things
with.

The main reason is that we may want to put some semantic into the name passed with the get/setValue functions.

E.g.: we already decide to add some "nested" fields that should be accessed with a "dotted" name (e.g. "address.street"). If we would not restrict the name syntax someone would be able to define a name like "3.1415972" but this would not fit into our "nesting" feature. Or assume we want to add an "array access", with a notation like "names[3]". Without the restriction we would break any code that used names containing brackets.

I don't understand where the restrictions on the "name" attribute come
from,

I am not sure if there really is a restriction, I need to check this. I just wanted to be on the "safe side" first.
Sorry for the confusion.

Udo

[Bob McElrath]

Jeremy Ruston [jeremy...@gmail.com] wrote:
> Quite agree, we should restrict the metadata field names tightly for the moment.

What's the motivation here?

I'd like to think of the metadata as a persistent associative array.

One is usually not restricted in the contents of a string used as a key
into e.g. a python dictionary or perl hash.

--
Cheers,

[Bob McElrath]

Udo Borkowski [Udo.Bo...@gmx.de] wrote:
> >
> >
> >Why not just escape the 'fieldName' argument all the time? Then you
> >don't care what kind of line noise names the user tries to store things
> >with.
> >
> The main reason is that we may want to put some semantic into the name
> passed with the get/setValue functions.
>
> E.g.: we already decide to add some "nested" fields that should be
> accessed with a "dotted" name (e.g. "address.street"). If we would not
> restrict the name syntax someone would be able to define a name like
> "3.1415972" but this would not fit into our "nesting" feature. Or assume
> we want to add an "array access", with a notation like "names[3]".
> Without the restriction we would break any code that used names
> containing brackets.

It seems you're trying to simulate a more complex data type with a
simpler one.

Using the idea that I'm going to store a persistant associative array,
one could in principle have (prototype):
metadata["address"]["street"] = "Evergreen Terrace";
or
metadata["3.14159"] = "\pi";
metadata["\pi"] = "3.14159";
metadata["$x$"] = ...xml...

An arbitrarily complex tree structure could be accomplished by nesting
div's, allowing XML to be your tree structure rather than simulating it
with the syntax of the field name (e.g. "address.street"):

<div class="data">
<div name="fieldName1">\pi</div>
<div name="fieldValue1">3.14159</div>
<div name="fieldName2">3.14159</div>
<div name="fieldValue2">\pi</div>
<div name="fieldName3">address</div>
<div name="fieldValue3">
<div name="fieldName1">street</div>
<div name="fieldValue1">Evergreen Terrace</div>
</div
</div>

Values could be retrieved with
getValue(key1, key2, ...)
getValue("address", "street");

or even return arrays, which would allow:
getValue("address")["street"]

[Udo Borkowski]

>
>
>Ouch. It looks like the name attribute is deprecated in XHTML:
>
>http://www.w3.org/TR/xhtml1/#h-4.10
>

Oh No!

That makes XHMTL an even more verbose language than I thought.

But "class" is still a valid attribute in XHTML elements?

Udo

[Bob McElrath]

It seems I want serialized JSON: http://www.json.org/js.html or YAML:
http://yaml.org/

Both of which could handle arbitrary strings as keys (with proper
escaping).

[Udo Borkowski]

I must admit that I initially planned to use JSON as the storage format
for the MetaData, but because of the high level of interest to get an
XHTML/XML compliant storage I decided to switch the direction. As a
consequence all the coding done in JSON now needs to be reprogrammed for
the XHMTL format. Taking into account the limited time I also decided
not to cover all the JSON functionality in the first step, but leave all
the doors open to extend the meta data later.

Udo

[Daniel Baird]

 

Wow, threads grow when you're sleeping instead of checking every ten minutes.

 

Re metadata as inner divs / metadata as attribs:

Jeremy, I think that enclosing the tiddler content in a pre tag will allow plugin handling the way you want without having to use attributes, as the plugin info is available before you parse the pre.  or am i missing something here?

 

And this raises a more intresting point.. if we have meta as inner divs (or pre or whatever) how does the newbie copy a plugin into their TW?  Previously you could just view the raw text, copy and paste, but that won't work unless we're happy to show our entire tiddler div in the textbox, where Joe Newbie might type over it.

 

The only way I can imagine is to have a separate "view source" button that shows the (readonly) full text of the tiddler div, including the metadata stuff.  Still, I'd hate to lose the simplicity of the old way.

 

Re the pre tag:

I think if we use pre for some data, we should use it for all data, and have consistent whitespace handling.  I'm not too concerned about the two extra bytes per field.

 

Re fieldNames:

It's just the name of the field we're talking about, not it's content.. is it that much of a problem to only allow [a-zA-Z_] [a-zA-Z_0-9] *  and special meaning for [.]?  Non-developer users should never see a field name, it doesn't need to carry that much meaning.

 

Re the name attribute, and also type indicators:

the class attribute is a space-separated list, so one option is to put everything in there -- first class is the field name, second is the type:

<pre class="core.title txt">MainMenu</pre>

 

An added bonus to this is that the "no javascript" css could target core.title as a class name (assuming you can have a dot in a class name?) and make it look a bit nicer.

 

Re using a . to group field names:

I'm pretty sure everything that you could do with "proper" nesting you can do with a dot and some regex work -- as in, the two methods are equivalent in expressiveness.  You could even do relative paths if you had to, as long as you remember the "current position".  It actually doesn't matter how the store is, erm, stored, as long as you use a dot-path to get your data in and out it will all look the same.  Remember, your heirarchical file system is a metaphor -- your hard disk is really a sequential list of bytes :)

 

I think the strength of keeping our tiddler structure simply-specified and fieldnames very simple, is that the store itself could be made into anything quite easily -- YAML/JSON, sqlite, a remote server, etc etc.

 

 

Cheers

 

;Daniel

--
Daniel Baird
http://danielbaird.com (TiddlyW;nks! :: Whiteboard Koala :: Blog :: Things That Suck)
[[My webhost uptime is ~ 92%.. if no answer pls call again later!]]

[Udo Borkowski]

if we have meta as inner divs (or pre or whatever) how does the newbie copy a plugin into their TW?  Previously you could just view the raw text, copy and paste, but that won't work unless we're happy to show our entire tiddler div in the textbox, where Joe Newbie might type over it.

I don't think that is is really an issue since Joe Newbie does not need to care about metadata (at least when it come to plugin installation). He still copies the text, adds the systemConfig tag and a nice title and reload the page. For now plugin tiddlers don't contain any meta data that are necessary during installation. "EmbeddedFields" are contained in the text and will be copied together with the text (This is an important difference between "embedded fields" and "meta data" !)

In the more general case that a tiddler with meta data needs to be copied from one TiddlyWiki to an other I think the way to go is using the new "Import" feature to be added with 2.1. I would not like to see the user work on the DIV directly, especially since for me the suggested DIV structure is just one way the tiddlers may be stored. E.g. for serverside solutions stuff may be stored in an SQL database structure, etc.

Re the pre tag:

I think if we use pre for some data, we should use it for all data, and have consistent whitespace handling.  I'm not too concerned about the two extra bytes per field.

FOUR bytes, 4 bytes per field ;-) 

But I argee with you!

Re the name attribute, and also type indicators:

the class attribute is a space-separated list, so one option is to put everything in there -- first class is the field name, second is the type:

<pre class="core.title txt">MainMenu</pre>

Excellent idea. I will it out,



Udo

[Jeremy Ruston]

> Re metadata as inner divs / metadata as attribs:
> Jeremy, I think that enclosing the tiddler content in a pre tag will allow
> plugin handling the way you want without having to use attributes, as the
> plugin info is available before you parse the pre. or am i missing
> something here?

I'm proposing that we don't use the extended metadata mechanism for
the plugin properties, but rather a separate mechanism I'm calling
embedded fields which basically interprets the first table in a
tiddler as a name-value list.

The distinction between the two is that EMD is for data about the
content of a tiddler, while EF is for accessing structured content
within a tiddler.

The EF mechanism is intended to work well with the copy-paste use case.

[Jules]

I think that you should use ID rather than name shouldn't you? This has
been the case for some time. Ah, yes - having actually read the
reference ;)

What is wrong with using ID instead of name? Damn, must actually think
before typing - of course, ID's are supposed to be unique (though I
don't know of any problems at all when they aren't apart I suppose from
making DOM traversal harder).

CLASS? That would work though it may increase name clashes.

Ah, what about TITLE? I think that would be better.

--
Jules
www.knightnet.org.uk
http://knighjm.googlepages.com/knightnet-default-tw.html

[Jules]

Sorry, should have also said that using TITLE would mean that you could
use more characters than ID or CLASS might allow (if you are retaining
XML compatability).

[Jules]

That's nice - as long as it gets suitable documented for newcomers! A
little bit confusing though for newbies to have two meta mechanisms.

Would it be possible for the wikifier to give that table a slightly
different format (by applying an CSS class) so that it is obvious when
it is looked at?

[Udo Borkowski]

>
>
>Sorry, should have also said that using TITLE would mean that you could
>use more characters than ID or CLASS might allow (if you are retaining
>XML compatability).
>

Thanks.

Just to be sure: do you know if it is valid wrt XHTML to put any Unicode
string into the title attribute? In that case I would also use the title
attribute to hold the Tiddler title (sounds somehow logical ;-)

Udo

[Jules]

Daniel Baird wrote:
>
> And this raises a more intresting point.. if we have meta as inner divs (or
> pre or whatever) how does the newbie copy a plugin into their TW?
> Previously you could just view the raw text, copy and paste, but that won't
> work unless we're happy to show our entire tiddler div in the textbox, where
> Joe Newbie might type over it.

I feel a copy/paste macro coming on!

>
> *Re fieldNames:*

> It's just the name of the field we're talking about, not it's content.. is
> it that much of a problem to only allow [a-zA-Z_] [a-zA-Z_0-9] * and
> special meaning for [.]? Non-developer users should never see a field name,
> it doesn't need to carry that much meaning.
>

> *Re the name attribute, and also type indicators:*

> the class attribute is a space-separated list, so one option is to put
> everything in there -- first class is the field name, second is the type:
> <pre class="core.title txt">MainMenu</pre>

I think I'd have to agree here. I suspect that the way to cope with the
Math issue is to have TWO attributes per math "attribute" one for the
extended title and one for the data, that would certainly get around
the problem.

>
> An added bonus to this is that the "no javascript" css could target
> core.title as a class name (assuming you can have a dot in a class name?)
> and make it look a bit nicer.

I'm pretty certain that you cannot have a . in a classname.

As I thought, you cannot have a . in either a classname nor an id.

[Daniel Baird]

On 19/04/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

I'm proposing that we don't use the extended metadata mechanism for
the plugin properties, but rather a separate mechanism I'm calling
embedded fields which basically interprets the first table in a
tiddler as a name-value list.

[...]

The EF mechanism is intended to work well with the copy-paste use case.

I can understand that embedded fields would be copy-and-paste-able and whatnot, and I guess we are already trusting newbies not to edit the actual code of a plugin.  But it still seems a bit wrong to be adding two different ways of storing structured data.

Maybe we should have a shim that reads embedded fields inside tiddlers and makes them available in the metadata structure -- maybe a prefix like "embedded.thefieldname " or whatever.

That way we'd at least have a consistent way to get at data from the two places.  Wotcha think?

;D

[Jules]

Udo Borkowski wrote:
> Just to be sure: do you know if it is valid wrt XHTML to put any Unicode
> string into the title attribute? In that case I would also use the title
> attribute to hold the Tiddler title (sounds somehow logical ;-)
>

Yes you can. See:
http://www.opentag.com/xfaq_charrep.htm
Under the question:
"Can element and attribute names use non-ASCII characters?"

Though it looks like you should set the encoding attribute in the xml
declaration. In fact, that site seems to have quite a lot of good stuff
about XML and language encodings.

Of course, TW is HTML 4.01 at the moment and so doesn't have an xml
declaration. For HTML, you can use either the META tag with charset,
e.g.:
<META http-equiv="Content-Type" content="text/html; charset=EUC-JP">
or the charset attribute can be used. It is rather easier if the page
is delivered from a server as that can set the encoding in the HTTP
headers.

Not sure how that helps really, I suspect we will have to experiment to
find the right combination.

[Daniel Baird]

On 20/04/06, Jules <kni...@googlemail.com> wrote:

I'm pretty certain that you cannot have a . in a classname.
As I thought, you cannot have a . in either a classname nor an id.

Have you got a reference for this?  I think the xhtml spec says that the "class" attribute has a flavour of "NMTOKENS", which eventually leads here which says that's a bunch of NMTOKENs with whitespace between, and a NMTOKEN is (NameChar)+, and NameChar is.. well I'll spare the details but it basically is a letter OR digit OR combiningChar OR extender OR - OR _ OR . OR : , and they tell you not to use the : because it has a special meaning in XML.

So that seems to say a . should be ok in a class name -- I could imagine it would screw up some CSS rules though.

;D

[Jeremy Ruston]

> That way we'd at least have a consistent way to get at data from the two
> places. Wotcha think?

I think there's a useful analogy with filesystems here. Most file
systems support standard metadata like creation date, modified date
etc. Some file systems (like NTFS and Mac OS X) support extended
metadata so that applications can add extra attributes to files that
don't disturb their contents. For example, OS X's Spotlight uses
extended metadata to control some of it's indexing activities, and XP
SP2 uses extended metadata to track the origin of files downloaded
from the internet.

In that context, the analogy with our extended metadata is clear. I
think that a CSV file is analogous to the proposed embedded fields
mechanism: it's a format for storing structured data in the content of
a file.

[Jeremy Ruston]

> Not sure how that helps really, I suspect we will have to experiment to
> find the right combination.

Early versions of TiddlyWiki did use the 'id' attribute to hold the
title of a tiddler. I changed it because it turns out that there's
some very nasty subtle rules about the semantics of ids.

First, case sensitivity is weird. Search for "The Special Case of id"
on this page:

http://www.ericmeyeroncss.com/bonus/render-mode.html

You'll also see the comment that it's illegal to create an ID that
starts with a digit.

These rules are subtle, as I say and inevitably are actually
implemented differently in different browsers, hence why I made the
change originally.

[Jules]

Daniel,

I simply tried it out using FF (latest version) and it no work. Didn't
try IE though.

[Jules]

Sheesh, why am I not surprised!
J.

[Daniel Baird]

Hmm.. oh well.  I'm sure there's some kludge-around.  At worst, we might have to put the field type first, and all subsequent classes can be the path.  eg instead of
<pre class="core.tags strList">
we could have:
<pre class="strList core tags">
..that's not exactly ideal though.

;D

On 20/04/06, Jules <kni...@googlemail.com> wrote:

[Bob McElrath]

This is a lot of restrictions...what's wrong with using embedded div's
again?

I think I'm missing something here.

Daniel Baird [danie...@gmail.com] wrote:
> Hmm.. oh well. I'm sure there's some kludge-around. At worst, we might
> have to put the field type first, and all subsequent classes can be the
> path. eg instead of
> <pre class="core.tags strList">
> we could have:
> <pre class="strList core tags">
> ..that's not exactly ideal though.
>
> ;D
>
> On 20/04/06, Jules <kni...@googlemail.com> wrote:
> >
> >
> > Daniel,
> >
> > I simply tried it out using FF (latest version) and it no work. Didn't
> > try IE though.
--

[Daniel Baird]

On 20/04/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

> That way we'd at least have a consistent way to get at data from the two
> places.  Wotcha think?

I think there's a useful analogy with filesystems here [...] I

think that a CSV file is analogous to the proposed embedded fields
mechanism: it's a format for storing structured data in the content of
a file.

I can see that analogy.

In our case as well as the filesystem case, there's two kinds of data -- stuff describing the content, and stuff describing the file object itself.  Eg the dimensions of a GIF image and what it's transparent mask colour is are stored in the file's content, not in file metadata, so if you do a file copy by reading hex digits over the phone to someone, you still keep that content-specific info.  Info about the file object, like modification date, wouldn't be preserved, but the GIF is still fully described and therefore useable.

In our case we need to be mindful of what items apply to a tiddler's content, and what applies to the tiddler object itself.  A tiddler's title and tags are probably more about how the tiddler fits into it's parent TW than descriptive of the tiddler content.  Same with mod dates etc.  So it seems right that those things don't need to be copy-and-paste-able.

However, I'd also assumed that EMD would be useful for stuff like giving a tiddler a reminder date, so that tiddler could pop up somehow on that date.  If I found a tiddler called "Mr T's Birthday" I'd want the reminder to still be working after I copy it into my own TW.

I guess that's the main thing that's bugging me -- we're talking about a great, powerful, structured data model, but it seems to be crippled in certain situations.


;Daniel

[Daniel Baird]

On 20/04/06, Bob McElrath <bob.mc...@gmail.com> wrote:

This is a lot of restrictions...what's wrong with using embedded div's
again?

I think I'm missing something here.

Jeremy seemed against it, at least vaguely..

Secondly, nesting. I don't think that any code is actually needed to
do this; can't we just adopt a convention to use 'dot' as a separator
in metadata field names? I don't see much advantage in nesting the
underlying DIV storage.

The main reason I can think of is that it's annoying to translate minesweeper.highscores.score01.name into a DOM node.. but that's not exactly a big deal.  Maybe if we can't get a separator like dot or whatever, we should go to nested <pre>s after all.

;D

--
Daniel Baird

[Bob McElrath]

Daniel Baird [danie...@gmail.com] wrote:
> On 20/04/06, Bob McElrath <bob.mc...@gmail.com> wrote:
> >
> > This is a lot of restrictions...what's wrong with using embedded div's
> > again?
> >
> > I think I'm missing something here.
>
> Jeremy seemed against it, at least vaguely..
>
> > Secondly, nesting. I don't think that any code is actually needed to
> > do this; can't we just adopt a convention to use 'dot' as a separator
> > in metadata field names? I don't see much advantage in nesting the
> > underlying DIV storage.
>
> The main reason I can think of is that it's annoying to translate
> minesweeper.highscores.score01.name into a DOM node.. but that's not exactly
> a big deal. Maybe if we can't get a separator like dot or whatever, we
> should go to nested <pre>s after all.

Why would anyone ever type the relevant HTML by hand?

I'm thinking some kind of programmatic interface
saveValue("minesweeper", "highscores", "score01", "name", "Bob");
or
saveValue("minesweeper.highscores.score01.name", "Bob");
where the core doesn't know or care about any hierarchy. See for
example about:config in mozilla -- those values are one big list -- the
hierarchy with '.' is not an actual hierarchy.

Is it cut and paste for plugins that's hard then?

On loading your plugin could set some variables:
metadata = {
author: "Bob McElrath",
version: "1.2.2"
};
The core could then examine the variable 'metadata' to see if the plugin
put something there. This is easy to cut and paste.

Do I understand the intended use correctly?

Do people want other metadata on non-plugin tiddlers? I don't see any
not-hideously-ugly way to cut & paste that, nor any reason to restrict
key/values to [a-zA-Z] or some such. Cut and paste for such tiddlers
seems to scream for a kind of serialization of the content. What kind
of HTML that is is pretty much irrelevant, but I sure don't want to see
it in the editor as HTML, and it should be accessed programatically.
Perhaps a new toolbar button for cutting, pasting, & serialization?

For reference, the metadata that I would save for math would never need
to be copied/pasted.

[Daniel Baird]

On 20/04/06, Bob McElrath <bob.mc...@gmail.com> wrote:

> The main reason I can think of is that it's annoying to translate
> minesweeper.highscores.score01.name into a DOM node.. but that's not exactly
> a big deal.  Maybe if we can't get a separator like dot or whatever, we
> should go to nested <pre>s after all.

Why would anyone ever type the relevant HTML by hand?

I don't mean humans typing, I just meant taking that string and, in code, traversing the DOM to find the right value.  But as you point out you'd only write that code once, anyway.  Like I said, it's not a big deal.. it doesn't matter how it's stored (we should pick a way that's easy for the core and servside plugins to work with), and there's a bunch of ways to reference heirarchical data that are all more or less equivalent (here we should pick that one that's easiest for humans to code with).

On loading your plugin could set some variables:
    metadata = {
        author: "Bob McElrath",
        version: "1.2.2"
    };
The core could then examine the variable 'metadata' to see if the plugin
put something there.  This is easy to cut and paste.

Do I understand the intended use correctly?

Even cooler would be if the tiddler loader checked for embedded data in any tiddler (Jeremy's "EF" idea was to interpret the first table in a tiddler as data) and load that into the EMD structure.  The table gets copied around as part of the content, and we can still get at the data with the nice EMD API.

Do people want other metadata on non-plugin tiddlers?  I don't see any
not-hideously-ugly way to cut & paste that, nor any reason to restrict
key/values to [a-zA-Z] or some such.  Cut and paste for such tiddlers
seems to scream for a kind of serialization of the content.  What kind
of HTML that is is pretty much irrelevant, but I sure don't want to see
it in the editor as HTML, and it should be accessed programatically.
Perhaps a new toolbar button for cutting, pasting, & serialization?

Yeah it's a problem.. there's a real elegance to a plain old copy-paste from the editing textbox.  Shame to lose that.  I think Jeremy's EF is there to keep copy-paste working.

;D

[tagbert]

If you want a way to store the metadata and want to preserve
copy-paste, could you add a single metadata field similar to the "tags"
field? Then you could store any defined data in there as name=value
pairs much like data is stored in cookies. It could be manipulated by
any macro or by hand. It would be nice if it were exposed as attributes
to the tiddler object. maybe tiddler.metadata.namex=blah

Of course, I may be missing the real issue as I'm only just getting a
glimmer of how TW works under the hood.

[Jules]

Daniel, I think that one thing that is missing from the copy/paste
discussion is that it is ALREADY difficult to copy/paste a tiddler
because it is already three seperate fields.

So a copy/paste function is already needed (which would be easier than
the export/import that is currently the only alternative to three
copies and pastes). That being the case, having the metadata on a
seperate field such as tagbert suggests may well be the most elegent
solution.

[Saq Imtiaz]

That being the case, having the metadata on a
seperate field such as tagbert suggests may well be the most elegent
solution.

I worry about newbie users being exposed to and confused and scared by the metadata. Not to mention that the metadata might get rather large and 'messy' at times.....

Saq

[Jeremy Ruston]

> However, I'd also assumed that EMD would be useful for stuff like giving a
> tiddler a reminder date, so that tiddler could pop up somehow on that date.
> If I found a tiddler called "Mr T's Birthday" I'd want the reminder to still
> be working after I copy it into my own TW.

Yup, that's a good use case for EMD.

> I guess that's the main thing that's bugging me -- we're talking about a
> great, powerful, structured data model, but it seems to be crippled in
> certain situations.

How so? I guess I thought we were dealing with *two* great, powerful,
structured data models, and the issue is that they're confusingly
similar... :)

Cheers

Jeremy.

[Jules]

Its true Saq.

I was kind of thinking about an expandable field of some kind tucked
away at the bottom of the edit template.

[Daniel Baird]

On 20/04/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

> However, I'd also assumed that EMD would be useful for stuff like giving a
> tiddler a reminder date, so that tiddler could pop up somehow on that date.
> If I found a tiddler called "Mr T's Birthday" I'd want the reminder to still
> be working after I copy it into my own TW.

Yup, that's a good use case for EMD.

Except that the reminder info in EMD won't survive a copy-n-paste to my TW (unless there's a bad interface kludge that makes it copy-able).  I'd pity the fool who didn't get a reminder about Mr T's birthday.  ..I'm missing something again aren't I.

> I guess that's the main thing that's bugging me -- we're talking about a
> great, powerful, structured data model, but it seems to be crippled in
> certain situations.

How so? I guess I thought we were dealing with *two* great, powerful,
structured data models, and the issue is that they're confusingly
similar... :)

I really want to have a unified interface for these two data stores.  Are you planning on making data in the ED table code-writable, just like EMD?  Without the plugin author fiddling around in the tiddler content themselves, I mean.  This whole time I've iimagined that the ED was read only, but if it wasn't then I think that'd be much happier..

;D

[Jeremy Ruston]

> Except that the reminder info in EMD won't survive a copy-n-paste to my TW
> (unless there's a bad interface kludge that makes it copy-able). I'd pity
> the fool who didn't get a reminder about Mr T's birthday. ..I'm missing
> something again aren't I.

Indeed one would have to either use ImportTiddlers or a variant of the
MS LiveClipboard thingy to safely transfer a tiddler complete with
metadata. Just as you would if you wanted to preserve the existing
metadata.

> I really want to have a unified interface for these two data stores. Are
> you planning on making data in the ED table code-writable, just like EMD?
> Without the plugin author fiddling around in the tiddler content themselves,
> I mean. This whole time I've iimagined that the ED was read only, but if it
> wasn't then I think that'd be much happier..

The mechanisms are differently implemented, and serve different
purposes. I'd be concerned that unifying the interfaces would tend to
blur the distinction, and make it more likely that people will choose
to the wrong one for a particular application.

I do also see embedded fields as being readonly from a code perspective.

Having said all of that, I quite like the suggestion parsing the EF
into the EMD store...

[Udo Borkowski]

Having written most of the code for the meta data I think it would be not too hard to give read access to the embedded fields through the getValue function.

If you give me a function that accepts the name of an embedded field and returns the value (e.g. getEmbeddedFieldValue) I would just check for a certain prefix in the getValue routine (e.g. "embedded.") and call the getEmbeddedFieldValue instead of the metadata code.

I could even do a similar check for "embedded." in the setValue routine and throw an error if someone tries to modify an embedded methods this way.

Since "getValue" is a very generic name users may not be too surprised both for metadata, embedded data and "temporary" data. I guess the distinction between the various kinds of "fields" will become very obvious in the actual usage.


Udo

[Jeremy Ruston]

> Having written most of the code for the meta data I think it would be not
> too hard to give read access to the embedded fields through the getValue
> function.

I wasn't really thinking it would be hard, or at least only hard in a
subtle sense. The issue is whether it's desirable - I'm arguing that
despite the superficial similarities of them both being name-value
mechanisms, they are nonetheless different and shouldn't be unified.

The subtle aspect is that it might be very constraining to keep the
semantics the same, in terms of, for instance, the character set that
can be used in names. In EF users will be typing the field names
themselves, and it seems much less reasonable to restrict them, as
well as harder to enforce the restriction.

[Udo Borkowski]

>
>
>In EF users will be typing the field names
>themselves, and it seems much less reasonable to restrict them, as
>well as harder to enforce the restriction.
>

Good point.

Udo

[Bob McElrath]

Jeremy Ruston [jeremy...@gmail.com] wrote:
> > I guess that's the main thing that's bugging me -- we're talking about a
> > great, powerful, structured data model, but it seems to be crippled in
> > certain situations.
>
> How so? I guess I thought we were dealing with *two* great, powerful,
> structured data models, and the issue is that they're confusingly
> similar... :)

For my math cache concerns, why don't I *actually* use JSON?

I can create a tiddler jsMathCache that is written by my plugin. The
JSON code is available and small: http://www.json.org/json.js . When my
plugin is loaded it can read jsMathCache and populate it's cached data
structure.

So, plugin authors could persistantly store any data structure they
want this way. Copy/paste is simple because JSON is just text. (but
for jsMath I would never want to copy/paste this)

It isn't necessary for this to be part of the core though. Who else
would use it besides me? I can do this independent of what goes on with
Extended Attributes...

It would be nice to make the jsMathCache tiddler hidden from the user.
Otherwise it will always be on the top of the Timeline...

For that matter if hiden tiddlers are possible, why not make a
metadata.MyTiddler tiddler for each tiddler 'MyTiddler'? JSON could
load the contents of the metadata tiddler and make it available as a
javascript data structure. Likewise the metadata could be serialized on
output.

GetMetaDataForTiddler("Minesweeper").highscores[0].name = "Bob";

Plugins could simply create their own *javascript* data structure with
metadata.

metadata = {
author : "Bob",
needsTag : "systemConfig",
...
}

easy to cut/paste.

I still feel like I'm not understanding this discussion though. ;)

[Daniel Baird]

On 21/04/06, Bob McElrath <bob.mc...@gmail.com> wrote:

Jeremy Ruston [jeremy...@gmail.com] wrote:
> > I guess that's the main thing that's bugging me -- we're talking about a
> > great, powerful, structured data model, but it seems to be crippled in
> > certain situations.
>
> How so? I guess I thought we were dealing with *two* great, powerful,
> structured data models, and the issue is that they're confusingly
> similar... :)

For my math cache concerns, why don't I *actually* use JSON?

I can create a tiddler jsMathCache that is written by my plugin.  The
JSON code is available and small: http://www.json.org/json.js .  When my
plugin is loaded it can read jsMathCache and populate it's cached data
structure.

So, plugin authors could persistantly store any data structure they
want this way.  Copy/paste is simple because JSON is just text.  (but
for jsMath I would never want to copy/paste this)

I guess this means that your cache is a good candidate for EMD rather than ED.  No copy-paste requirement, and it'd keep the confusing stuff out of the user's view.

So, you could do it a couple of different ways.

- have data in EMD like this:

jsMath.cache.render00.raw = "x/y+10" (or whatever your expressions look like)
jsMath.cache.render00.rendered = "a bunch of spans and inline css and whatnot"

jsMath.cache.render01.raw = "x^2 + y^2 = z^2"
jsMath.cache.render01.rendered = "another bunch of spans and inline css and whatnot"

..then when you have an expression, loop through the renderXX.raw texts to see if you have it cached already.  If you have this data for each tiddler (instead of in only one place), the list won't be particularly long.

- or, have the whole structure in JSON:

jsMath.cache = "[a really long JSON string that is the entire cache]"

..and you'd just eval it in the plugin:

cacheobj = eval( '(' + thisTiddler.getEMD("jsMath.cache") + ')' );

..and then the cacheobj can be structured however you like:

if (cacheobj[expression])
    output = cacheobj[expression]
else
    cacheobj[expression] = (output = render(expression));

But anyway, it seems like EMD will make your life a fair bit easier when you are implementing your cache.

It would be nice to make the jsMathCache tiddler hidden from the user.
Otherwise it will always be on the top of the Timeline...

If you use EMD rather than ED, the mod date won't be changed (I assume we all agree that EMD mods shouldn't update the mod date?)

For that matter if hiden tiddlers are possible, why not make a
metadata.MyTiddler tiddler for each tiddler 'MyTiddler'?  JSON could
load the contents of the metadata tiddler and make it available as a
javascript data structure.  Likewise the metadata could be serialized on
output.

Interesting idea.  There's two different decisions here:
- store metadata in another, hidden tiddler (vs. store it in a hidden div in the existing tiddler)
- store metadata as JSON (vs. storing it as XML)

we've talked about the XML / JSON thing, but AFAIK no-one's come up with the hidden tiddler idea before.  It's actually kind of neat.. the store would even be 100% backwards compatible (unless we want to fiddle the attributes anyway, to get XHTML compliance).  The titles could just be "//METADATA//" + originalTiddlerTitle or whatever, hidden in normal operation, but accessible when required.  Seems like a cool idea.

;D