Should we stick to names as ID in external stores?

[Danielo Rodríguez]

Hello everyone,

I know that the topic of titles as tiddlers ID have been discussed several times for TW itself. But what about external stores? This is directly related to sync adaptors and external databases as store for the tiddlers.

Normally in databases there is a primary key, that is unique and incremental, and then the user data/fields. This have several advantages like traceability of a single element among all their changes. Sticking to the name as an unique identifier we will lose track of tiddler in every database system once you rename it. This is because how TW handles tiddlers: every change on a tiddler means deleting the old version and substitute it with a totally new one. This is against any DB system I know, and recently I have faced several problems (like loosing related attachments) because of that.

That's why I'm asking community for the best approach. Should we stick to titles as IDs when we sync to a remote database? Or should be treat the tiddler just as a collection of fields that we store in a particular entry of our DB? Of course the syncadaptor should handle the conversion however it's needed.

Thanks for your point of view.

[Danielo Rodríguez]

I just realized that this has another problem: when syncing several databases of different wikis, we will need an unique name for every tiddler, which is not always possible.

[Felix Küppers]

Hi Danielo,

I faced the same problem with tiddlymap. When just renaming a tiddler, you have no information about what was the original title and what were the original fields. This makes it hard for any external db or even a plugin (like tiddlymap) to track which original object changed. Therefore, I created an issue a few months ago that should append the previous tiddler object to the changed tiddlers list during refresh so we know the name and fields of the original tiddler.

https://github.com/Jermolene/TiddlyWiki5/issues/1550

Sticking to the name as an unique identifier we will lose track of tiddler in every database system once you rename it.

Your problem of "losing related attachments" is not nice but you have NO CHANCE to fix that as it is the tiddlywiki design and the same as link-breaking when changing the title. I think the only reason tiddlywiki preferred titles as primary key over ids is because in contrast to a db the user is directly involved and linking titles is easier to handle than linking ids or adding a layer that abstracts the id. This decision is highly understandable because it simplifies interaction and allows quick indexing by name but it comes with many disadvantages as you described them already, especially that renaming a tiddler will make you lose track. But as said, maybe https://github.com/Jermolene/TiddlyWiki5/issues/1550 could be a solution here.

For tiddlymap I partly solved it like this: I give all tiddlers used in the maps a unique id which helps me to trace name changes (because the id stays the same) and therefore I can still display relations even when you changed the name! When I detect dublicate ids (e.g. after cloning) I automatically fix them (and display a dialog) right when they occur.

I would love to see Tiddlywiki assign a unique (=repeatedly choose highly random value until unique) id to every tiddler it creates which cannot be changed by the user and is hidden (like the e.g. the "draft.of" field we cannot see or the "created" field that cannot be modified). So we can use it for db operations or other. Tiddlywiki itself does not need to use this id at all but it is nice for us to have it.

Or should be treat the tiddler just as a collection of fields that we store in a particular entry of our DB

Then you could as well use the title

Hope it helps
Felix

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/e53f5ef2-225a-4f88-8483-3d094300a259%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Danielo Rodríguez]

El martes, 26 de mayo de 2015, 11:55:06 (UTC+2), Felix Küppers escribió:

Hi Danielo,

I faced the same problem with tiddlymap. When just renaming a tiddler, you have no information about what was the original title and what were the original fields. This makes it hard for any external db or even a plugin (like tiddlymap) to track which original object changed. 

I knew, In fact I was thinking about your comments and your tickets when I was writing this.

 

Your problem of "losing related attachments" is not nice but you have NO CHANCE to fix that as it is the tiddlywiki design and the same as link-breaking when changing the title.

I do have a chance! please, don't tell me NO CHANCE, I get depressed easily :P  You were my inspiration, your plugin is my inspiration! Reading this is impossible coming from you has more meaning than anyone else!

I think that if I use some kind of unique id, I can avoid losing attachments and any information since I just have to update the content of the row, keeping the same ID.

 

I would love to see Tiddlywiki assign a unique (=repeatedly choose highly random value until unique) id to every tiddler it creates which cannot be changed by the user and is hidden (like the e.g. the "draft.of" field we cannot see or the "created" field that cannot be modified). So we can use it for db operations or other. Tiddlywiki itself does not need to use this id at all but it is nice for us to have it.

 Exactly. TW does not need to change, it can still working the same, just assign an unique ID, even if you don't use it! Maybe a plugin for this can be created. A good plugin, focused on that task, only adding an unique ID to every tiddler. Then every plugin that needs an unique ID can ask the user to include this plugin. This have several advantages, one of them is that your plugin does not need to take care of ID generation. The other advantage is, if every plugin adds an unique id we will end with tiddlers with one unique id per plugin,a central place for IDs solves this.

Or should be treat the tiddler just as a collection of fields that we store in a particular entry of our DB

Then you could as well use the title

I don't understand this last answer. Why should I use the title? That is indeed the problem!

[Mat]

Just two notes, to dismiss if nothing else:

I believe the intention is to introduce version handling for tiddlers. I assume this would mean saving copies of old versions(?)

When we get the TWederation set up, unique tiddler IDs might become of general interest.

<:-)

[Danielo Rodríguez]

Hello Mat

El martes, 26 de mayo de 2015, 13:18:29 (UTC+2), Mat escribió:

Just two notes, to dismiss if nothing else:

I believe the intention is to introduce version handling for tiddlers. I assume this would mean saving copies of old versions(?)

Not exactly. I'm just talking about being able to rename a tiddler without losing track of it, and many other features that comes with unique IDS. Versioning is just one of those features.

 

When we get the TWederation set up, unique tiddler IDs might become of general interest.

This is something that confuses me. Jeremy has expressed his interest on federation, but all the comments and attitudes of an important part of the community are against the unique IDs. I don't know how do they plan to handle this.

[Felix Küppers]

Hi Danielo,

I do have a chance! please, don't tell me NO CHANCE, I get depressed easily :P  You were my inspiration, your plugin is my inspiration! Reading this is impossible coming from you has more meaning than anyone else!

Sorry, sorry. I didn't mean to say no chance, I just wanted to say you need to be prepared to develop some workarounds :) Nothing is impossible it is just a matter of creativity and we know you are one of the creative guys around ;)

 Exactly. TW does not need to change, it can still working the same, just assign an unique ID, even if you don't use it!

YES! exactly my thinking!

Maybe a plugin for this can be created. A good plugin, focused on that task, only adding an unique ID to every tiddler. Then every plugin that needs an unique ID can ask the user to include this plugin.

Yes, good point! TiddlyMap already has this build in and I would really like to outsource it as plugin but my problem so far has been that I am afraid of telling the user "tiddlymap is easy to install, you just need to drag and drop 100 separate plugins".

This have several advantages, one of them is that your plugin does not need to take care of ID generation. The other advantage is, if every plugin adds an unique id we will end with tiddlers with one unique id per plugin,a central place for IDs solves this.

Exactly what I am thinking. Sorry, but I am very busy at the moment but as soon a I find the time, I'll create a plugin from the code I am already using at tiddlymap and write you a message.

Or should be treat the tiddler just as a collection of fields that we store in a particular entry of our DB

Then you could as well use the title

I don't understand this last answer. Why should I use the title? That is indeed the problem!

Sorry, this is a misunderstanding. I thought you want to use the tiddler fields as "multi attribute primary keys" (also known as combined primary key). Say "prename" + "surname" + "adress" = primary key. So just ignore this part of my answer.

-Felix

[Danielo Rodríguez]

Hello Felix 

Sorry, sorry. I didn't mean to say no chance, I just wanted to say you need to be prepared to develop some workarounds :) Nothing is impossible it is just a matter of creativity and we know you are one of the creative guys around ;

No problem, I was just joking (I think  you got it ;D ) and Thank you for the kind words. 

Yes, good point! TiddlyMap already has this build in and I would really like to outsource it as plugin but my problem so far has been that I am afraid of telling the user "tiddlymap is easy to install, you just need to drag and drop 100 separate plugins".

TIddlymap is a totally special plugin, and I think it deserves some effort in its installation. Anyway, I see TiddlyMap more like a tiddlywiki flavor/application rather than a plugin. I don't see people installing tiddly-map in their day to day use wiki, but creating a separated wiki for that. You have already very good documentation about your plugin and it's very easy to download an empty wiki with tiddlymap, so I don't see it as a deal breaker. Anyway, you could keep the ID generation as part of the core of TW-map but create the plugin anyway.

 

Exactly what I am thinking. Sorry, but I am very busy at the moment but as soon a I find the time, I'll create a plugin from the code I am already using at tiddlymap and write you a message.

No problem. We have lived with this from the beginning, so we can wait a couple of weeks more. I really trust that you will end with a great product.

 

Sorry, this is a misunderstanding. I thought you want to use the tiddler fields as "multi attribute primary keys" (also known as combined primary key). Say "prename" + "surname" + "adress" = primary key. So just ignore this part of my answer.

Ok, clear now.

Thank you very much Felix for your inputs. 

[PMario]

On Tuesday, May 26, 2015 at 11:06:23 AM UTC+2, Danielo Rodríguez wrote:

I just realized that this has another problem: when syncing several databases of different wikis, we will need an unique name for every tiddler, which is not always possible.

TiddlyWeb defined Bags and Recipes to solve this problem.


see: http://tiddlyweb.readthedocs.org/en/latest/tiddlyweb.web.handler.html
and: http://tiddlyweb.tiddlyspace.com/bag
and: http://tiddlyweb.tiddlyspace.com/recipe

-mario

[PMario]

IMO we should go with an UUID V4 as a tiddler ID.

Felix has an algorithm in his tiddlymap library. ... IMO we should make a pull request for the core.

http://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_.28random.29

-m

[PMario]

On Tuesday, May 26, 2015 at 11:55:06 AM UTC+2, Felix Küppers wrote:

Your problem of "losing related attachments" is not nice but you have NO CHANCE to fix that as it is the tiddlywiki design and the same as link-breaking when changing the title. I think the only reason tiddlywiki preferred titles as primary key over ids is because in contrast to a db the user is directly involved and linking titles is easier to handle than linking ids or adding a layer that abstracts the id.

This is because TWs initial nature is a single file based system and non techy users get very confused by machine IDs.

It is very, very unlikely, that Jeremy will change this. ... but it doesn't prevent us from adding an UUID field with plugins.
We as plugin authors should just find a common path. ..

-mario

[PMario]

On Tuesday, May 26, 2015 at 12:26:45 PM UTC+2, Danielo Rodríguez wrote:

I do have a chance! please, don't tell me NO CHANCE, I get depressed easily :P  

:)  As I wrote imo used UUIDs ... You will have a bit more work to do. Let the users use names and internally use your uuids.

m

[PMario]

On Tuesday, May 26, 2015 at 1:51:29 PM UTC+2, Danielo Rodríguez wrote:

When we get the TWederation set up, unique tiddler IDs might become of general interest.

This is something that confuses me. Jeremy has expressed his interest on federation, but all the comments and attitudes of an important part of the community are against the unique IDs. I don't know how do they plan to handle this.

I think, the important part here is. Users will always face tiddler names only. .. you can easily prefix them.

eg: pmario/myTiddler, rdanielo/myTiddler, .... So TW names are different, depending on there origin. .. This is called a "namespace"
We use this mechanism a lot with system tiddlers, so all the functions should be in the core already.

-m 

[Alex Hough]

Emphasising cloning approach

How about this for a work-around: When a tiddler is cloned, a ClonedFrom field is created. When you want to change the title of a tiddler, clone it then change give it a new name.

There could be a "refactor links" plugin which could seek out cloned tiddlers, references from the tiddlers being cloned, then alter links. The cloned tiddler could be given a "1" in a isDead field.

This work-flow would emphasise cloning and the self-replicating aspects of TW. It promotes a cautious approach where tiddlers are not deleted: if you want to change it clone it! I have sometimes deleted a tiddler and really wished there was a undo button. Cloning is also more closely related to "forking" in GitHub: its more of the zeitgeist than deleting...

Clone to delete

It might be worth considering changing the delete buttons function: rather than deleting it, it could just be demoted in the evolution of the TiddlyWiki.

Delete could also be a cloning function: instead of it being removed from the store forever, a new tiddler could be created with a tittle appended with "deleted", the tag "isDead" and a field "clonedFrom". "isDead" tiddlers could then be hidden, like shadow tiddlers.

(The isDead technique was shown to me by a friend who developed his own "fractal database" method. He was inspired by attempting to model Stafford Beers recursive Viable System Model. Given that names of things are subjective and socially constructed (from some philosophical standpoints) emphasising the process and relationships hidden behind the name can be seen as more important.

As a user, I want to create new tiddlers and give them whatever name I want. I want to be able to track the development of my idea from "original" though various "clones". When we start to use words like "parent" and "child" we are adopting a metaphor related to sexual reproduction – subtly different from "cloning": its the difference between a plant grown from a seed produced from a pollinated flower and one produced from a cutting. TW is all about growing from cuttings..... TW is a quine, a self replicating program. I think this fact is a sleeping giant in our thinking about branding

Cloning language, cloning metaphor: TW –The Quine Wiki

I think this would be a strategy for branding and marketing. Bringing the philosophy related to the underlying computer science and the language we use to create tiddlywikis and knowledge closer together will enable cohesive devel.

We could consider the wiki and the quine as a muse and metaphor for some of the issues related to knowledge creation identified by Quine and elaborated on by Hofstader in GEB

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6ff8f85c-80de-4cb6-99cc-f530739c8dc7%40googlegroups.com.

[Matabele]

Hi Danielo

It would be advantageous to give each instance of TW a  unique ID -- in this way, two different tiddlers in two TW's with the same title can be distinguished from one another -- in particular, this may involve two different revisions of the same source tiddler. 

I believe it would also be advantageous to give each tiddler a unique ID via a plugin (could be a hash of the TW ID and the title of the tiddler) but this would be redundant if the idea above were adopted (as titles within each TW must be unique in any case.) 

regards

[Matabele]

Hi Danielo

I'm being dumb -- to use a random hash as an ID for a tiddler would be a waste of resources. Rather -- each tiddler should be given a hex string as an ID, where the hex string determines that tidders position in a Table of Contents (TOC). 

The existing TOC mechanism is cumbersome to use -- useable in a static situation, but useless for generating a dynamic TOC. What would be really nice is a TOC that functions alike Workflowy -- where each entry is the title (clickable link) of the respective tiddler. 

This could be implemented if each tiddler were allocated a hex string as an ID which marked it's position in the TOC -- 16 branches at each level (I suppose two hex characters might be used for the first level -- allowing 256 branches at the root level.)

Once this mechanism had been implemented (adding a suitable hex string field to each tiddler) -- creating a TOC tiddler (in the sidebar) to display the hierarchy should not be too difficult. With a little more effort, a drag and drop editor could be written to ease the creation and repositioning of tiddlers within the tree -- and TW could mimic the behaviour of Workflowy!

regards

[PMario]

On Wednesday, May 27, 2015 at 4:30:48 AM UTC+2, Matabele wrote:

I believe it would also be advantageous to give each tiddler a unique ID via a plugin (could be a hash of the TW ID and the title of the tiddler)

Title hashes are not unique. .. eg:

 - create a tiddler "test" at example.com
 - calculate the hash
 - delete the tiddler
 - after a month create a tiddler "test" at example.com
 - the hash will be the same, but the tiddler is not. ---------> problem!

-m

[PMario]

On Wednesday, May 27, 2015 at 7:13:45 AM UTC+2, Matabele wrote:

This could be implemented if each tiddler were allocated a hex string as an ID which marked it's position in the TOC -- 16 branches at each level (I suppose two hex characters might be used for the first level -- allowing 256 branches at the root level.)

What if I have 257 elements in the the root? We need to expand to 3 bytes aka 4096 possibilities. .... but what if we need 4097 elements in one hierarchy level. ........
IMO Murphy's law [1] applies here.

Murphy's law is an adage or epigram that is typically stated as: Anything that can go wrong, will go wrong.

-m

[1] http://en.wikipedia.org/wiki/Murphy%27s_law

[Matabele]

Hi Mario

In my view -- 16 branches at each level would be sufficient -- it should be advantageous to structure categories in the TOC in depth, rather than having long lists in parallel. The current TOC at tiddlywiki.com has 12 branches at root level and no more than 20 branches in the second level (which could easily be organised to <16.)

At a maximum, 2 x hex (256) branches should be more than sufficient for any user (and this is likely necessary only in the first couple of layers.) Perhaps it would be better to use the full range of ascii characters -- this would give about 95 possibilities (case sensitive) at each level whilst using only 1 character per level.

regards

[Danielo Rodríguez]

I think, the important part here is. Users will always face tiddler names only. .. you can easily prefix them.

As you said, we want to focus on humans. People usually don't want to add a prefix to each of their tiddlers only to determine the wiki or user they belongs to. Also, we don't have anything that enforces usernames.

 

 

 IMO we should go with an UUID V4 as a tiddler ID. 

Felix has an algorithm in his tiddlymap library. ... IMO we should make a pull request for the core. 

It is very, very unlikely, that Jeremy will change this. ... but it doesn't prevent us from adding an UUID field with plugins. 
We as plugin authors should just find a common path. .. 

This both things have confused me a bit. You said more or less the same that Felix and me, but I'm not sure if you subscribe our ideas or not. What do you think exactly? A pull to the core? A separate plugin? For me, providing unique IDs it's a feature that should be inside the core. Even if the core does not takes care of them neither uses them. If you think this should be a plugin, I'm fine with that as soon as all plugin authors stick to the same plugin. 

[Jeremy Ruston]

This topic has come up several times; I probably ought to add a faq to tiddlywiki.com/dev.

Here's one thread from last year about it:

https://groups.google.com/d/topic/tiddlywiki/Y1V7oyBLpSs/discussion

The tl;dr version is that TiddlyWiki already uses unique IDs to identify tiddlers, the "title" field. The intention is that people should be able to use GUIDs as titles if that is what they need but that support is not yet complete (eg more support for "caption" field). I believe that the end result would be the same as the proposal here.

To respond to a few points in the thread:

* Wikis generally do not use IDs because it means that it is no longer possible to uniquely identify a tiddler with a human readable name

* GUIDs are not needed to tiddler revision handling

* I don't think the suggestion in #1550 is feasible; I've added a note there

* Federation does not need GUIDs, nor for tiddler titles to be globally unique

Best wishes

Jeremy.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/83d128aa-629b-4573-9880-1349bc3eede7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jeremy Ruston
mailto:jeremy...@gmail.com

[Danielo Rodríguez]

Hello Jeremy,

I don't know why every time someone talks about UIDS everyone thinks automatically in titles. I'm not talking about using this IDS as titles, neither to identify a tiddler in any way. We are just talking about assigning one unique ID to each tiddler. Why is this so problematic? Why everyone thinks that we want to kill the current title mechanism? It's not about that.

[Jeremy Ruston]

Hi Danielo

I understand that you are suggesting a new UUID field that is independent of the title field. I'm responding that there is already a unique ID field for tiddlers; it's pointlessly expensive to enforce two unique IDs. Hence my proposal of what I think is a viable way for TiddlyWiki to be extended to support GUIDs.

Best wishes

Jeremy.

On Wed, May 27, 2015 at 1:39 PM, Danielo Rodríguez <rdan...@gmail.com> wrote:

Hello Jeremy,

I don't know why every time someone talks about UIDS everyone thinks automatically in titles. I'm not talking about using this IDS as titles, neither to identify a tiddler in any way. We are just talking about assigning one unique ID to each tiddler. Why is this so problematic? Why everyone thinks that we want to kill the current title mechanism? It's not about that.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f59c8696-374f-4034-ae95-bf720d03b244%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danielo Rodríguez]

Hello Jeremy

I think we are moving from the original topic. My real question was about external databases. So I want advise from the community and more experienced users if sync-adaptors should take care of giving each tiddler an unique ID or we should stick to tiddlers titles to make the process more easy.

Thanks to anyone.

[Felix Küppers]

Hi Jeremy,

I understand that you are suggesting a new UUID field that is independent of the title field. I'm responding that there is already a unique ID field for tiddlers; it's pointlessly expensive to enforce two unique IDs.

I don't think it is pointless in Danielo's case because the title is a unique id field but when it changes he has no way to update any "attachments" in his database that linked to the object that was identified by the title.

[Felix Küppers]

Hi Danielo,

if you do not have any attachments and just want to recreate the wiki (its tiddlers) in a database then it would be ok to stick with titles as ids.

Everytime Tiddlywiki deletes a tiddler you delete the document in the db.
Everytime a tiddler gets modified (added or changed) you update the tiddler in your db.

This works well for a non-relational db where data is not linked (no attachments).

In a relational db you will lose track when renaming is done (as you already said) just as links break in tiddlywiki. In this case I will add a plugin that adds UUIDs to tiddlers.

-Felix

[Felix Küppers]

yes, mario. I agree. I think there is a strong interest in having each tiddler getting a unique id as field for various reasons!
 

-mario

[Felix Küppers]

Felix has an algorithm in his tiddlymap library. ... IMO we should make a pull request for the core.

Its basically yours with some slight modifications (only lowercase and removed parameters to specify the length) ;)