New JSON Store Area Now in v5.2.0 Prerelease

535 views
Skip to first unread message

Jeremy Ruston

unread,
Jul 14, 2021, 7:35:46 AM7/14/21
to tiddl...@googlegroups.com
It doesn’t sound very exciting, but the new JSON-based store area has now been merged into the prerelease.


This is the big change that has prompted us moving from v5.1.23 to v5.2.0. It brings one major benefit to all users of TiddlyWIki: the characters used in field names are now unrestricted, making it possible to use fieldnames like "My^Field” or “☃️”.

While that is an important change that we hope will lead to the development of interesting new techniques for working with data in TiddlyWiki, the biggest risk of this change is that it is not fully backwards compatible.

That is to say that any tools that work directly with TiddlyWiki files will no longer work as expected, and need updating to understand the new format.

We have tried to anticipate the problems that this change might cause, for example working with Simon Baird to fix TiddlyHost ready for the change. You can follow the technical discussions here:


But now we need everyone’s help to try the new version, to help us identify any incompatibilities before the final release.

Please try the prerelease with as many different saving mechanisms and server backends as possible. You can report any problems here, or by opening a new ticket on GitHub.

For me personally this feels like a big milestone. TW5 has used the same DIV-based tiddler store area since the very beginning in 2011, having directly inherited it from TiddlyWiki Classic. I reused the old format to give TW5 compatibility with the ecosystem of tools around TiddlyWiki Classic. Now that the TW5 ecosystem is so much bigger, it makes sense to move to a store area format that has fewer restrictions and is easier for developers to work with.

Just for fun, I wrote a Node.js one-liner to extract the tiddlers from a TiddlyWiki HTML file. In contrast, a few hundred lines were needed to read the old format:

var fs=require("fs"),tiddlers=JSON.parse(fs.readFileSync("./editions/prerelease/output/index.html").toString().match(/<script class="tiddlywiki-tiddler-store" type="application\/json">([^<]*)<\/script>/)[1]);console.log(tiddlers)

Questions and comments welcome,

Best wishes

Jeremy

Mohammad Rahmani

unread,
Jul 14, 2021, 12:59:04 PM7/14/21
to tiddl...@googlegroups.com
Hi Jeremy,

 Thank you for all your efforts. Does this testing include single file save with tools like Timimi? Or is this more related to server backends?

 I noticed I have to update the Tiddler Commander plugin, as it forces users to use lowercase names for field names and remove all nasty characters! Now this restriction is lifted!
 

Best wishes
Mohammad


--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/10C57C7B-76DE-4FDE-AB94-F778FEE85BC2%40gmail.com.

PMario

unread,
Jul 14, 2021, 1:07:58 PM7/14/21
to TiddlyWiki
On Wednesday, July 14, 2021 at 6:59:04 PM UTC+2 Mohammad wrote:

 Thank you for all your efforts. Does this testing include single file save with tools like Timimi?

It's related to single file wiki actions and especially 3rd party plugins. So it should be tested with real-world wikis. BUT

MAKE BACKUPS and don't use your production wikis to test. ... Use "clones" ;)
 
Or is this more related to server backends?

no
 
I noticed I have to update the Tiddler Commander plugin, as it forces users to use lowercase names for field names and remove all nasty characters! Now this restriction is lifted!

That's right

-m

Jeremy Ruston

unread,
Jul 14, 2021, 2:00:56 PM7/14/21
to tiddl...@googlegroups.com
Hi Mohammad, Mario,


 Thank you for all your efforts. Does this testing include single file save with tools like Timimi?

It's related to single file wiki actions and especially 3rd party plugins. So it should be tested with real-world wikis. BUT

MAKE BACKUPS and don't use your production wikis to test. ... Use "clones" ;)

Yes, saving solutions like Timini are exactly the sort of thing that needs testing.

Or is this more related to server backends?

no

Actually, these changes do impact server backends (see the changes to the TiddlyWeb plugin in the PR), and so it is definitely worth testing them too.

Best wishes

Jeremy.

 
I noticed I have to update the Tiddler Commander plugin, as it forces users to use lowercase names for field names and remove all nasty characters! Now this restriction is lifted!

That's right

-m

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

Mohammad Rahmani

unread,
Jul 14, 2021, 2:31:13 PM7/14/21
to tiddl...@googlegroups.com

Thank you both for your clarification!
I did take some backups!

This is a crazy test and it works. 
Windows 10 User: use windows key + ; to see emojis


image.png



Best wishes
Mohammad


To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/17BC643D-2EFF-4013-9D22-BEE6ED198BA9%40gmail.com.

Jeremy Ruston

unread,
Jul 14, 2021, 3:41:16 PM7/14/21
to tiddl...@googlegroups.com
A key area for testing is any process that reads or writes a TiddlyWiki HTML file.

With savers, that means testing that saving a wiki and reloading the changes works as expected, including with tiddlers that have emojis in fieldnames.

With syncers, that means testing that changes get synced to the server, and that subsequently reloading the wiki works correctly.

Another place where TiddlyWiki HTML files are read is the import process, so it would be useful to test that old and new format wikis import as expected.

Many thanks,

Jeremy



On 14 Jul 2021, at 19:31, Mohammad Rahmani <mohammad...@gmail.com> wrote:

ďťż

Thank you both for your clarification!
I did take some backups!

This is a crazy test and it works. 
Windows 10 User: use windows key + ; to see emojis


<image.png>




Best wishes
Mohammad

TiddlyTweeter

unread,
Jul 15, 2021, 1:36:31 PM7/15/21
to TiddlyWiki
Ciao Jeremy

This more a footnote than a big comment.

First I am grateful you documented the change so well and so thoroughly!

Just FYI, I have done a lot of "direct insertion" over the last year for wiki that simply host materials I pre-process in  Powershell and directly insert into TW.

The new structure actually looks like it will simplify what I need to do as  appending to to end of the TW will work just as well, and better, now as the mid-insertion I was doing before. That, for me, has benefits. Not least it makes it much easier to identify problems if I make a boo-boo.

Many thanks
TT

Simon Baird

unread,
Jul 15, 2021, 5:11:47 PM7/15/21
to TiddlyWiki
Speaking of Tiddlyhost, you can try a prerelease version of TiddlyWiki 5.2.0 there now.

Do that by clicking "Advanced Options" when creating a site (at https://tiddlyhost.com/sites/new ) to access the empty version selector.

-Simon


Ps, it is a little confusing that the default empty file on Tiddlyhost is also an older pre-release version. There is a reason for it - Tiddlyhost doesn't work smoothly with the official 5.1.23 version of TW, so it's been using a pre-release build all along. As soon as there's an official 5.2.0 release, I'll likely change it so Tiddlyhost uses that as the default empty.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

springer

unread,
Jul 16, 2021, 12:06:03 PM7/16/21
to TiddlyWiki
Jeremy and all: I "broke" the pre-release this morning, recklessly pushing the limits with field-names (saving on github).

Alas, it's not easy for me to reconstruct what the last straw was. I don't know how to restore (to the most recent non-broken version) with github, so as to retrace the most recent steps. (What's public-facing, now, is now a very early state that I restored from Chrome, where I had opened it up prior to digging in much: https://springerspandrel.github.io/tw/new-json-tw-experiment.html)

Here's the github history url: 
https://github.com/springerspandrel/tw/commits/c5a8c5807f22eb35cec72baf9b5a496a1d7a67ac/new-json-tw-experiment.html

My best quick reconstruction: 

1. I had begun by pushing the boundaries with field-names (Japanese words (yay!); emoji; names with spaces and eventually getting bold with special code-like strings). I switched from Chrome to Safari to observe some of the ways that the browsers differ in how/whether they display emoji (such as ⁉️)

2. I spent a while tinkering with getting Shiraz dynamic tables to show a row for each non-system field, then making each field-name into a tiddler by filling in its caption column and various other silly field parameters, like a ★★★☆☆ rating column. (I discovered Shiraz can apparently can also render columns not only for emojis and such, but also for field names with spaces, if you just put [[field name]] in double-brackets). All was behaving well, and saves were surprisingly fast...

3. I had moved on to trying to define a macro to expand within Shiraz, to get Shiraz to work with the string $(whatever-this-tiddlers-non-system-fields-are)$ as column set... (I had pulled in a tobibeer code about exercise-values to reverse-engineer, so any version with the string "exercise" is close to when it crashed).

It was while I was tinkering with that macro-variable tiddler (and perhaps with a stray keystroke I unwittingly committed the tiddler and triggered a save?) that the whole thing crashed. Somehow with one keystroke  the browser window blinked to white, and all attempts to reload yielded a completely blank screen (until I over-wrote with the much earlier version from other browser).

So, as Jeremy says, the pre-release is not for vital projects yet. ;)  

I'm happy to "help" by trying to break it again, but since I'm not sure what went wrong, I'll take a break until someone else has some insight. If I need to do anything at github to help others see "under the bonnet" ;) , I'm happy to do that. I'd also welcome tips on how to get github to actually download files from the history (When I try, I just get html within a new browser tab).

-Springer

Jeremy Ruston

unread,
Jul 16, 2021, 12:30:19 PM7/16/21
to tiddl...@googlegroups.com
Hi Springer

Was it a "red screen of embarrassment" crash? It would certainly be helpful to pin it down if we can.

> I'd also welcome tips on how to get github to actually download files from the history (When I try, I just get html within a new browser tab).

It's a bit convoluted: first, click on the three dots at the top right of the file in the commit listing:

image2.jpeg
image1.jpeg

Saq Imtiaz

unread,
Jul 16, 2021, 12:32:46 PM7/16/21
to TiddlyWiki
I don't have the opportunity to debug this at the moment but I believe this is the commit responsible:

Seems like the boot process just doesn't load anything and we get a blank screen.

springer

unread,
Jul 16, 2021, 12:37:26 PM7/16/21
to TiddlyWiki
No red screen! Just "poof" no tiddlywiki interface.

But here's one clue: I tried exporting a json set of tiddlers from the non-crash "early" version (which you can load at url above), and THAT json export file, when re-imported, yields an error that suggests the dynamic table's json entry didn't get proper "closure" (the error is "between" that one and the next tiddler in the json list).

So, it may be safe to assume, for now, that something about Shiraz's dynamic tables needs updating. Which of course would be in some ways surprising if it weren't so, since it was built within the old assumptions about how fields can be named.

Also, I *do* have a window with the failed file "loaded" (white screen). So I could use "inspect" to get at the json list...

From Clipboard.jpg

springer

unread,
Jul 16, 2021, 12:50:12 PM7/16/21
to TiddlyWiki
Since I need to be away for a while, and in case it's helpful, here's a link to the entire json segment (copied from "inspect" tool from crash window):


Oddly, I can drag this whole json file onto the official pre-release, and apart from duplications and plugin version-conflicts, I see no errors on import preview, and import seems to work fine. 

But of course I'm not saving to github from there...

Tapping out for now...

-Springer

Saq Imtiaz

unread,
Jul 21, 2021, 3:34:54 PM7/21/21
to TiddlyWiki
@Jeremy regarding this bug, I think it is related to having a space character in a field name.


Create a new tiddler called "test" with a field name "my field" with a value of "hello".
Save and open the resultant file.

You get a blank white screen, no errors in the console.

Saq Imtiaz

unread,
Jul 21, 2021, 4:11:52 PM7/21/21
to TiddlyWiki
I believe this check for field names is the cause:

TW Tones

unread,
Jul 21, 2021, 9:24:54 PM7/21/21
to TiddlyWiki
I am glad you are onto this bug, spaces in fieldnames,

However am I the only one who still intends to keep field names to a defacto naming standard such as i-am-afield no caps or spaces? so It is easy for me to tell field names apart from text and titles!

Of course I look forward to naming fields using tiddler titles (and the value a relationship type etc..). and to contain special prefixes or suffixes such as "-link" or (link)  but as a rule I would like a little structure and naming rules. 
This always helps retain a degree of simplicity when managing lots of complexity.

Regards
Tones


PMario

unread,
Jul 21, 2021, 11:56:18 PM7/21/21
to TiddlyWiki
On Thursday, July 22, 2021 at 3:24:54 AM UTC+2 TW Tones wrote:
 
However am I the only one who still intends to keep field names to a defacto naming standard such as i-am-afield no caps or spaces? so It is easy for me to tell field names apart from text and titles!

No -- you are not.
 
Of course I look forward to naming fields using tiddler titles (and the value a relationship type etc..). and to contain special prefixes or suffixes such as "-link" or (link)  but as a rule I would like a little structure and naming rules. 

You wrote, you want to have tiddler titles as field names. So your field-rules have to be the same or a superset of your title rules. I would be surprised, if "no caps or spaces" is one of them.
 
This always helps retain a degree of simplicity when managing lots of complexity.

Oh it will be funny, to deal with all the support requests here in the group, because some ancient and / or unsupported plugins will break. ... I'm not even sure, what my own plugins will do with "spaces" in field names. ...  I will have to check that.

have fun!
mario

TW Tones

unread,
Jul 22, 2021, 12:24:46 AM7/22/21
to TiddlyWiki
Mario,

I am glad someone concurs. For me with my de facto standards and the new freedom, if a field name looks like a tiddler title it possibly is.

Do you think many ancient and / or unsupported plugins will break?,. if they use fields even extensively the issue will only be if they are used against new fieldname standards. And until now at least we tended to honour any values if not fieldnames.

I will be honest though few if any system tiddlers I create have spaces or quotes of any kind. even tags are usually single text strings, and I do think I will stick to this, as it simplifies code. but yes, use it when the exception makes sense.

I have not yet considered what we may achieve by having field names that can be wikified into something else, but again I personally intend to proceed with caution. Perhaps the documentation can give some advice to this effect, for example currently the fieldname restrictions help us when generating csv files and if field names (and values) contain spaces, or commas and quotes this may no longer be true. The documentation should continue to list the previous limited naming standards as well. 

I must say I also have de facto naming standards for titles even although they are quite flexible, in fact most that I code follow a simple and limited range. I like to leave the flexibility for when its needed. The one exception is when a title is also textual content like a sentence or paragraph.

Regards
Tones

springer

unread,
Jul 22, 2021, 10:50:25 AM7/22/21
to TiddlyWiki
Tones and Mario,

For public-facing projects, being able to name tiddlers with natural language expressions (including spaces) has been essential, and it seems much of the expanded power of fields in 5.2.0 will come from scanning for matches between field names and tiddler titles. 

(Emojis and such may seem trivial, but certain specialized unicode characters like Σ, plus expressions in two-byte languages like 字, are surely going to be leveraged by a significant user base as well.)

So, as much as it may seem logical to stick to CamelCase titles and a-z_special.string fieldnames for certain users, it seems the flexible-field-names horse is going to be galloping out of the barn as soon as the new version is available.

Might we figure out some way to test the compatibility of existing plugins (such as Shiraz) more systematically than by trial and error?

-Springer

Jeremy Ruston

unread,
Jul 22, 2021, 11:57:34 AM7/22/21
to tiddl...@googlegroups.com

On 21 Jul 2021, at 21:11, Saq Imtiaz <saq.i...@gmail.com> wrote:


Thanks for your help with this, the problem with fields with spaces in their names has been resolved here:


The update will be live at https://tiddlywiki.com/prerelease shortly,

Best wishes

Jeremy

springer

unread,
Jul 22, 2021, 3:23:58 PM7/22/21
to TiddlyWiki
Jeremy and all,

I've played around with the updated prerelease file, starting by importing all the tiddlers from the "broken" version. So far, saves on github are going fine.

Only one (!) major glitch. If you name a field with a double-angle-bracket expression (Who would do such a thing?!), Shiraz gets confused. (Probably other tools would get confused too, but Shiraz is the thing I've been testing.)

For one proof-of-concept, I created a little ViewTemplate footer that provides a dynamic table overview of any tiddlers that use the current tiddler's title as a (non-empty) field name.

I do see some minor glitches with ViewTemplates (contents not looking as they should, conditional filters not behaving as they should), but it's possible my own limited trouble-shooting skills are to blame. It's also possible that some interaction with one or more <<dangerously named tiddlers>> is to blame.

If anyone wants to play around see here: 

-Springer

TW Tones

unread,
Jul 23, 2021, 1:14:08 AM7/23/21
to TiddlyWiki
Springer,

I totally agree, we need to prepare for any ungainly titles and fieldnames. I suppose my point is to not to abuse this and restate the value of keeping a limited set of naming rules when developing macros and plugins. Basically proposing the use of Okams razor. The rules are meant to be broken but giving one self some naming standards will always help. If I write a plugin or macro I would not make unnecessary use of the freedom, but I certainly will where it is of benefit. Of course for some with different languages they have good reason to make use of the new possibilities.

I suppose for me the hard part is sharing this knowledge and experience to advise people they may have being given sufficient rope to kill there wiki, and to remain circumspect.

Regards
Tones

PMario

unread,
Jul 23, 2021, 8:37:23 AM7/23/21
to TiddlyWiki
Hi,
Nice tests, but you mainly test with tiddler titles, and not fields. Even if you have strange field names in your tiddlers.

https://springerspandrel.github.io/tw/new-json-tw-experiment.html#Show%20related%20field-table%20(ViewTemplate) .. The filter is probably broken, because the number of opne/close braces are wrong. The last closing brace is missing ...first[]]

BUT I didn't test the code.

-mario

springer

unread,
Jul 23, 2021, 9:58:23 AM7/23/21
to TiddlyWiki
Mario,

Thanks for the bracket fix! (I should have caught that, but was getting bleary-eyed.)

When you say, "you mainly test with tiddler titles, and not fields. Even if you have strange field names...," I'm not sure what you mean. I have focused mostly on having strange field names that correspond to strange tiddler titles, partly because I anticipate that the newly open field-name space will get half its value from letting field names correspond to tiddler titles. And because of the centrality of Shiraz to my own workflow, I have been testing how Shiraz (especially dynamic tables) behaves with the new field name possibilities.

My other anticipated use for strange field names is to serve as compact headings in dynamic tables, where the difference between "tennis" and 🎾 or "recyclable" and ♻︎ (in a field with √ or X values) is conservation of horizontal table space.

So: What other kinds of tests do you (and others) see as important? Though my coding skills are minimal, the challenge to test the limits of a new feature is one area where I enjoy chipping in as best I can.

-Springer

PMario

unread,
Jul 23, 2021, 6:23:27 PM7/23/21
to TiddlyWiki
Hi,
There was no offence intended in my last post. Your tests are good and you already discovered a problem, that would have caused a severe problem in production, if you hadn't found it. ... So -- well done!

What I wanted to say is that we need to test all kind of filter operators, that can deal with fields. We need to test them with "strange" field names eg: {{xx}} [[xx]] {x} [x] <<xx>> <x> and so on. ..

I'm pretty sure some of them will cause problems. Either filter expressions can't be created or they cause parser problems.

For some solutions we will need the help of variables similar to some regexp expressions with tiddler titles. See the last example at https://tiddlywiki.com/#regexp%20Operator%20(Examples)

and so on. I think, the range of problems that will come up are very similar to those we had or have with "strange tiddler titles". ... We'll see.

-mario

TW Tones

unread,
Jul 23, 2021, 9:53:46 PM7/23/21
to TiddlyWiki
Springer,

From a design perspective for column headings I would stick to a standard fields such as column-heading with the value"🎾" on a tiddler called "tennis", or as the case may be a sport tiddler.

Already dynamic tables support alternate column names in a standard field/value and there is not reason in my view to tamper with fieldnames.

I am sure there are reasons, but your suggestion to me, is like I posted previously, you may be about to hang yourself with the extra rope you have being given for field names. I would avoid this excursion unless you can establish a good reason otherwise you are possibly forgoing the ability to use your tiddlers, with the large library of plugins and macros that have gone before.

Just my viewpoint
Tones

Flibbles

unread,
Jul 24, 2021, 3:21:17 PM7/24/21
to TiddlyWiki
Hey all,

Question: Is this going to impact the way NodeJS servers saves tiddlers to file? Will they no longer be .tid files, and maybe be .json files instead?

-Flibbles

Saq Imtiaz

unread,
Jul 24, 2021, 4:36:50 PM7/24/21
to TiddlyWiki
Hi Flibbles,

No significant changes to saving no node.js. Currently tiddlers of type text/vnd.tiddlywiki are saved as .tid, files unless they have mutliline fields in which case those tiddlers are saved as .json files. This will remain the same with the addendum that a tiddler with field names with a colon in them will also be saved as a .json file. 

Saq

Flibbles

unread,
Jul 24, 2021, 4:39:27 PM7/24/21
to TiddlyWiki
Thank you. I am much relieved. I would not have looked forward to having every file in my git repository touched, and to have all of the tiddlers' text bodies crammed into one line.

Mohammad Rahmani

unread,
Aug 1, 2021, 2:41:52 AM8/1/21
to tiddl...@googlegroups.com
I am testing Tiddlywiki 5.2.0 p with different plugins I use.

Today I noticed Uglify (Uglify — Plugin compression made easy (flibbles.github.io)) which is used to compress JS and make TW a smaller html works in reverse order!
I mean using Uglify makes TW 5.2.0 single html files with basic plugins (codemirror, highlight.js, katex,...) get larger in size!
Strangely I do the unglifing several times, each time I get a bigger file! This is not the case with TW 5.1.23

@flibbles would you please have a look!



Best wishes
Mohammad


On Wed, Jul 14, 2021 at 4:05 PM Jeremy Ruston <jeremy...@gmail.com> wrote:
It doesn’t sound very exciting, but the new JSON-based store area has now been merged into the prerelease.


This is the big change that has prompted us moving from v5.1.23 to v5.2.0. It brings one major benefit to all users of TiddlyWIki: the characters used in field names are now unrestricted, making it possible to use fieldnames like "My^Field” or “☃️”.

While that is an important change that we hope will lead to the development of interesting new techniques for working with data in TiddlyWiki, the biggest risk of this change is that it is not fully backwards compatible.

That is to say that any tools that work directly with TiddlyWiki files will no longer work as expected, and need updating to understand the new format.

We have tried to anticipate the problems that this change might cause, for example working with Simon Baird to fix TiddlyHost ready for the change. You can follow the technical discussions here:


But now we need everyone’s help to try the new version, to help us identify any incompatibilities before the final release.

Please try the prerelease with as many different saving mechanisms and server backends as possible. You can report any problems here, or by opening a new ticket on GitHub.

For me personally this feels like a big milestone. TW5 has used the same DIV-based tiddler store area since the very beginning in 2011, having directly inherited it from TiddlyWiki Classic. I reused the old format to give TW5 compatibility with the ecosystem of tools around TiddlyWiki Classic. Now that the TW5 ecosystem is so much bigger, it makes sense to move to a store area format that has fewer restrictions and is easier for developers to work with.

Just for fun, I wrote a Node.js one-liner to extract the tiddlers from a TiddlyWiki HTML file. In contrast, a few hundred lines were needed to read the old format:

var fs=require("fs"),tiddlers=JSON.parse(fs.readFileSync("./editions/prerelease/output/index.html").toString().match(/<script class="tiddlywiki-tiddler-store" type="application\/json">([^<]*)<\/script>/)[1]);console.log(tiddlers)

Questions and comments welcome,

Best wishes

Jeremy

--

Saq Imtiaz

unread,
Aug 1, 2021, 2:54:40 AM8/1/21
to TiddlyWiki
@Mohammad I recommend posting this as an issue at https://github.com/flibbles/tw5-uglify

Mohammad Rahmani

unread,
Aug 1, 2021, 5:27:11 AM8/1/21
to tiddl...@googlegroups.com
Hi Saq,
 Sure I do!


Best wishes
Mohammad


To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/65c754f0-582d-4258-8ead-986f109a2b25n%40googlegroups.com.

Flibbles

unread,
Aug 1, 2021, 11:02:37 AM8/1/21
to TiddlyWiki
I'll look into it as soon as I have some free time.


-Flibbles
Reply all
Reply to author
Forward
0 new messages