JSON - why?

[Mat]

What is the point with JSON tiddlers over regular tiddlers? What do they enable and what limitations are there?

One point that I do get is that other software often has the possibility to export/import JSON so it could enable data transfer with TW.

There does not seem to be an advantage when it comes to data tiddlers because the JSON data tiddlers can only be on the JSON root level whereas a regular tiddler is deeper.

One reason why I'm asking is because I hope to build a UI for Jeds FederationCore plugin that extracts data from the so called tiddler "bundle" which is the format that fetched tiddlers come in. Bundles can be packed into a special bundle format or into JSON. 

Thank you!

<:-)

[TonyM]

Mat,

Some Quick facts as an explanation, and for anyone passing Why JSON?, some ways we use them

• JSON tiddlers can contain JSON data as per JSON standards. 
• JSON is an alternative way to store additional data in a single tiddler like when it is used as a dtatiddler
• In TiddlyWiki a particular use of JSON is to pack one or more tiddlers into a single tiddler by converting them to a JSON format known by Tiddlywiki. 
• Export multiple tiddlers to JSON will create one of these JSONs containing multiple tiddlers.
• Dragging such a JSON file onto a Wiki will trigger import and you can selectively import tiddlers from within that "bundle".
• Dragging and dropping tiddlers on to a TiddlyWiki actually creates a temporary JSON tiddler full of the tiddlers the advantage being the tiddlers are now in your wiki but remain contained within the temporary JSON file until you finish importing them.

I can add more but I need to reboot my computer...

[Mohammad]

Without JSON, I could not develop Trashbin plugin, and Todolist plugin!

JSON tiddlers let you create more robust databases comparing to ordinary tiddlers

JSON entries are somehow hidden from many operations while tiddler fields are not!

In summary JSON and Data Tiddler are great features in TW

[Mat]

Thanks guys. Yes, I did miss to mention the point about enclosing several tiddlers in one json tid.

To what extent is JSON content sandboxed? I.e can harmful tiddlers that are packaged up as JSON still be harmful when in the JSON format?

<:-)

[PMario]

On Wednesday, July 24, 2019 at 12:44:47 PM UTC+2, Mat wrote:

...

To what extent is JSON content sandboxed?

JSON (JavaScript Object Notation) is just a data format. .. So as long as you don't execute, executable content, nothing happens. ... BUT it's not a sandbox.

 

I.e can harmful tiddlers that are packaged up as JSON still be harmful when in the JSON format?

If a JSON tiddler contains the right structure _and_ type fields. eg: "type": "application/javascript", Tiddlywiki will interpret it as a plugin. If you import the content, save and reload the page it will be activated. ... Then it can do harm!

For TW the same rules apply as for any other content from the web. Don't install stuff from sources, you don't trust.

This is really important, since with tiddlywiki, we don't have any signing mechanism in place, where a user could verify, if a plugin source has been modified. ...

-m

[Mat]

Thanks!

BTW, in the docs I find the jsontiddler macro and jsontiddlers macro. Is there any directly usable macro or widget to do the reverse, i.e to convert from json into regular tiddler format? The native import mechanism does allow previews of json tiddlers.

<:-)

[@TiddlyTweeter]

Ciao Mat

For import/export JSON is extremely good. Very reliable. One tool I use all the time is PMario's bundler. Its the most flexible import/export tool TW has.

As an internal live data structure in TW, to get the best from it needs work. Otherwise it just acts as a flat-file database. Which has no particular exciting benefit over other methods.

This is something Joshua Fontany is trying to address with his JSON editor https://joshuafontany.github.io/TW5-jsoneditor/ to better leverage its power.

TT

[Jeremy Ruston]

Hi Mat

Some history that may help… data tiddlers were added quite close to the start of TW5. They were part of the work to implement colour palettes; I was concerned about the proliferation of tiddlers if each system colour were to be given its own tiddler.

https://github.com/Jermolene/TiddlyWiki5/commit/baff9016858133d300a9662ffd1782568454f8eb

My initial intention was to provide support for generic JSON tiddlers, with syntax to address individual items within such a tiddler. Data tiddlers would have been one of a number of alternative representations for specific schemas. Joshua is now exploring ideas along those lines.

But my gradual conclusions were that:

* Adding the “index” mechanism for addressing items within a data tiddler introduced a lot of complexity right across the code base that is still there today

* Figuring out an addressing mechanism for items within a JSON object would end up re-inventing something as complex as JSONPath

* Proliferation of tiddlers isn’t actually a problem for performance with the core. The problem is more cognitive; too many random tiddlers and most of our lists become useless. There are lots of ways we can address that issue — for things like palettes I would we might pack the individual tiddlers into a plugin

We do of course also use JSON as a container format for tiddler files, but that is handled by the import mechanism.

Best wishes

Jereym

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/c8c24a82-a99e-402b-8b79-e6ff02ff8184%40googlegroups.com.

[TonyM]

Jeremy,

I appreciate the continued focus on leveraging tiddlers as the basic store but we need to extend their handling in some ways if we want them to also represent records in the equivalent of database tables. Here I am thinking of hiding the "data record" tiddlers from standard searches without prefixing them with $:/ and somehow enabling the same key (tiddlername) to be used to address different records in different logical tables/or namespaces (see below).

To me however allowing the designer to to make use of fully featured "data tiddlers" adds great strength to tiddlywikis ability to respond to all possible data structures someone may wish to represent in tiddlywiki. Enabling JSON to the fullest extent also increases TiddlyWikis ability to interact with external data sources as many sources such as xml can be converted to JSON if not yet available. We do not need to code for all possible data sources if we comprehensively adopt a robust standard like JSON because we can leverage external tools written for the conversion of data between standards and maintained by others.

Joshua's tools are extensive and I hope they may improve such handling in the future.

Given the maxim that if you place a problem on the table you should place a possible solution there as well;

• If we could introduce additional name spaces which are not in the tiddler name, such that when one of these is set the <<currentNamespace>> can refer to the tiddlers there in with <<currentTiddler>> then a database table could be created in its own namespace, with identical tiddler names to those in other namespaces. A names space can be used as a table name.
• Of course this would be a challenging thing to implement but its benefits would be extensive and would allow further use of tiddlers as the atom of data. Careful design would make the use of additional namespaces invisible until a user/designer starts to use additional ones.
• To be able to filter tiddlers in one namespace and use this as a key to tiddlers in another namespace would be needed.
• This can be done with prefixes, remove prefix and lookup operator but the logic is far more complex and harder to follow or design. 

Regards

Tony

To unsubscribe from this group and stop receiving emails from it, send an email to tiddly...@googlegroups.com.

[VikingMage]

I am still quite new to tiddlywiki, but I think what you are talking about can be achieved by using a json tiddler as the namespace and then having virtual tiddlers stored within the json tiddler.

you could then have a macro to get a list of tiddlers within that "namespace" tiddler. 

I imagin it would work like a plugin, but be kept out of the plugin list.

[TonyM]

VikingMage,

That is one approach that many of us currently take, which could be made easier, with "tiddler to json" and "json to tiddler" and "query tools" made available. 

These features are already in the core however not hackable or exposed as much as they could.

For example;

• Drop one or more tiddlers or json files on a wiki
• Edit the resulting $:/Import tiddler, this tiddler contains a json representation of the tiddlers already.
• The Import process "extracts" them to actual tiddlers.

Actually I think I will modify the import process to save imports to a JSON tiddler, or act on an existing JSON tiddler.

• however the current methods treat JSON as only a tiddler storage. It limits what content the JSON has to tiddlers formats. 

Joshua's JSON tools open this up, and whilst we need not go the whole way with what is possible with Joshua's tools, we could go part way in the standard distribution.

Regards

Tony

[TonyM]

I have found a way to modify the import process to save imports to a JSON tiddler, or act on an existing JSON tiddler.

It is 90% Done with only this issue and one other to resolve

• https://groups.google.com/forum/?hl=en#!topic/tiddlywikidev/5jVJNq13mXk

Regards

Tony

[Mat]

Thank you everyone for clarification and for historical background. Very interesting stuff.

<:-)

[Jeremy Ruston]

Hi Tony

I appreciate the continued focus on leveraging tiddlers as the basic store but we need to extend their handling in some ways if we want them to also represent records in the equivalent of database tables. Here I am thinking of hiding the "data record" tiddlers from standard searches without prefixing them with $:/

The way to do that is probably to make a custom search results tab that hides the tiddlers you want to ignore/

and somehow enabling the same key (tiddlername) to be used to address different records in different logical tables/or namespaces (see below).

The title is a unique key. If you want to address tiddlers by a non-unique key then you’d use a field.

To me however allowing the designer to to make use of fully featured "data tiddlers" adds great strength to tiddlywikis ability to respond to all possible data structures someone may wish to represent in tiddlywiki. Enabling JSON to the fullest extent also increases TiddlyWikis ability to interact with external data sources as many sources such as xml can be converted to JSON if not yet available. We do not need to code for all possible data sources if we comprehensively adopt a robust standard like JSON because we can leverage external tools written for the conversion of data between standards and maintained by others.

Are you saying that you’d like TW5 to have better interop with existing JSON files and APIs? I think Joshua is exploring the approach of extending the existing data tiddler mechanism. In some of my own work I have explored a different approach, “exploding” JSON documents into multiple tiddlers, where each tiddler represents a node of the document. It’s one of many incomplete plugins that I must get around to releasing.

Joshua's tools are extensive and I hope they may improve such handling in the future.

Given the maxim that if you place a problem on the table you should place a possible solution there as well;

• If we could introduce additional name spaces which are not in the tiddler name, such that when one of these is set the <<currentNamespace>> can refer to the tiddlers there in with <<currentTiddler>> then a database table could be created in its own namespace, with identical tiddler names to those in other namespaces. A names space can be used as a table name.

I’m not sure I’m following your exact meaning, but it sounds like you want TW5 to behave more like a database.

• Of course this would be a challenging thing to implement but its benefits would be extensive and would allow further use of tiddlers as the atom of data. Careful design would make the use of additional namespaces invisible until a user/designer starts to use additional ones.

I’m not sure what you mean by namespaces, but some of what you’re describing sounds like the plugin mechanism.

• To be able to filter tiddlers in one namespace and use this as a key to tiddlers in another namespace would be needed.

Maybe you’re talking about something like TiddlyWeb’s “bag” model of storage? We have considered implementing that model within TW5 itself, whereby the store would be a list of independent bags.

• This can be done with prefixes, remove prefix and lookup operator but the logic is far more complex and harder to follow or design. 

Core changes are expensive, even if only in opportunity cost. If something can be done without needing core changes then it’s hard to justify prioritising those changes, especially if the proposal is in such an early state.

In this particular case, it does sound like prefixes are the way to go. It’s a standard technique used endlessly by the core, and of course it’s the natural to slice up a single namespace.

Best wishes

Jeremy.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/f54ebf05-1eb0-4b9b-ac46-9e8230c196c1%40googlegroups.com.

[Joshua Fontany]

It took me a while, but I patched up Evan's Modloader plugin to work with 5.1.20-pre, so I can patch in the handful of function calls I need into the core. These then call my JSON utilities library functions. If it were easier to replace the built in ones in the core with user-added ones, that'd be neat.

As is, I've gone and updated Evan's Formula plugins & am very lose to having my JSON plugins running on 5.1.20-pre (see other announcements).

So, I'm comfortable saying we'll soon have the tools to start exploring this space.

Best,
Joshua Fontany

Hi Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/f54ebf05-1eb0-4b9b-ac46-9e8230c196c1%40googlegroups.com.

[TonyM]

Josua,

I thank you very much, on your formulas announcement.

On JSON recently came up with a method to allow the user to bypass a json tiddler containing tiddlers, of a set of files dropped on the wiki, and save it as a single JSON tiddler. Then on opening the json tiddler you can extract the tiddlers as desired, perhaps to overwrite one you were experimenting with. Let me know if you want more details.

I have a little barrier that the tm-perform-import message always wants to replace the json with an import report. I could work around this but as discussed above perhaps there is a way that can improve hackability and solve this.

Regards

Tony

[TonyM]

Jeremy;

The title is a unique key. If you want to address tiddlers by a non-unique key then you’d use a field.

I can see how we could use a tiddler name as a tablename, and within that use fieldnames as unique keys, and the value without polluting the tiddlername space.

However to do this effectively It would be nice if we could hide the fields defined in such tiddlers such that they do not pollute the field name list when clicking on the drop down.

This could replace the datatiddler model but faces the same problem if you want each record to contain multiple columns.

I can also see now how we could have $:/tablename/records/tiddlerkeyvalue when this tiddlers fields are the columns in a record/row called "tiddlerkeyvalue". The lookup operator helps with this but perhaps it could be extended to permit the provision of a fieldname so we can retrieve more than the text field, effectively additional columns.

Are you saying that you’d like TW5 to have better interop with existing JSON files and APIs? I think Joshua is exploring the approach of extending the existing data tiddler mechanism. In some of my own work I have explored a different approach, “exploding” JSON documents into multiple tiddlers, where each tiddler represents a node of the document. It’s one of many incomplete plugins that I must get around to releasing.

I expect my own work and Joshua's will achieve what I am looking for here. You suggestion would be interesting if applicable to other JSON data (rather than just tiddlers) as I kind of have the tiddler handling (See recent post of mine here).

I’m not sure I’m following your exact meaning, but it sounds like you want TW5 to behave more like a database.

 

Yes and no, I just want to insure there is not data structure that tiddlywiki can not represent. But of course it is already in many ways a database.

I’m not sure what you mean by namespaces, but some of what you’re describing sounds like the plugin mechanism.

Perhaps if I understood that more? Now I see the operation of Import more closely. Namespaces are like just as $:/ is treated one way the possibility of having other tiddler prefixes with other qualities. For example tablename, or even username. Perhaps where they can be searched for by their caption.

Maybe you’re talking about something like TiddlyWeb’s “bag” model of storage? We have considered implementing that model within TW5 itself, whereby the store would be a list of independent bags.

Maybe I do not know enough about bags. Some discussions about bags scared me off tw5 for a while in the early days. Fortunately there is not too much conversation about this today :)

I have no desire to influence the core or standard distribution unless there are well demonstrated reasons. 

As suggested above perhaps the extent of this is;

• The lookup operator extended to permit the provision of a field name
• A way to use WidgetMessage: tm-import-tiddlers such that it does not destroy the import tiddler (or any JSON file with tiddlers)

 Regards

Tony