[TW5] Accessing JSON tiddler data

[Joshua Fontany]

Hi all you tiddlywiki dev folks,

I am making progress with my RenderTable widget, but have run into a situation where I need to add or remove items from arrays listed in JSON tiddlers. This is so I can add/remove "table rows" when the table is being fed by "application/json" tiddlers. I have a solution sketched out for using javascript's array.splice() method, but had a question before I start.

Why can we currently access only the "root level" elements of a JSON tiddler?

Is this a performance or other technical limitation or has a solution for this just not been created yet?

Currently, I "flatten" my JSON tiddlers with another plugin I made in order to simulate storing data as nested objects & arrays. I can then access the data I want with a "path" key/index. If this "flattening" is unnecessary, I'd love to be able to work on allowing regular "nested" JSONs to be used within tiddlywiki. Otherwise, I'll have to work around the limitation.

Thank you all for any insight you can provide while I go and read through the core tiddlers that reference JSON stuff. :D

Best,
Joshua Fontany

[Evan Balster]

I have to assume that it's just a limitation in how the JSON structure is translated to and from text references.  JSON doesn't actually have an "official" addressing scheme for sub-objects, only add-on standards like JSON pointer.  TiddlyWiki allows most characters (including dots) in field names.

Relevant functions in the TiddlyWiki core code:  getTextReference --> extractTiddlerDataItem --> getTiddlerDataCached --> getTiddlerData

Looks like internally, the whole JSON structure is parsed, but extractTiddlerDataItem has no rule to look for a delimiter (like ".") and access sub-objects.  This bit of code in that function would need to be expanded into something more complicated:

	if(data && $tw.utils.hop(data,index)) {
		text = data[index];
	}

...So at a glance this looks like it would be a pretty easy "mod" to implement, if you're willing to mess with some core code.

[Joshua Fontany]

That is also the direction I was leaning (text-references and the getindex[] filter).

Mahalo (thanks) Evan! Definitely gives me a good place to start. Now to think about path syntax....

Best,
Joshua Fontany

[Evan Balster]

I strongly recommend riding the coat-tails of an existing convention for something like this.  Specifically, JSON Pointer (RFC 6901) should fit the bill.  It uses a slash delimiter.  Here is a widely-used JavaScript library providing some utility functions for them.

[Joshua Fontany]

Oooh. Thanks for that. I have also been considering JsonPath syntax (http://goessner.net/articles/JsonPath/) and this library:

https://www.npmjs.com/package/jsonpath

I will have to compare them.

Best,
Joshua Fontany

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywikidev/Cs-lykcn_JE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/3b3591f8-fe52-4d64-be21-6d347c2b7710%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Evan Balster]

For my part, I suspect that the use of brackets in JSON path and company would prevent the syntax from being usable in filter expressions.  The filter-parser will interpret the first right-bracket as closing the operand.

To unsubscribe from this group and all its topics, send an email to tiddlywikide...@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

[Joshua Fontany]

Yup, that is exactly the rabbit hole that led me back to examining the core json parser. I like the pointer syntax much better, it will also simplify pushing json key:value pairs to and from tiddlers as field:value pairs. I will only have to replace the '/' char with a legal field-name char like period. Now to see what I can implement in the core and refactor my JsonMangler and RenderTable plugin. ;) Thanks again!

Best,
-Joshua Fontany

[Joshua Fontany]

I have opened an issue on the TW5 github page to discuss this.

https://github.com/Jermolene/TiddlyWiki5/issues/3074

Thanks again for the advice so far Evan.

Best,
Joshua Fontany

[Joshua Fontany]

Coming back around to this after a bit.

I think the best way to preserve backwards compatibility is to setup a new text-reference field-name prefix.

!! = tiddler field-name reference

## = data tiddler index-name reference

What would a good doubled-char prefix be for 'nested json data-tiddler path-name'?

Is @@ reserved for anything? What would be another good one? 

@@
||

>>

The idea would be to have a parallel method to invoke parsing through the nested-json style structures and returning a data (or a stringified piece of text for data that includes further nesting) with the new prefix. Then alter the ## flat-json data tiddler function have a fallback to return a stringified piece of text, if the top-level index referenced provided data with nested sub-objects/arrays/etc.

Best,
Joshua F

[coda coder]

Is there some reason ## can't used again? And how about strict "[[...]]" and '[[...]]' for arrays?

{
  a: {
    b: {
      c: {
        x: 5,
        y: "six",
        z: "[[7, "eight"]
      }
    }
  }
}

-> "a##b##c##x",  'a##b##c##z"[[0]]"' etc.

[Joshua Fontany]

Per the thread I opened on github, we need to keep it compatible with existing wikis that may use those types of "path strings" as top level index names. Having the code automatically try both the top-level string-index _and_ the nested object index and then somehowd3cide which one to use in each case was getting overcomplex. And that's just reading data not writing to index/path addresses. Simplest solution is to have a parallel "path" operator for text references so it falls on the user to decide which approach would be best for any given data-toddler (with a fallback to stringify nested objects if ## is used on a nested data-tiddler). Currently, nesting just breaks ## completely.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywikidev/Cs-lykcn_JE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to tiddlywikidev+unsubscribe@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/e5312fa6-f8a5-4bcb-9b24-7797dd059908%40googlegroups.com.