[tw5] drag n drop plugins and tiddlyweb/tank

[chris...@gmail.com]

Someone is experimenting with importing plugins into a TW5 hosted on
Tank[1] and having some difficulties. I've tried to replicate their
experience and am also having difficulties (but I'm not sure they
are the same).

There are at least two different issues:

1. When dragging a plugin[2] from its origin into the tank-hosted
tw5 there is a 409 conflict trying to save the tiddler. This is
because it is trying to save to a bag that does not exist:
"default". The tiddler appears to be assigned this bag when it is
dropped (or maybe it gets it from the origin?).

If I change the bag field the tiddler does save:

https://tank.peermore.com/bags/cjdtw5/tiddlers/%24%3A%2Fplugins%2Fskeeve%2Flistselect.txt

2. However, since the tiddler has a type of application/json,
getting it as JSON results in JSON which is _not_ the expected
structure of JSON tiddler, it is the object structure with a
single key of "tiddlers" and sub object of a few tiddlers:

https://tank.peermore.com/bags/cjdtw5/tiddlers/%24%3A%2Fplugins%2Fskeeve%2Flistselect.json

This means that the adaptor mechanism ends up with a tiddler
which has "undefined" text and other fields.

Presumably a bit more manipulation is required in a couple different
places in the stack.

I'm not posting this as an issue as it is not clear what part of the
stack should be held responsible.


[1] https://tank.peermore.com/

[2] The sample plugin is
http://tiddlystuff.tiddlyspot.com/#%24%3A%2Fplugins%2Fskeeve%2Flistselect
--
Chris Dent http://burningchrome.com/
[...]

[Jeremy Ruston]

Hi Chris

> This is because it is trying to save to a bag that does not exist: "default". The tiddler appears to be assigned this bag when it is dropped (or maybe it gets it from the origin?)

Ah, yes, sounds like the plugin is coming in with a "bag" field, and TW5 is blindly accepting it. I guess we should remove the "bag" field on import so that the same processing would occur as with a manually newly created tiddler?

> getting it as JSON results in JSON which is _not_ the expected structure of JSON tiddler, it is the object structure with a single key of "tiddlers" and sub object of a few tiddlers

The JSON structure you see is the correct format for the body of a TW5 plugin (which is just a bunch of tiddlers packed into one as a blob of JSON).

If a tiddler has the type "application/json" does TiddlyWeb serve the text field directly as the JSON representation, rather than serialising the tiddler as normal? That's a bit unexpected if so. Is there a way I can retrieve the ordinary serialized JSON representation?

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[chris...@gmail.com]

On Mon, 7 Apr 2014, Jeremy Ruston wrote:

>> This is because it is trying to save to a bag that does not exist: "default".
>> The tiddler appears to be assigned this bag when it is dropped (or maybe it
>> gets it from the origin?)
>
> Ah, yes, sounds like the plugin is coming in with a "bag" field, and TW5 is
> blindly accepting it. I guess we should remove the "bag" field on import so
> that the same processing would occur as with a manually newly created
> tiddler?

Yes, I just tried simply removing the contents of the bag field
rather than setting it to an appropriate bag and the right thing
happened. So I would guess killing that field on drag would be the
right way to go.

> If a tiddler has the type "application/json" does TiddlyWeb serve the text
> field directly as the JSON representation, rather than serialising the
> tiddler as normal? That's a bit unexpected if so. Is there a way I can
> retrieve the ordinary serialized JSON representation?

The results that are happening is because of two things:

The tiddler is being PUT with content-type: application/json

AND

The JSON object has a type field set to 'application/json'.

The upshot of this is that the tiddler is being accepted as a
pseudo-binary tiddler. When you GET the tiddler with either a
default Accept header OR with an Accept header that matches its
type you will get it in its raw form (the contents of the text
field) with a sent content-type field of the stored 'type' field.

This has a side effect of making text/plain, text/html and
application/json have "special" meaning when accessing a tiddler.

To get the tiddler back as a tiddler in JSON form it needs to put
with a 'type' field that is not application/json.

[Jeremy Ruston]

Hi Chris

Yes, I just tried simply removing the contents of the bag field

rather than setting it to an appropriate bag and the right thing
happened. So I would guess killing that field on drag would be the
right way to go.

Great, I'll fix that for 5.0.9.

 

If a tiddler has the type "application/json" does TiddlyWeb serve the text
field directly as the JSON representation, rather than serialising the
tiddler as normal? That's a bit unexpected if so. Is there a way I can
retrieve the ordinary serialized JSON representation?

The results that are happening is because of two things:

The tiddler is being PUT with content-type: application/json

   AND

The JSON object has a type field set to 'application/json'.

The upshot of this is that the tiddler is being accepted as a
pseudo-binary tiddler. When you GET the tiddler with either a
default Accept header OR with an Accept header that matches its
type you will get it in its raw form (the contents of the text
field) with a sent content-type field of the stored 'type' field.

This has a side effect of making text/plain, text/html and
application/json have "special" meaning when accessing a tiddler.

To get the tiddler back as a tiddler in JSON form it needs to put
with a 'type' field that is not application/json.

Ouch. This is tricky.

The tiddler clearly contains JSON data, and so I'd struggle to find a rationale for making the type field be anything other than "application/json".

Wouldn't any code out there that reads JSON tiddlers expecting the usual tiddler JSON structure going to get confused by this behaviour if they ever try to read a tiddler that actually contains JSON?

Anyhow, no doubt it's too late to change. So, I guess the question is whether there's any way for a client to read both the body and fields of a JSON tiddler in one HTTP GET operation? Perhaps I could request it in XML or something? (ugh!) Or maybe I should just use .tid format? Then I'd have to special case JSON tiddlers I guess.

Best wishes

Jeremy.

 

--
Chris Dent                                   http://burningchrome.com/
                                [...]

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

On Mon, 7 Apr 2014, Jeremy Ruston wrote:

>> To get the tiddler back as a tiddler in JSON form it needs to put
>> with a 'type' field that is not application/json.
>
> Ouch. This is tricky.

Is it too late for a single tiddler that contains tiddlers to have a
type of its own like application/tiddlywiki+json? If you then PUT as
application/json with a type of application/tiddlywiki+json you
should be able to get the right thing back.

> The tiddler clearly contains JSON data, and so I'd struggle to find a
> rationale for making the type field be anything other than
> "application/json".

Text tiddlers clearly contain plain text but we call them
text/x-tiddlywiki and text/vnd.tiddlywiki frequently?

> Wouldn't any code out there that reads JSON tiddlers expecting the usual
> tiddler JSON structure going to get confused by this behaviour if they ever
> try to read a tiddler that actually contains JSON?

Apparently this case hasn't come up in the past. The standard way to
store structured data has been to use the existing attributes and
fields _on_ a tiddler, not put structured data _in_ the text field
of the tiddler.

Obviously the aggregate tiddlers-in-a-single-tiddler case doesn't
map well to that. From the server's standpoint this would make the
most sense: what we have here are tiddlers and thats it, none of
these "weird" things that are actually containers of tiddlers. This
buys some granularity of reuse too, in case that's of interest.

Would it be possible to explode the single tiddler to its constituent
tiddlers before PUTting to the server and recompose them on GET
(perhaps by way of transclusion)?

[PMario]

On Monday, April 7, 2014 8:10:10 PM UTC+2, Chris Dent wrote:

>
> Ah, yes, sounds like the plugin is coming in with a "bag" field, and TW5 is
> blindly accepting it. I guess we should remove the "bag" field on import so
> that the same processing would occur as with a manually newly created
> tiddler?

Yes, I just tried simply removing the contents of the bag field
rather than setting it to an appropriate bag and the right thing
happened. So I would guess killing that field on drag would be the
right way to go.

As long, as there is some feedback, that part of the tiddler content has been removed, I think I'm fine with removing the bag field from an imported tiddler.

-m

[Jeremy Ruston]

Hi Chris

Is it too late for a single tiddler that contains tiddlers to have a
type of its own like application/tiddlywiki+json? If you then PUT as
application/json with a type of application/tiddlywiki+json you
should be able to get the right thing back.

It's not too late to change the plugin content type, no. There's a bunch of JSON tiddlers that TW5 uses, though, not just plugins. We'd have to create an application/vnd.tiddlywiki-blah+json type for each one, I guess.

More to the point, there is support for users creating and using JSON tiddlers.

 

Text tiddlers clearly contain plain text but we call them
text/x-tiddlywiki and text/vnd.tiddlywiki frequently?

Good point. But we use application/javascript for javascript, regardless of what role that JavaScript plays.

 

Apparently this case hasn't come up in the past. The standard way to
store structured data has been to use the existing attributes and
fields _on_ a tiddler, not put structured data _in_ the text field
of the tiddler.

Right. But one wouldn't expect that a consequence of deciding to try putting structured data in a tiddler would be to change the retrieval behaviour (I would expect it to change the rendering behaviour, but I think that's different).

 

Obviously the aggregate tiddlers-in-a-single-tiddler case doesn't
map well to that. From the server's standpoint this would make the
most sense: what we have here are tiddlers and thats it, none of
these "weird" things that are actually containers of tiddlers. This
buys some granularity of reuse too, in case that's of interest.

Would it be possible to explode the single tiddler to its constituent
tiddlers before PUTting to the server and recompose them on GET
(perhaps by way of transclusion)?

We could perhaps do it for plugins (at a ridiculous cost; we've got plugins with hundreds of tiddlers in them). But it wouldn't solve the problems for the other cases of JSON being stored in tiddlers.

All I need is a reliable way to read the text and fields of a tiddler, regardless of its content type.

Best wishes

Jeremy.

 

--
Chris Dent                                   http://burningchrome.com/
                                [...]

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

I'm not sure how to resolve this, some comments within.

On Tue, 8 Apr 2014, Jeremy Ruston wrote:

> More to the point, there is support for users creating and using JSON
> tiddlers.

I think part of the problem here is the the term "tiddler" has become
quite diverse in meaning, especially when it gets mixed between the
abstract idea of a tiddler and representations of that idea.

The clearest way to see the issue here is:

* PUT a tiddler with content-type application/json, with type of
application/json and text which _is_ JSON:

curl -X PUT \
-H "Content-Type: application/json"
http://0.0.0.0:8080/bags/common/tiddlers/f1 \
--data-binary '{"type": "application/json", "text": "{\"text\": \"foo\"}"}'

* GET that tiddler as application/json:

curl -v \
-H "Accept: application/json"
http://0.0.0.0:8080/bags/common/tiddlers/f1

The output will be:

{"text": "foo"}

Then compare with:

* PUT a tiddler with content-type application/json, with no type and
text which _is_ JSON:

curl -X PUT \
-H "Content-Type: application/json" \
http://0.0.0.0:8080/bags/common/tiddlers/f2 \
--data-binary '{"text": "{\"text\": \"foo\"}"}'

* GET that tiddler as application/json:

curl -v \
-H "Accept: application/json" \
http://0.0.0.0:8080/bags/common/tiddlers/f2

The output will be:

{"creator": "GUEST", "fields": {}, "created": "20140408125125",
"recipe": null, "modified": "20140408125125", "uri":
"http://0.0.0.0:8080/bags/common/tiddlers/f2", "bag": "common",
"text": "{\"text\": \"foo\"}", "title": "f2", "permissions":
["read", "write", "create", "delete"], "modifier": "GUEST", "type":
null, "tags": [], "revision": 1}

This mechanism is true of any media type pairing. If you PUT a JSON
tiddler with type 'application/javascript' and GET it as
'application/javascript' you will get back just the javascript.

curl -X PUT \
-H "Content-Type: application/json" \
http://0.0.0.0:8080/bags/common/tiddlers/f3 \
--data-binary '{"type": "application/javascript", "text": "alert(\"hi\")"}'

curl -H "Accept: application/javascript" \
http://0.0.0.0:8080/bags/common/tiddlers/f3

alert("hi")

From the semantics of the HTTP API this makes sense.

> Right. But one wouldn't expect that a consequence of deciding to try
> putting structured data in a tiddler would be to change the retrieval
> behaviour (I would expect it to change the rendering behaviour, but I think
> that's different).

One might not expect it, but it is how it has worked for several
years and is aligned with the semantics of the API with regard to
tiddlers that want to "override" the value of the text field[1].

A quick fix is to set a different type field (not application/json) on
the JSON generated before being PUT to the server (with content-type
application/json) and then reverse that mapping when the tiddler is
being processed.

> We could perhaps do it for plugins (at a ridiculous cost; we've got plugins
> with hundreds of tiddlers in them). But it wouldn't solve the problems for
> the other cases of JSON being stored in tiddlers.

No, indeed that doesn't sound great.

> All I need is a reliable way to read the text and fields of a tiddler,
> regardless of its content type.

I think the best thing is going to be to set the 'type' field to
something different. There's already the "unholy freaking of content
types" in tiddlywebadaptor.js so there's some precedent.

It's not ideal but is probably workable and keeps the (I think
important) distinction between the content-type of the representation
being PUT and the media type of the content.

Is that workable?

[1] I guess the underlying cause of issues here is how the "text"
attribute is being used across the various representations and
contexts. TiddlyWeb optimizes for a short code path and a flexible
default web representation for single tiddlers that want to declare
arbitrary content types. This is okay for single web requests of the
"content" but comes at the cost of lost metadata on some of those
requests and weirdness when dealing with "content" which is JSON.

[chris...@gmail.com]

On Tue, 8 Apr 2014, chris...@gmail.com wrote:

>> All I need is a reliable way to read the text and fields of a tiddler,
>> regardless of its content type.
>
> I think the best thing is going to be to set the 'type' field to
> something different. There's already the "unholy freaking of content
> types" in tiddlywebadaptor.js so there's some precedent.
>
> It's not ideal but is probably workable and keeps the (I think
> important) distinction between the content-type of the representation
> being PUT and the media type of the content.
>
> Is that workable?

I spent quite a while going through several scenarios related to
this topic with FND. The result of that were two goals that though
related are not mutually dependent.

One is that, long term, Jeremy is correct: There needs to be a
"reliable way to read the text and fields of a tiddler" even if the
"content" of the tiddler is application/json. At the moment the way
to do that for every single type, except for application/json, is to
get the tiddler as application/json.

Two is that, despite the above, it makes semantic sense that a
tiddler which has as content JSON but is expected to be manipulated
as a tiddler (that is the text attribute of a tiddler is JSON) should
have a content type that is distinct from straight
"application/json".

Doing both of these things preserves the some unique aspects of both
TiddlyWeb and TW5: TiddlyWeb is not merely a data store for
tiddlers, it is also a content delivery system. TW5, in one of its
forms, is a container and manipulator of tiddlers, including some which
contain content that is not wikitext.

These things suggest the following actions:

* TW5 institutes a media-type for tiddlers which contain JSON in the
text field. Because of that text field use I wonder if something
like "text/vnd.tiddler+json" is in the vicinity of right? Or maybe
application. Not sure and not sure it really matters other than
that it is custom in some form.

When a tiddler is being PUT it is PUT with either the "old"
content-type application/json or the below new type and a type
attribute of the above.

* TiddlyWeb extends itself to treat "application/x-tiddler+json" in
much the same way it treats "application/json" now, deprecating
but continuing to support the latter.

The caveat here is that if some bright bulb chooses to PUT a tiddler
with content-type 'application/x-tiddler+json' and type attribute
of the same they are not going to get the results they expect.

I think these changes will help maintain the important distinction
between the content type being transported and the content type
being stored or contained while still allowing "easy" binary and
pseudo-binary tiddlers.

Thoughts?

[Jeremy Ruston]

Hi Chris

> Two is that, despite the above, it makes semantic sense that a

> tiddler which has as content JSON but is expected to be manipulated
> as a tiddler (that is the text attribute of a tiddler is JSON) should
> have a content type that is distinct from straight
> "application/json".

This is the bit that I still struggle with. There's still entirely special behaviour for application/json tiddlers. From the point of view of TiddlyWiki these tiddlers really are application/json, and not some special TW variant implied by the special content type. When the user drags in a .json file from the desktop they get an application/json tiddler automatically. End users are going to be confused if dragging text/html content gives them text/html tiddlers, but dragging application/json content gives them some different type.

> TiddlyWeb is not merely a data store for tiddlers, it is also a content delivery system

Yes I was thinking the same thing in relation to the work that Jon Udell has been doing making tiddlywebadaptor.js talk to PouchDB: the representation transformation aspects of TiddlyWeb can be seen as a layer on top of the data store layer.

The trouble is that I wouldn't expect the existence of a representation transformation layer to break the basic data store layer protocol of "thou shall GET what thou PUT".

The real issue for TW5 is that I have been trying to keep the serverside compatible with TiddlyWeb, but I can't really come up with a rationale for duplicating TiddlyWeb's handling of JSON tiddlers.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

On Wed, 9 Apr 2014, Jeremy Ruston wrote:

> This is the bit that I still struggle with. There's still entirely special
> behaviour for application/json tiddlers. From the point of view of
> TiddlyWiki these tiddlers really are application/json, and not some special
> TW variant implied by the special content type. When the user drags in a
> .json file from the desktop they get an application/json tiddler
> automatically. End users are going to be confused if dragging text/html
> content gives them text/html tiddlers, but dragging application/json
> content gives them some different type.

Strictly speaking in neither case are they getting text/html nor
application/json, they are getting tiddlers which _contain_ text/html
and application/json. I think this is important.

In the TiddlyWeb context the situation is somewhat different.

> The trouble is that I wouldn't expect the existence of a representation
> transformation layer to break the basic data store layer protocol of "thou
> shall GET what thou PUT".

Actually, that's the rule that is being followed:

If you PUT something in a JSON container that you are claiming
is JSON if you ask for JSON back, that's what you get. Same
thing happens if you PUT something in a JSON container that you
are claiming is image/png when you ask for image/png back you
will get an image.

This rule works fine for everything _but_ application/json which is
why I've suggested this change:

>> * TiddlyWeb extends itself to treat "application/x-tiddler+json" in
>> much the same way it treats "application/json" now, deprecating
>> but continuing to support the latter.

If that change is made _and_ TW5 starts PUTting and GETting tiddlers
as application/x-tiddler+json, then you will get the results that
you want whatever value you put in the type field of the tiddler.

That change makes the claim of "we've got a thing called a tiddler
_in_ some JSON" and makes no claim on what the tiddler itself is
claiming as its type.

I'm happy to make that change (but it would need to be a few days
from now, not immediately).

However, the same logic that drives that change seems like it might
be sensible for TW5 too, thus why I've brought it up. Up to you
though, I suppose.

[Jeremy Ruston]

Hi Chris

Strictly speaking in neither case are they getting text/html nor
application/json, they are getting tiddlers which _contain_ text/html
and application/json. I think this is important.

That's a useful distinction. So, TW5 wants to PUT/GET a representation of tiddlers that is always JSON, regardless of the type(s) that they contain.

Actually, that's the rule that is being followed:

    If you PUT something in a JSON container that you are claiming
    is JSON if you ask for JSON back, that's what you get. Same
    thing happens if you PUT something in a JSON container that you
    are claiming is image/png when you ask for image/png back you
    will get an image.

This rule works fine for everything _but_ application/json which is
why I've suggested this change:

But doesn't that add up to the rule *not* being followed because of the inconsistent treatment of tiddlers that contain application/json?

 

* TiddlyWeb extends itself to treat "application/x-tiddler+json" in
  much the same way it treats "application/json" now, deprecating
  but continuing to support the latter.

If that change is made _and_ TW5 starts PUTting and GETting tiddlers
as application/x-tiddler+json, then you will get the results that
you want whatever value you put in the type field of the tiddler.

Does that mean that TW5 should always put tiddlers as application/x-tiddler+json, regardless of the type field on the tiddler? And would still be able to pass along the tiddler's type field in the usual way in the body?

Thanks for your patience, I'm sure I'm still misunderstanding what's going on,

Best wishes

Jeremy

That change makes the claim of "we've got a thing called a tiddler
_in_ some JSON" and makes no claim on what the tiddler itself is
claiming as its type.

I'm happy to make that change (but it would need to be a few days
from now, not immediately).

However, the same logic that drives that change seems like it might
be sensible for TW5 too, thus why I've brought it up. Up to you
though, I suppose.

--
Chris Dent                                   http://burningchrome.com/
                                [...]

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

On Wed, 9 Apr 2014, Jeremy Ruston wrote:

>> This rule works fine for everything _but_ application/json which is
>> why I've suggested this change:
>
> But doesn't that add up to the rule *not* being followed because of the
> inconsistent treatment of tiddlers that contain application/json?

That's what I just said.

>> If that change is made _and_ TW5 starts PUTting and GETting tiddlers
>> as application/x-tiddler+json, then you will get the results that
>> you want whatever value you put in the type field of the tiddler.
>
> Does that mean that TW5 should always put tiddlers as
> application/x-tiddler+json,
> regardless of the type field on the tiddler? And would still be able to
> pass along the tiddler's type field in the usual way in the body?

Yes, if/when that change is made the content type for PUTting a JSON
container of _any_ type of tiddler will be as above, and you can set
the 'type' to whatever you want.

There are, of course, other content types with which you can PUT a
tiddler but JSON is the primary one.

[Jeremy Ruston]

Hi Chris

Yes, if/when that change is made the content type for PUTting a JSON
container of _any_ type of tiddler will be as above, and you can set
the 'type' to whatever you want.

So, it sounds like the only change on the TW5 side would be to change the content type for PUTs here:

https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js#L165

Is that right? Would you expect anything else to need changing?

Best wishes

Jeremy.

 

There are, of course, other content types with which you can PUT a
tiddler but JSON is the primary one.

--
Chris Dent                                   http://burningchrome.com/
                                [...]

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[FND]

> So, it sounds like the only change on the TW5 side would be to change
> the content type for PUTs

Well, actually...

I've had some scenic train time to think about this today, or perhaps it
was the wind tunnel clearing my pathways - either way, I've come to the
conclusion that a tiddler's `type` refers exclusively to that tiddler's
`text`. Thus PUTing a tiddler with `type: text/x-tiddler` is not only
redundant, it also eliminates the standard mechanism for describing the
content type of the tiddler body. IOW, it's a hack.

In short: If the body is $contentType, then `type` should say so.

Where `text/x-tiddler` (or whatever) makes perfect sense is on GET, as
that's where the client states what it expects to receive - and clearly
in the case of tiddler editors, that's a tiddler rather than "some
JSON".

(Note that `text/x-tiddler` on PUT would make sense if the HTTP
Content-Type header didn't end up as the respective tiddler's `type`
attribute - but that's a different, probably unhelpful discussion.)

cdent: I realize this contradicts yesterday's conclusion, but I'd warned
you that my brain wasn't reliable then (not that it ever is). Not that
we didn't discuss those aspects, but it wasn't quite phrased in these
terms.

[Jeremy Ruston]

Hi FND

I've had some scenic train time to think about this today, or perhaps it
was the wind tunnel clearing my pathways

Wind tunnel?!? Sounds like that could be helpful for my occasional mental befuddlement.

 

- either way, I've come to the
conclusion that a tiddler's `type` refers exclusively to that tiddler's
`text`.

If I understood cdent correctly, it was to suggest the distinction between the type of the tiddler (which is to say the type that should be used to interpret it's body text) and the type of the representation that is PUT to change it.

 

Thus PUTing a tiddler with `type: text/x-tiddler` is not only
redundant, it also eliminates the standard mechanism for describing the
content type of the tiddler body. IOW, it's a hack.

At the moment, the content-type for the HTTP PUT to write a tiddler is "application/json"; the type field of the tiddler is in the JSON body of the request. cdent is suggesting that we change it to something else, which would be an application specific json content type. 

In short: If the body is $contentType, then `type` should say so.

I think we agree, if `type` is the `type` field of the tiddler, not the content-type of the HTTP request.

 

Where `text/x-tiddler` (or whatever) makes perfect sense is on GET, as
that's where the client states what it expects to receive - and clearly
in the case of tiddler editors, that's a tiddler rather than "some
JSON".

(Note that `text/x-tiddler` on PUT would make sense if the HTTP
Content-Type header didn't end up as the respective tiddler's `type`
attribute - but that's a different, probably unhelpful discussion.)

I think that is the problem right there: at the moment TiddlyWeb conflates the two types in the special case of application/json.

 

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[FND]

> Note that `text/x-tiddler` on PUT would make sense if the HTTP
> Content-Type header didn't end up as the respective tiddler's `type`
> attribute - but that's a different, probably unhelpful discussion.

Correction (my tests last nights were misleading): The representation's
`type` actually takes precedence over the Content-Type header.

$ echo '{ "type": "application/json" }' | \
curl -X PUT -H "Content-Type: text/x-tiddler" --data-binary @- \
http://example.org/.../foo

The resulting tiddler has `type: application/json`.

However, while semantically that seems more correct ("here's a tiddler",
rather than some arbitary JSON), I'm not sure the distinction matters on
PUT. On GET it matters because it actually changes the representation.

> Wind tunnel?!? Sounds like that could be helpful for my occasional
> mental befuddlement.

https://www.indoor-skydiving.com (went there to practice for my upcoming
license completion) - can definitely recommend it.

[chris...@gmail.com]

On Thu, 10 Apr 2014, FND wrote:

> However, while semantically that seems more correct ("here's a tiddler",
> rather than some arbitary JSON), I'm not sure the distinction matters on
> PUT. On GET it matters because it actually changes the representation.

As I said elsewhere I think having parity on GET and PUT is useful,
largely because of the semantic correctness.

(Note the same semantics logic is why I think TW5 should consider
being more particular about tiddlers labeled as JSON.)

[chris...@gmail.com]

On Wed, 9 Apr 2014, Jeremy Ruston wrote:

>>> Yes, if/when that change is made the content type for PUTting a JSON
>> container of _any_ type of tiddler will be as above, and you can set
>> the 'type' to whatever you want.
>
> So, it sounds like the only change on the TW5 side would be to change the
> content type for PUTs here:
>
> https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js#L165
>
> Is that right? Would you expect anything else to need changing?

Both PUT and GET would change. The driving force is providing a way to
ensure you can always GET an encapsulated tiddler (text, tags, fields,
etc) in JSON form even if the tiddler contains JSON. As expressed
elsewhere the change on the PUT side is for symmetry.

It's becoming increasingly unlikely that I will get to this before mid
week next week. So if anyone is feeling motivated, a pull request
would be most exciting, even if it were just failing tests.

[Jeremy Ruston]

Hi Chris

Sorry to draw on your patience again, but I'm back to being confused.

> (Note the same semantics logic is why I think TW5 should consider being more particular about tiddlers labeled as JSON.)

This is what I don't understand. What is it that TW5 is doing with JSON that is wrong?

TW5 allows tiddlers to have the type "application/json", and I still don't see why that is wrong.

TW5 uses "application/json" as the content type for PUTting modifications to the server; at one point I thought that this discussion was suggesting that the content-type for PUTting should be something else.

> The driving force is providing a way to ensure you can always GET an encapsulated tiddler (text, tags, fields, etc) in JSON form even if the tiddler contains JSON

But isn't the fact that we can't reliably GET a tiddler with all its fields a symptom of a problem with the design of TiddlyWeb? Other data stores like PouchDB don't appear to have any similar behaviour. I don't understand why it is TW5's problem.

> It's becoming increasingly unlikely that I will get to this before mid week next week

At my present rate of progress it's going to take me longer than that to understand what's going on :)

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

On Thu, 10 Apr 2014, Jeremy Ruston wrote:

>> (Note the same semantics logic is why I think TW5 should consider being
> more particular about tiddlers labeled as JSON.)
>
> This is what I don't understand. What is it that TW5 is doing with JSON
> that is wrong?

Not wrong, just semantically confusing. I'm not saying anything _has_
to change, just that the conversations that have led to the suggested
change in TiddlyWeb may be meaningful to TW5. They are, however, two
separate issues and can progress independently of one another.

> TW5 allows tiddlers to have the type "application/json", and I still don't
> see why that is wrong.

This is one issue. I'm not sure it is wrong but it might be
incomplete. A tiddler which contains JSON should say so in some
fashion, but there are different types of tiddlers which contain JSON.
Some of them are aggregates of tiddlers, a form that TW5 has awareness
of what to do with. Some of them are raw JSON of a form that TW5 doesn't
know what to do with. It (seems to me that it) would be better if that
decision making was controlled by the type field rather inspection of
the contents of the text field, or some other field (if, in fact, that
is what is going on).

> TW5 uses "application/json" as the content type for PUTting modifications
> to the server; at one point I thought that this discussion was suggesting
> that the content-type for PUTting should be something else.

This is the other issue. I've conceded that the content-type for
PUTting and GETting should be something different.

> But isn't the fact that we can't reliably GET a tiddler with all its fields
> a symptom of a problem with the design of TiddlyWeb? Other data stores like
> PouchDB don't appear to have any similar behaviour. I don't understand why
> it is TW5's problem.

It's not. I've already agreed to change TiddlyWeb.

> At my present rate of progress it's going to take me longer than that to
> understand what's going on :)

I think we're trying to figure out what the type attribute means and
in what contexts.

[Jeremy Ruston]

Hi Chris

Great, that's very helpful, thank you.

TW5 allows tiddlers to have the type "application/json", and I still don't
see why that is wrong.

This is one issue. I'm not sure it is wrong but it might be
incomplete. A tiddler which contains JSON should say so in some
fashion, but there are different types of tiddlers which contain JSON.
Some of them are aggregates of tiddlers, a form that TW5 has awareness
of what to do with. Some of them are raw JSON of a form that TW5 doesn't
know what to do with. It (seems to me that it) would be better if that
decision making was controlled by the type field rather inspection of
the contents of the text field, or some other field (if, in fact, that
is what is going on).

Just to be clear, this issue is independent? I mean, if we fixed the PUTting and GETting content-type TiddlyWiki could continue to use tiddlers with the type field set to "application/json" without any limitations or peculiarities? Meaning that this is a local issue around TW5's handling of JSON?

Anyhow, I think the fact that the first JSON tiddler you saw happened to be a bundle of tiddlers is perhaps a red herring. TiddlyWiki5 has JSON tiddlers as first class citizens, and they are used for all kinds of purposes by the core, and can be freely employed by users too. The role of a JSON tiddler is determined by its fields (typically tags, but it could be any criteria), just as the role of a wikitext tiddler is determined by its fields rather than a subtype of wikitext.

> Some of them are raw JSON of a form that TW5 doesn't know what to do with

There aren't any JSON tiddlers that TW5 doesn't know what to do with; it has features to enable it to deal with any arbitrary JSON in tiddlers (I plan more, such as a hierarchical JSON editor).

 

TW5 uses "application/json" as the content type for PUTting modifications
to the server; at one point I thought that this discussion was suggesting
that the content-type for PUTting should be something else.

This is the other issue. I've conceded that the content-type for
PUTting and GETting should be something different.

That clarifies things, thank you.

Best wishes

Jeremy

[chris...@gmail.com]

On Thu, 10 Apr 2014, Jeremy Ruston wrote:

> Just to be clear, this issue is independent? I mean, if we fixed the
> PUTting and GETting content-type TiddlyWiki could continue to use tiddlers
> with the type field set to "application/json" without any limitations or
> peculiarities? Meaning that this is a local issue around TW5's handling of
> JSON?

Yes, that's why I keep saying there are two separate issues.

> Anyhow, I think the fact that the first JSON tiddler you saw happened to be
> a bundle of tiddlers is perhaps a red herring. TiddlyWiki5 has JSON
> tiddlers as first class citizens, and they are used for all kinds of
> purposes by the core, and can be freely employed by users too. The role of
> a JSON tiddler is determined by its fields (typically tags, but it could be
> any criteria), just as the role of a wikitext tiddler is determined by its
> fields rather than a subtype of wikitext.

I guess what I'm trying to say that those tiddlers, in contexts
other than TW5, will be easier to make do "special" things if those
doings are dispatched via one aspect: their type.

This is already true of tiddlers which are text: we know that a
tiddler of text/x-tiddlywiki should be rendered and rendered as
tiddlywiki clasic; we know that a tiddler of text/x-markdown should
be rendered and rendered as markdown; etc. This is despite the fact
that what those tiddlers really are is text/plain.

You could make a very similar argument for various structures of
JSON.

Like I've said before, considering this as a change is entirely
up to you. The reason I keep mentioning it is because I want to be
sure that tiddlers are capable (by whatever definition) in lots of
different contexts. Have a single mode of dispatch makes that
easier.

If you have some which are special rather than generic, it would make
sense to signify that now rather than later. Doing it now would avoid
needing to make the kind of abrupt (and biggish) change that the _other_
issue is bringing to TiddlyWeb.

>> Some of them are raw JSON of a form that TW5 doesn't know what to do with
>
> There aren't any JSON tiddlers that TW5 doesn't know what to do with; it
> has features to enable it to deal with any arbitrary JSON in tiddlers (I
> plan more, such as a hierarchical JSON editor).

But TiddlyWeb, for example, not only doesn't know what to do with
them but also would struggle to add code to do interesting things
with them if it has to do dispatch in a variety of ways.

Does that help to clarify it a bit? I'm trying to make the tiddlers
more portable. The bug you identified in TiddlyWeb demonstrated that
TiddlyWeb was working against that portability. I want to be sure
that TW5 doesn't work against it too.

[Jeremy Ruston]

Hi Chris

OK that's very helpful thanks.

As far as changing the content type for PUTs and GETs, I can flick the switch for TW5 very easily, but I guess we might worry about backwards compatibility - when the time comes would it be reasonable for me to sniff the TiddlyWeb version number in /status to determine which content type to use?

With respect to using semantically meaningful content types for TW5's JSON tiddlers, I've opened a ticket:

https://github.com/Jermolene/TiddlyWiki5/issues/551

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscribe@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[PMario]

Reading the whole thread is confusing. So let me recap, if I do understand it right.

There is an existing convention, that is established between every server and every browser (client).
The "content-type". It defines, how a server has to deliver content to the browser. So the browser is able to display it.
This convention can not be changed, because every server has to work in the same way. ...

To make it easier for me, I'm not using application/json for the beginning.

Eg I did upload a "button image" tiddler in 2 different formats. [1] content-type: text/plain and [2] content-type:image/svg+xml

If I request button-text-plain [1] with the browser https://dev.tweb.at/api/bags/default_public/tiddlers/button-text-plain ...
   I get the tiddler.fields.text as text  -> OK

If I request button-image [2] : GET https://dev.tweb.at/api/bags/default_public/tiddlers/button-image  ...
   I get tiddler.fields.text - the browser shows an image, because of the response header: "Content-type: image/svg+xml; charset=UTF-8."    -> right? -> OK

So imo for TiddlyWeb as a content delivery service, this is the expected behaviour.


For tiddlywiki the mechanism is slightly different than for the default browser. ...
If you open the page [3] https://dev.tweb.at/ and click the "button-image" tiddler,
    then TiddlyWiki request: GET https://dev.tweb.at/api/bags/default_public/tiddlers/button-image  --- Accept: "application/json"

which is the same as if you click: https://dev.tweb.at/api/bags/default_public/tiddlers/button-image.json  with the browser.

What we get is a tiddler with a response header: "Content-type: application/json" -> OK ... but the tiddler contains a tiddler.fields.type, that tells TW to display it as "image/svg+xml" -> OK ... Cool the result for the browser and TiddlyWiki is the same. Both display an image, even if they did request different tings from the server.

---------------

Now comes the crux.

If I create a tiddler named: data-json    type: application/json
with the content

{"key": "value"}

Everything is perfectly fine if I click: https://dev.tweb.at/api/bags/default_public/tiddlers/data-json
It returns: {"key": "value"} -> OK as expected for the browser. 
There has to be a possibility to get exactly that behaviour, because it my be needed that way. 

But if I open https://dev.tweb.at/ and open the "data-json" tiddler 

TiddlyWiki GET request header is   Accept:"application/json" 
The server sends {"key": "value"} ... Which TiddlyWiki can't handle.  ->> ERROR

IMO if TiddlyWiki requests "Accept: application/json" the TiddlyWeb behaviour is right. 

The missing thing now is a possibility to retrieve the tiddler json representation. So that TW can handle it. 

------------
a test only: https://dev.tweb.at/api/bags/default_public/tiddlers/data-json.txt is fine. 
------------

I think the cleanest way is, TW changes the request to 
GET the tiddler json representation to: Accept: text/vnd.tiddlywiki+json

If you have a look at: http://www.iana.org/assignments/media-types/media-types.xhtml  
There are several vnd.xxx+json definitions. So it should be ok. 

---------

IMO TiddlyWeb needs to change the mechanism too, because if I PUT application/json 
I expect this link: https://dev.tweb.at/api/bags/default_public/tiddlers/data-json to deliver application/json ... 

but there still needs to be a way to get the tiddler json representation. 
So imo : https://dev.tweb.at/api/bags/default_public/tiddlers/data-json.json  should return it. 

Since ".json" is a convention that returns a tiddler it should be true for every tiddler.

just my thoughts
have fun!
mario

[chris...@gmail.com]

On Sat, 12 Apr 2014, Jeremy Ruston wrote:

> As far as changing the content type for PUTs and GETs, I can flick the
> switch for TW5 very easily, but I guess we might worry about backwards
> compatibility - when the time comes would it be reasonable for me to sniff
> the TiddlyWeb version number in /status to determine which content type to
> use?

That's fair assuming status is being run on the server (it's a
plugin so though it is usually there, it doesn't _have_ to be).

I'm not super concerned with maintaining aggressive backwards
compatibility because I want people to aggressively upgrade as and
when they need.

> With respect to using semantically meaningful content types for TW5's JSON
> tiddlers, I've opened a ticket:
>
> https://github.com/Jermolene/TiddlyWiki5/issues/551

I've created the pair on the tiddlyweb side:

https://github.com/tiddlyweb/tiddlyweb/issues/99

[Chris Dent]

On Tuesday, April 15, 2014 9:07:47 PM UTC+1, Chris Dent wrote:

I've created the pair on the tiddlyweb side:

     https://github.com/tiddlyweb/tiddlyweb/issues/99

This fix has been released in TiddlyWeb 2.3.0 and Tank has been updated with that version.

If tw5 is updated with the necessary changes (I think they're described in the thread above) I'll push that version into Tank and we can confirm it is okay.

Thanks.

[chris...@gmail.com]

On Mon, 12 May 2014, Chris Dent wrote:

> If tw5 is updated with the necessary changes (I think they're described in
> the thread above) I'll push that version into Tank and we can confirm it is
> okay.

Presumably tiddlyspace will need to be update as well, but how that
will happen is unclear.

[Jeremy Ruston]

Hi Chris

> This fix has been released in TiddlyWeb 2.3.0 and Tank has been updated with that version.

What would be the best way for the client to detect that it's running against a tiddlyweb server that would prefer the new content type?

Best wishes

Jeremy

--

You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[chris...@gmail.com]

On Mon, 12 May 2014, Jeremy Ruston wrote:

> What would be the best way for the client to detect that it's running
> against a tiddlyweb server that would prefer the new content type?

Did we not already say for most deployments asking /status
or /status.js is probably the easiest way? That'll get you:

https://tank.peermore.com/status

{"username": "cdent",
"challengers": ["tiddlywebplugins.tank.challenger"],
"version": "2.3.0"}

or

https://tank.peermore.com/status.js

var tiddlyweb = tiddlyweb || {};
tiddlyweb.status = {
"username": "cdent",
"challengers": ["tiddlywebplugins.tank.challenger"],
"version": "2.3.0"
};

Status is an optional plugin but is included on most servers because
it's the best way to get at username info from JavaScript.

So parse the version string and if > 2.3.0, win.

A far more hacky way, but usable on a system with no /status is to GET
a known resource[1] with `accept: application/vnd.tiddlyweb+json` and
inspect the content-type of the response. If it is `application/json`
then the new type is supported. If it is `text/html` or anything else
then it is not.

Compare:

curl -v -H "Accept: application/json" \

http://tiddlyspace.com/bags/common/tiddlers/normalize.css

and

curl -v -H "Accept: application/vnd.tiddlyweb+json" \
http://tiddlyspace.com/bags/common/tiddlers/normalize.css

with

curl -v -H "Accept: application/vnd.tiddlyweb+json" \
https://tank.peermore.com/bags/common/tiddlers/normalize.css

I would think that depending on /status (and thus making
tiddlywebplugins.status a requirement for any server hosting tw5
content) is reasonable.

Another option is just expect and require people use 2.3.0. That will
put some leverage behind getting people (notably tiddlyspace.com) to
keep up to date.

[1] As long as that resource is not a tiddler with type attribute of
`application/json`. If you choose one of those then you'll trigger the
bug on older systems. I suppose that provides a third avenue for
discovery...