Anyway, I'm starting to think it might be time to have a group for discussing a server API. I've got a summary at the bottom of this post if you want to skip all the guff ^_^
Tom Otovos and I have been discussing this a bit; we're both involved in one way or another with GTD projects and it would be great to add some friendly support for server side functionality. Imagine having a server call where you could ask for tiddlers using a statement like this:
SELECT title, tags, fields <-- choosing the parts of each tiddler to return WHERE ..a flexible logical selector here.. title = [[Some Tiddler Title]] OR ( ISTAGGED [[some tag]] AND ISTAGGED [[some other tag]] AND NOTTAGGED [[a tag you don't want]] )
Or if no-one likes the SQL thing the language could be more like:
getTiddlerInfo components:title tags fields" condition:"title:[[Some Tiddler Title]]" alternatecondition:"tags:[[some tag]] [[some other tag]] -[[a tag you don't want]]" which looks different, but basically the idea is that it'd be cool to specify both which tiddlers to return, and what you wanted in the reply. Pretty much all of the time you'd want title tags and fields -- ie. everything-but-the-content, but I reckon we should be looking at a server api to allow for more flexible usage than that, because once you're on a dynamic serverside, you take away the single-file size restriction on TiddlyWikis, I reckon you would want to use it for stuff that people aren't using it for today.
A fairly tame example might be handling "done" projects in a GTD system, for example. Imagine if you could get metadata info for everything -except- project/action tiddlers marked as "archived". You could use your GTD TW for years, have thousands of archived projects and tens of thousands of archived actions, and you wouldn't need to download 100k of archived tiddler titles every time you opened the TW.
Or having a TW server side that did dynamic generation of tiddlers -- perhaps stats for sports teams that are generated on request, or a tiddler for every verse in the bible, or whatever. You wouldn't want to pre-fetch every title for that list of tiddlers. How about a TW interface to Wikipedia.. asking for a tiddler called "Mr T" will send the Wikipedia article to your local TW.
Admittedly there are some issues to address about searching, unique naming etc, but I like the idea that the _potential_ for this kind of stuff should allowed in a common serverside API.
Anyway, once Tom and I got to that point we realised that more people should be talking about it than us.
*Summary:*
I'd love to have a common server API that means you could link any client (from a full serverside TW plugin, to a tiny TW plugin to save Minesweeper scores to a common server, to a TiddlySnip-type browser extention) to any server (db-backed with versioning services like ccT and ZW, a read-only plugin repository, etc etc).
Does anyone else think that it'd be nice to have a big ol' chat between all the possible stakeholders in a serverside API? Maybe it should be another Google Group.
I think, such a group is a good idea. Even though I am currently not involved in serverside stuff I would definitely be interested in working in that group, e.g. to better integrate my YourSearchPlugin with serverside installations.
Why don't you go ahead and create that group (TiddlyWikiServerSideDev ?) ?
BTW: Some things planned for the core development are also touching this area (like "Support lazy loading of tiddlers" ( http://trac.tiddlywiki.org/tiddlywiki/ticket/209). So some results from this group will probably also influence future TiddlyWiki versions
> There's been a bit of discussion about server sides lately.. some > questions and requests from users, and this interesting thread about (among > other things) a common serverside api:
> Anyway, I'm starting to think it might be time to have a group for > discussing a server API. I've got a summary at the bottom of this post if > you want to skip all the guff ^_^
> Tom Otovos and I have been discussing this a bit; we're both involved in > one way or another with GTD projects and it would be great to add some > friendly support for server side functionality. Imagine having a server > call where you could ask for tiddlers using a statement like this:
> SELECT > title, tags, fields <-- choosing the parts of each tiddler to return > WHERE > ..a flexible logical selector here.. > title = [[Some Tiddler Title]] > OR ( ISTAGGED [[some tag]] > AND ISTAGGED [[some other tag]] > AND NOTTAGGED [[a tag you don't want]] > )
> Or if no-one likes the SQL thing the language could be more like:
> getTiddlerInfo > components:title tags fields" > condition:"title:[[Some Tiddler Title]]" > alternatecondition:"tags:[[some tag]] [[some other tag]] -[[a tag you > don't want]]" > which looks different, but basically the idea is that it'd be cool to > specify both which tiddlers to return, and what you wanted in the reply. > Pretty much all of the time you'd want title tags and fields -- > ie. everything-but-the-content, but I reckon we should be looking at a > server api to allow for more flexible usage than that, because once you're > on a dynamic serverside, you take away the single-file size restriction on > TiddlyWikis, I reckon you would want to use it for stuff that people aren't > using it for today.
> A fairly tame example might be handling "done" projects in a GTD system, > for example. Imagine if you could get metadata info for everything -except- > project/action tiddlers marked as "archived". You could use your GTD TW for > years, have thousands of archived projects and tens of thousands of archived > actions, and you wouldn't need to download 100k of archived tiddler titles > every time you opened the TW.
> Or having a TW server side that did dynamic generation of tiddlers -- > perhaps stats for sports teams that are generated on request, or a tiddler > for every verse in the bible, or whatever. You wouldn't want to pre-fetch > every title for that list of tiddlers. How about a TW interface to > Wikipedia.. asking for a tiddler called "Mr T" will send the Wikipedia > article to your local TW.
> Admittedly there are some issues to address about searching, unique naming > etc, but I like the idea that the _potential_ for this kind of stuff should > allowed in a common serverside API.
> Anyway, once Tom and I got to that point we realised that more people > should be talking about it than us.
> *Summary:*
> I'd love to have a common server API that means you could link any client > (from a full serverside TW plugin, to a tiny TW plugin to save Minesweeper > scores to a common server, to a TiddlySnip-type browser extention) to any > server (db-backed with versioning services like ccT and ZW, a read-only > plugin repository, etc etc).
> Does anyone else think that it'd be nice to have a big ol' chat between > all the possible stakeholders in a serverside API? Maybe it should be > another Google Group.
I would support having a separate group but am also happy with using the TWDev group.
As for plugin development, I think the best way is to make the TW plugin a standard set of API to interact with the server-side. server-side developers could just act according to the set of API. I myself is more a PHP developer while a javascript newbie so I think having a person who knows javascript to develop the plugin and server-side developers could develop the backend. Probabily feature request in the server-side group as well from server-side developers :)
Of course I think a server-side API is a fine idea. ;) Unfortunately I'm quite busy (trying to secure job for next year -- which means 100% physics for a while).
Anyway, I have modularized ZiddlyWiki to a large extent, so that 1) every part of the interface lives in tiddlers 2) a common ZiddlyWikiPlugin and ZiddlyStatus which can be (re)used for other server-sides.
The basic API is in URL query parameters: action=get&title=... body of response should be a properly formatted TW DIV. action=save& body of POST request should be a properly formatted TW DIV. action=delete&title=... Self-explanatory action=lock&title=... Lock a tiddler for editing action=unlock&title=... Unlock a tiddler when done editing (e.g. an edit is cancelled) action=get_revisions Get a list of tiddlers and their "revision". Used to implement "show old versions of this tiddler".
"revision" is a new extended attribute that is needed by any server-side that wants to do version control, undelete, or notice renames. Currently I am using the Zope revision string. I do not think the value of this string matters as long as it is unique. (e.g. concatenate 'modified' time/date with tiddler text and compute the md5 would be fine) Sending DIV's is kind of obvious, and allows us to encode all TW information, including extended fields. (note ZW does not handle extended fields yet)
A key part of the API is the special tiddler ZiddlyStatus, which is a systemConfig tiddler, but is also readOnly, and I also re-evaluate it when it is downloaded. (unlike other systemConfig tiddlers) This should contain any status information like: 1) whether the user is logged in 2) a list of tiddler revisions (so TW can notice and re-download tiddlers when they change on the server) 3) Any serverside-specific info you want to keep. (though, I hope we can minimize/eliminate this) I suggest that the server respond with a HTTP '304 Not Modified' if nothing has changed on the server to decrease traffic.
Also I have attempted to make all status info be passed using HTTP status codes. e.g. if I do an action=get for a tiddler that doesn't exist, the server should return HTTP '404 Not Found' rather than some TW-specific or server-side-specific status message.
Not all of the above features are yet implemented (in particular, action=save does not send a DIV yet), and there are probably other bugs. But I invite everyone to take this plugin and run with it. When I have more time to devote to TW I will take the plugin and adapt to whatever you do. Note this is in my tiddlywiki.org svn repository. So if you can clone my repo and put changes in svn that would be ideal so I can merge them later.
I would prefer to keep this on the TiddlyWikiDev list. I don't think there's critical mass to justify another list, and others not developing server-sides (such as GTD and TiddlySnip) might not have any idea to join such a list, but can benefit from the discussion.
A couple more comments below about your suggestions...
> Anyway, I'm starting to think it might be time to have a group for > discussing a server API. I've got a summary at the bottom of this post if > you want to skip all the guff ^_^
> Tom Otovos and I have been discussing this a bit; we're both involved in one > way or another with GTD projects and it would be great to add some friendly > support for server side functionality. Imagine having a server call where > you could ask for tiddlers using a statement like this:
> SELECT > title, tags, fields <-- choosing the parts of each tiddler to return > WHERE > ..a flexible logical selector here.. > title = [[Some Tiddler Title]] > OR ( ISTAGGED [[some tag]] > AND ISTAGGED [[some other tag]] > AND NOTTAGGED [[a tag you don't want]] > )
> Or if no-one likes the SQL thing the language could be more like:
I personally prefer to keep things in URL query parameters, and responses in TW-format DIV's. If we could get rid of the unneccessary encoding in TW (\n and \n) the response, as DIV's, would be viewable in any browser...even search engines could parse it. (Using <pre> instead has been discussed before)
> getTiddlerInfo > components:title tags fields" > condition:"title:[[Some Tiddler Title]]" > alternatecondition:"tags:[[some tag]] [[some other tag]] -[[a tag you > don't want]]" > which looks different, but basically the idea is that it'd be cool to > specify both which tiddlers to return, and what you wanted in the reply. > Pretty much all of the time you'd want title tags and fields -- > ie. everything-but-the-content, but I reckon we should be looking at a > server api to allow for more flexible usage than that, because once you're > on a dynamic serverside, you take away the single-file size restriction on > TiddlyWikis, I reckon you would want to use it for stuff that people aren't > using it for today.
My action=get API above is quite primitive. But I see no reason that one couldn't do: action=get&tag=foo but I'm a little wary of forcing any server-side author to implement SQL in order to have a compliant server.
An API extension to get metadata only might be quite useful here. Imagine action=get_metadata, where the response is a list of DIV's (as appear in the HTML store) but tagged with 'metadataOnly'. That way TW can put all of the tiddlers into the timeline, perform complex queries on the tags, but the content would be missing. (This is effectively a mechanism for load-on-demand too)
> A fairly tame example might be handling "done" projects in a GTD system, for > example. Imagine if you could get metadata info for everything -except- > project/action tiddlers marked as "archived". You could use your GTD TW for > years, have thousands of archived projects and tens of thousands of archived > actions, and you wouldn't need to download 100k of archived tiddler titles > every time you opened the TW.
Then, how would you access archived tiddlers if you wanted to? In what you describe above, "archived" is identical to "deleted".
At a very minimum, I think one would need to download the metadata for every tiddler. Making some of them load-on-demand would be up to the server-side. Making all tiddlers with the tag 'archived' load on demand would be a fine idea. Though I suspect the metadata for GTD tiddlers is larger than the content. .e.g. "Doctor's appointment" with a lot of metadata.
> Or having a TW server side that did dynamic generation of tiddlers -- > perhaps stats for sports teams that are generated on request, or a tiddler > for every verse in the bible, or whatever. You wouldn't want to pre-fetch > every title for that list of tiddlers. How about a TW interface to > Wikipedia.. asking for a tiddler called "Mr T" will send the Wikipedia > article to your local TW.
Dynamic generation of content is kind of a kludge. The old ZW had it and I've effectively disabled it all.
I don't have any problem with dynamically generated content, but we should not pretend that these are tiddlers. It should somehow be obvious that I'm looking at a sports stat page and not a tiddler I can edit. If the tiddler contains scripting, should "edit" show the server-side script or the results of the server-side script? (What if different server-sides use different scripting languages?) How do I know that I just edited a tiddler with a server-side script and I have to reload it an extra time after an edit?
> Admittedly there are some issues to address about searching, unique naming > etc, but I like the idea that the _potential_ for this kind of stuff should > allowed in a common serverside API.
Unique naming is only a problem when mixing tiddlers from different servers. In which case we should follow the InterWiki format: [[Wikipedia:Mr T]] where everything left of the ':' specifies the server.
Of course your server-side must proxy such a request since AJAX will not allow this, and if we do proxy such a request, we need to be vigilant that the response not contain HTML or javascript that could be an XSS vulnerability.
This is easy to do within a single get API: action=get&title=Wikipedia:Mr%20T
> Anyway, once Tom and I got to that point we realised that more people should > be talking about it than us.
> *Summary:*
> I'd love to have a common server API that means you could link any client > (from a full serverside TW plugin, to a tiny TW plugin to save Minesweeper > scores to a common server, to a TiddlySnip-type browser extention) to any > server (db-backed with versioning services like ccT and ZW, a read-only > plugin repository, etc etc).
I don't think we should ever automatically pull plugins. We shouldn't ever pull any javascript or HTML from an untrusted source.
What we *should* pull is tiddlers (which get rendered by TW -- no systemConfig), and tiddlers in other wiki markups (you can pull page source from MediaWiki -- and any other wiki).
I wonder if we could implement a crypto signature system for plugins? Put keys on tiddlywiki.com, verify plugins on download?
> Does anyone else think that it'd be nice to have a big ol' chat between all > the possible stakeholders in a serverside API? Maybe it should be another > Google Group.
-- Cheers, Bob McElrath [Univ. of California at
...
> I'd love to have a common server API that means you could link any client > (from a full serverside TW plugin, to a tiny TW plugin to save Minesweeper > scores to a common server, to a TiddlySnip-type browser extention) to any > server (db-backed with versioning services like ccT and ZW, a read-only > plugin repository, etc etc).
Excellent stuff, definitely agree. I've recently been working on serverside interfaces for TW as part of Osmosoft's current consulting work. We've also listed some server-side support as part of revision 2.2:
REST makes for a very nice, discoverable serverside API that you can explore from a webbrowser. Also, done right, it lays the ground work for support of 'static' servers, which is essentially just a file server, either accessed via file:// or http:// (including webDAV). In fact, it's possible to have TiddlySpot-level functionality on a plain webdav server (such as apple's mac.com), without having to install any serverside script. (Most current webservers ship with webdav support, including IIS and Apache).
> Does anyone else think that it'd be nice to have a big ol' chat between all > the possible stakeholders in a serverside API? Maybe it should be another > Google Group.
I wouldn't really favour a separate group just for API discussions, I think this belongs in the tiddlywikidev group.
I wasn't quite clear why use header instead of error code. I personally think passing error code is better than header because 1/ it is more flexible since we get to define what they mean 2/ a common function to interpret what is returned (may still be the case with header) 3/ ease debugging since header cannot be passed after there are output.
I am not sure if I am missing something important and would love to know what you think.
PS: I am also doing physics. Is there a relation with physics and server-side? XD
> I wasn't quite clear why use header instead of error code. I personally > think passing error code is better than header because > 1/ it is more flexible since we get to define what they mean
HTTP status codes define basically all conceivable states. If you have a specific error state that needs to be communicated that is not covered by HTTP status codes, then I'd reconsider...
> 2/ a common function to interpret what is returned (may still be the > case with header)
If we have a common ServerSidePlugin (some evolution of my ZiddlyWikiPlugin, hopefully) then we can choose any mechanism at all, since the code is shared. :) So this isn't an argument for any one mechanism over any other.
> 3/ ease debugging since header cannot be passed after there are output.
I don't understand this statement. The status code is passed to event handlers in exactly the same way as the response. So, it should be no more difficult to debug status codes than response text. See zw.addTiddler and config.commands.revisions in http://ziddlywiki.com/test2.1#ZiddlyWikiPlugin They're symmetric WRT response (parameter r) and status.
> I am not sure if I am missing something important and would love to > know what you think.
I'm just trying to make everything as SIMPLE as possible. Here I take any custom interface to be less simple than any standardized interface.
> PS: I am also doing physics. Is there a relation with physics and > server-side? XD
Dunno. Maybe I should make my physics notes public. :)
-- Cheers, Bob McElrath [Univ. of California at Davis, Department of Physics]
"A single death is a tragedy, a million deaths is a statistic." -- Joseph Stalin
Jeremy Ruston [jeremy.rus...@gmail.com] wrote: > Technically, I'm strongly in favour of a REST-style interface, which > Daniel's description pretty much fits. See
> REST makes for a very nice, discoverable serverside API that you can > explore from a webbrowser. Also, done right, it lays the ground work > for support of 'static' servers, which is essentially just a file > server, either accessed via file:// or http:// (including webDAV). In > fact, it's possible to have TiddlySpot-level functionality on a plain > webdav server (such as apple's mac.com), without having to install any > serverside script. (Most current webservers ship with webdav support, > including IIS and Apache).
I think without realizing it, I am also arguing for REST. (I didn't know what REST was until just now...) I think CC is arguing for RPC.
Note that WebDAV already has everything we need for a server-side.
-- Cheers, Bob McElrath [Univ. of California at Davis, Department of Physics]
"A single death is a tragedy, a million deaths is a statistic." -- Joseph Stalin
I wasn't quite clear why use header instead of error code. I personally think passing error code is better than header because 1/ it is more flexible since we get to define what they mean 2/ a common function to interpret what is returned (may still be the case with header) 3/ ease debugging since header cannot be passed after there are output.
I am not sure if I am missing something important and would love to know what you think.
PS: I am also doing physics. Is there a relation with physics and server-side? XD
CC
On 05/11/06, Bob McElrath - bob.mcelr...@gmail.com<+google+coolcold+271b7624a7.bob.mcelrath#gmail.com@
> Of course I think a server-side API is a fine idea. ;) Unfortunately > I'm quite busy (trying to secure job for next year -- which means 100% > physics for a while).
> Anyway, I have modularized ZiddlyWiki to a large extent, so that > 1) every part of the interface lives in tiddlers > 2) a common ZiddlyWikiPlugin and ZiddlyStatus which can be (re)used for > other server-sides.
> The basic API is in URL query parameters: > action=get&title=... > body of response should be a properly formatted TW DIV. > action=save& > body of POST request should be a properly formatted TW DIV. > action=delete&title=... > Self-explanatory > action=lock&title=... > Lock a tiddler for editing > action=unlock&title=... > Unlock a tiddler when done editing (e.g. an edit is cancelled) > action=get_revisions > Get a list of tiddlers and their "revision". Used to implement > "show old versions of this tiddler".
> "revision" is a new extended attribute that is needed by any server-side > that wants to do version control, undelete, or notice renames. > Currently I am using the Zope revision string. I do not think the value > of this string matters as long as it is unique. (e.g. concatenate > 'modified' time/date with tiddler text and compute the md5 would be > fine) Sending DIV's is kind of obvious, and allows us to encode all TW > information, including extended fields. (note ZW does not handle > extended fields yet)
> A key part of the API is the special tiddler ZiddlyStatus, which is a > systemConfig tiddler, but is also readOnly, and I also re-evaluate it > when it is downloaded. (unlike other systemConfig tiddlers) This > should contain any status information like: > 1) whether the user is logged in > 2) a list of tiddler revisions (so TW can notice and re-download > tiddlers when they change on the server) > 3) Any serverside-specific info you want to keep. (though, I hope > we can minimize/eliminate this) > I suggest that the server respond with a HTTP '304 Not Modified' if > nothing has changed on the server to decrease traffic.
> Also I have attempted to make all status info be passed using HTTP > status codes. e.g. if I do an action=get for a tiddler that doesn't > exist, the server should return HTTP '404 Not Found' rather than some > TW-specific or server-side-specific status message.
> Not all of the above features are yet implemented (in particular, > action=save does not send a DIV yet), and there are probably other bugs. > But I invite everyone to take this plugin and run with it. When I have > more time to devote to TW I will take the plugin and adapt to whatever > you do. Note this is in my tiddlywiki.org svn repository. So if you > can clone my repo and put changes in svn that would be ideal so I can > merge them later.
> I would prefer to keep this on the TiddlyWikiDev list. I don't think > there's critical mass to justify another list, and others not developing > server-sides (such as GTD and TiddlySnip) might not have any idea to > join such a list, but can benefit from the discussion.
> A couple more comments below about your suggestions...
> Daniel Baird [danielba...@gmail.com] wrote: > > Hi all,
> > There's been a bit of discussion about server sides lately.. some > questions > > and requests from users, and this interesting thread about (among other > > things) a common serverside api:
> > Anyway, I'm starting to think it might be time to have a group for > > discussing a server API. I've got a summary at the bottom of this post > if > > you want to skip all the guff ^_^
> > Tom Otovos and I have been discussing this a bit; we're both involved in > one > > way or another with GTD projects and it would be great to add some > friendly > > support for server side functionality. Imagine having a server call > where > > you could ask for tiddlers using a statement like this:
> > SELECT > > title, tags, fields <-- choosing the parts of each tiddler to return > > WHERE > > ..a flexible logical selector here.. > > title = [[Some Tiddler Title]] > > OR ( ISTAGGED [[some tag]] > > AND ISTAGGED [[some other tag]] > > AND NOTTAGGED [[a tag you don't want]] > > )
> > Or if no-one likes the SQL thing the language could be more like:
> I personally prefer to keep things in URL query parameters, and > responses in TW-format DIV's. If we could get rid of the unneccessary > encoding in TW (\n and \n) the response, as DIV's, would be viewable in > any browser...even search engines could parse it. (Using <pre> instead > has been discussed before)
> > getTiddlerInfo > > components:title tags fields" > > condition:"title:[[Some Tiddler Title]]" > > alternatecondition:"tags:[[some tag]] [[some other tag]] -[[a tag you > > don't want]]" > > which looks different, but basically the idea is that it'd be cool to > > specify both which tiddlers to return, and what you wanted in the reply. > > Pretty much all of the time you'd want title tags and fields -- > > ie. everything-but-the-content, but I reckon we should be looking at a > > server api to allow for more flexible usage than that, because once > you're > > on a dynamic serverside, you take away the single-file size restriction > on > > TiddlyWikis, I reckon you would want to use it for stuff that people > aren't > > using it for today.
> My action=get API above is quite primitive. But I see no reason that > one couldn't do: > action=get&tag=foo > but I'm a little wary of forcing any server-side author to implement SQL > in order to have a compliant server.
> An API extension to get metadata only might be quite useful here. > Imagine action=get_metadata, where the response is a list of DIV's (as > appear in the HTML store) but tagged with 'metadataOnly'. That way TW > can put all of the tiddlers into the timeline, perform complex queries > on the tags, but the content would be missing. (This is effectively a > mechanism for load-on-demand too)
> > A fairly tame example might be handling "done" projects in a GTD system, > for > > example. Imagine if you could get metadata info for everything -except- > > project/action tiddlers marked as "archived". You could use your GTD TW > for > > years, have thousands of archived projects and tens of thousands of > archived > > actions, and you wouldn't need to download 100k of archived tiddler > titles > > every time you opened the TW.
> Then, how would you access archived tiddlers if you wanted to? In what > you describe above, "archived" is identical to "deleted".
> At a very minimum, I think one would need to download the metadata for > every tiddler. Making some of them load-on-demand would be up to the > server-side. Making all tiddlers with the tag 'archived' load on > demand would be a fine idea. Though I suspect the metadata for GTD > tiddlers is larger than the content. .e.g. "Doctor's appointment" with > a lot of metadata.
> > Or having a TW server side that did dynamic generation of tiddlers -- > > perhaps stats for sports teams that are generated on request, or a > tiddler > > for every verse in the bible, or whatever. You wouldn't want to > pre-fetch > > every title for that list of tiddlers. How about a TW interface to > > Wikipedia.. asking for a tiddler called "Mr T" will send the Wikipedia > > article to your local TW.
> Dynamic generation of content is kind of a kludge. The old ZW had it > and I've effectively disabled it all.
> I don't have any problem with dynamically generated content, but we > should not pretend that these are tiddlers. It should somehow be > obvious that I'm looking at a sports stat page and not a tiddler I can > edit. If the tiddler contains scripting, should "edit" show the > server-side script or the results of the server-side script? (What if > different server-sides use different scripting languages?) How do I > know that I just edited a tiddler with a server-side script and I have > to reload it an extra time after an edit?
> > Admittedly there are some issues to address about searching, unique > naming > > etc, but I like the idea that the _potential_ for this kind of stuff > should > > allowed in a common serverside API.
> Unique naming is only a problem when mixing tiddlers from different > servers. In which case we should follow the InterWiki format: > [[Wikipedia:Mr T]] > where everything left of the ':' specifies the server.
> Of course your server-side must proxy such a request since AJAX will not > allow this, and if we do proxy such a request, we need to be vigilant > that the response not contain HTML or javascript that could be an XSS > vulnerability.
> This is easy to do within a single get API: > action=get&title=Wikipedia:Mr%20T
> > Anyway, once Tom and I got to that point we realised that more people > should
With respect to the header vs. error code issue, the REST approach is to use HTTP error codes as strictly as possible so that intermediate HTTP processors (like proxies) can process the traffic intelligently.
HTTP (and later WebDAV) were designed to solve pretty much exactly the problems we're looking at here so, as Bob says, these off-the-shelf mechanisms should suit us well.
Cheers
Jeremy
On 11/5/06, google.coolc...@dfgh.net <google.coolc...@dfgh.net> wrote:
> I wasn't quite clear why use header instead of error code. I personally > think passing error code is better than header because > 1/ it is more flexible since we get to define what they mean > 2/ a common function to interpret what is returned (may still be the case > with header) > 3/ ease debugging since header cannot be passed after there are output.
> I am not sure if I am missing something important and would love to know > what you think.
> PS: I am also doing physics. Is there a relation with physics and > server-side? XD
> CC
> On 05/11/06, Bob McElrath - bob.mcelr...@gmail.com > <+google+coolcold+271b7624a7.bob.mcelrath#gmail.com@ > spamgourmet.com> wrote:
> > Of course I think a server-side API is a fine idea. ;) Unfortunately > > I'm quite busy (trying to secure job for next year -- which means 100% > > physics for a while).
> > Anyway, I have modularized ZiddlyWiki to a large extent, so that > > 1) every part of the interface lives in tiddlers > > 2) a common ZiddlyWikiPlugin and ZiddlyStatus which can be (re)used for > > other server-sides.
> > The basic API is in URL query parameters: > > action=get&title=... > > body of response should be a properly formatted TW DIV. > > action=save& > > body of POST request should be a properly formatted TW DIV. > > action=delete&title=... > > Self-explanatory > > action=lock&title=... > > Lock a tiddler for editing > > action=unlock&title=... > > Unlock a tiddler when done editing (e.g. an edit is cancelled) > > action=get_revisions > > Get a list of tiddlers and their "revision". Used to implement > > "show old versions of this tiddler".
> > "revision" is a new extended attribute that is needed by any server-side > > that wants to do version control, undelete, or notice renames. > > Currently I am using the Zope revision string. I do not think the value > > of this string matters as long as it is unique. (e.g. concatenate > > 'modified' time/date with tiddler text and compute the md5 would be > > fine) Sending DIV's is kind of obvious, and allows us to encode all TW > > information, including extended fields. (note ZW does not handle > > extended fields yet)
> > A key part of the API is the special tiddler ZiddlyStatus, which is a > > systemConfig tiddler, but is also readOnly, and I also re-evaluate it > > when it is downloaded. (unlike other systemConfig tiddlers) This > > should contain any status information like: > > 1) whether the user is logged in > > 2) a list of tiddler revisions (so TW can notice and re-download > > tiddlers when they change on the server) > > 3) Any serverside-specific info you want to keep. (though, I hope > > we can minimize/eliminate this) > > I suggest that the server respond with a HTTP '304 Not Modified' if > > nothing has changed on the server to decrease traffic.
> > Also I have attempted to make all status info be passed using HTTP > > status codes. e.g. if I do an action=get for a tiddler that doesn't > > exist, the server should return HTTP '404 Not Found' rather than some > > TW-specific or server-side-specific status message.
> > Not all of the above features are yet implemented (in particular, > > action=save does not send a DIV yet), and there are probably other bugs. > > But I invite everyone to take this plugin and run with it. When I have > > more time to devote to TW I will take the plugin and adapt to whatever > > you do. Note this is in my tiddlywiki.org svn repository. So if you > > can clone my repo and put changes in svn that would be ideal so I can > > merge them later.
> > I would prefer to keep this on the TiddlyWikiDev list. I don't think > > there's critical mass to justify another list, and others not developing > > server-sides (such as GTD and TiddlySnip) might not have any idea to > > join such a list, but can benefit from the discussion.
> > A couple more comments below about your suggestions...
> > Daniel Baird [danielba...@gmail.com] wrote: > > > Hi all,
> > > There's been a bit of discussion about server sides lately.. some > questions > > > and requests from users, and this interesting thread about (among other > > > things) a common serverside api:
> > > Anyway, I'm starting to think it might be time to have a group for > > > discussing a server API. I've got a summary at the bottom of this post > if > > > you want to skip all the guff ^_^
> > > Tom Otovos and I have been discussing this a bit; we're both involved in > one > > > way or another with GTD projects and it would be great to add some > friendly > > > support for server side functionality. Imagine having a server call > where > > > you could ask for tiddlers using a statement like this:
> > > SELECT > > > title, tags, fields <-- choosing the parts of each tiddler to return > > > WHERE > > > ..a flexible logical selector here.. > > > title = [[Some Tiddler Title]] > > > OR ( ISTAGGED [[some tag]] > > > AND ISTAGGED [[some other tag]] > > > AND NOTTAGGED [[a tag you don't want]] > > > )
> > > Or if no-one likes the SQL thing the language could be more like:
> > I personally prefer to keep things in URL query parameters, and > > responses in TW-format DIV's. If we could get rid of the unneccessary > > encoding in TW (\n and \n) the response, as DIV's, would be viewable in > > any browser...even search engines could parse it. (Using <pre> instead > > has been discussed before)
> > > getTiddlerInfo > > > components:title tags fields" > > > condition:"title:[[Some Tiddler Title]]" > > > alternatecondition:"tags:[[some tag]] [[some other > tag]] -[[a tag you > > > don't want]]" > > > which looks different, but basically the idea is that it'd be cool to > > > specify both which tiddlers to return, and what you wanted in the reply. > > > Pretty much all of the time you'd want title tags and fields -- > > > ie. everything-but-the-content, but I reckon we should be looking at a > > > server api to allow for more flexible usage than that, because once > you're > > > on a dynamic serverside, you take away the single-file size restriction > on > > > TiddlyWikis, I reckon you would want to use it for stuff that people > aren't > > > using it for today.
> > My action=get API above is quite primitive. But I see no reason that > > one couldn't do: > > action=get&tag=foo > > but I'm a little wary of forcing any server-side author to implement SQL > > in order to have a compliant server.
> > An API extension to get metadata only might be quite useful here. > > Imagine action=get_metadata, where the response is a list of DIV's (as > > appear in the HTML store) but tagged with 'metadataOnly'. That way TW > > can put all of the tiddlers into the timeline, perform complex queries > > on the tags, but the content would be missing. (This is effectively a > > mechanism for load-on-demand too)
> > > A fairly tame example might be handling "done" projects in a GTD system, > for > > > example. Imagine if you could get metadata info for everything -except- > > > project/action tiddlers marked as "archived". You could use your GTD TW > for > > > years, have thousands of archived projects and tens of thousands of > archived > > > actions, and you wouldn't need to download 100k of archived tiddler > titles > > > every time you opened the TW.
> > Then, how would you access archived tiddlers if you wanted to? In what > > you describe above, "archived" is identical to "deleted".
> > At a very minimum, I think one would need to download the metadata for > > every tiddler. Making some of them load-on-demand would be up to the > > server-side. Making all tiddlers with the tag 'archived' load on > > demand would be a fine idea. Though I suspect the metadata for GTD > > tiddlers is larger than the content. .e.g. "Doctor's appointment" with > > a lot of metadata.
> > > Or having a TW server side that did dynamic generation of tiddlers -- > > > perhaps stats for sports teams that are generated on request, or a > tiddler > > > for every verse in the bible, or whatever. You wouldn't want to > pre-fetch > > > every title for that list of tiddlers. How about a TW interface to > > > Wikipedia.. asking for a tiddler called "Mr T" will send the Wikipedia > > > article to your local TW.
> > Dynamic generation of content is kind of a kludge. The old ZW had it > > and I've effectively disabled it all.
> > I don't have any problem with dynamically generated content, but we > > should not pretend that these are tiddlers. It should somehow be > > obvious that I'm looking at a sports stat page and not a tiddler I can > > edit. If the tiddler contains scripting, should "edit" show the > > server-side script or the results of the server-side script? (What if > > different server-sides use different scripting languages?) How do I > > know that I just edited a tiddler with a server-side script and I have > > to reload it an extra time
Sorry about the double posting since I tried to reply via email and it told me that didn't got through so I replied in the group again.....
Because I wanted to use returned result such as "insert successful", "update successful" etc and don't know if HTTP headers could represent them all. Besides I am thinking if there would be more code added later which HTTP may not allow.
>> 2/ a common function to interpret what is returned (may still be the > case with header)
If we have a common ServerSidePlugin (some evolution of my ZiddlyWikiPlugin, hopefully) then we can choose any mechanism at all, since the code is shared. :) So this isn't an argument for any one mechanism over any other.
In here, I meant to say is a common function to get and return the result within the code rather than common mechanism for various server-side.
>> 3/ ease debugging since header cannot be passed after there are output.
I don't understand this statement. The status code is passed to event handlers in exactly the same way as the response. So, it should be no more difficult to debug status codes than response text. See zw.addTiddler and config.commands.revisions in http://ziddlywiki.com/test2.1#ZiddlyWikiPlugin They're symmetric WRT response (parameter r) and status.
I am thinking of debugging a string in the middle of the operation (thus output). Sending header after output is sent would result in error.
I'll have a look at the REST approach first but I started to lean towards the header approach.
I've realised that my motivation for asking the question originally was mainly because I thought we should try and list the things we'd like a serverside to be able to do. I think that's what Tom Otovos and I had realised -- that potential serverside //user// developers, people working on stuff like TiddlySnip, GTD adaptations, plugin management, etc should try and collaborate a bit more with the serverside //service// developers.
The technical discussion that is going on now is really important, but I think it'd be worthwhile also spending some time talking about plain old requirements. I don't want to see a printed spec in a ring binder or anything, but we should try and roughly agree on the things that should be common to all serversides, the things that would be nice to have, etc.
As in, if a team of a hundred programmers appeared and announced that they were going to spend 6 months writing an all-new server side for us, what would we ask for?
I'll start:
- clear, basic API between client side (eg plugins) and server side advantages: - can mix-and-match servers and plugins - easy to write a plugin to do non-traditional server stuff -- for example, load a new sudoku game or TW page layout from a public server
- support for lazy-loading of tiddler content (eg ability to get tiddler metadata only) advantages: - efficiencies all over - mainly fast loading of TW
- support for "infinitely" large sets of tiddlers advantages: - ability to write TW interfaces to other systems
- server-end searches advantages: - pretty much required, if we want lazy loading - servers can do all sorts of indexing etc to speed things up - maybe even link to external search engines. TWGoogle, anyone?
> I think it'd be worthwhile also spending some time talking about plain > old requirements. I don't want to see a printed spec in a ring binder > or anything, but we should try and roughly agree on the things that > should be common to all serversides, the things that would be nice to > have, etc.
That is exactly the stuff I had in mind when initally replying to this thread.
I am mainly interested in the API and less in protocol things (REST, RPC, SOAP, URL formats). I prefer to look at these things more from the "inside" of TiddlyWiki: what are the (JavaScript) functions TiddlyWiki must call/provide to support ServerSide stuff.
Maybe we could just start with a set of well documented JavaScript function signatures (no implementations required), like an "interface definition" in Java. Any serverside implementation can/must then implement these functions accordingly. I.e. to use a different ServerSide implementation one just needs to swap these functions. E.g. we may come up with a class IBackend for this purpose.
Ideally these functions are functions already provided by TiddlyWiki, but we will probably also need more.
BTW: This was not yet mentioned but I think the API we will come up should be used both for the ServerSide implementations but also for the "Local Filesystem" implementation.
> I've realised that my motivation for asking the question originally > was mainly because I thought we should try and list the things we'd > like a serverside to be able to do. I think that's what Tom Otovos > and I had realised -- that potential serverside //user// developers, > people working on stuff like TiddlySnip, GTD adaptations, plugin > management, etc should try and collaborate a bit more with the > serverside //service// developers.
> The technical discussion that is going on now is really important, but > I think it'd be worthwhile also spending some time talking about plain > old requirements. I don't want to see a printed spec in a ring binder > or anything, but we should try and roughly agree on the things that > should be common to all serversides, the things that would be nice to > have, etc.
> As in, if a team of a hundred programmers appeared and announced that > they were going to spend 6 months writing an all-new server side for > us, what would we ask for?
> I'll start:
> - clear, basic API between client side (eg plugins) and server side > advantages: > - can mix-and-match servers and plugins > - easy to write a plugin to do non-traditional server stuff -- for > example, load a new sudoku game or TW page layout from a public server
> - support for lazy-loading of tiddler content (eg ability to get > tiddler metadata only) > advantages: > - efficiencies all over > - mainly fast loading of TW
> - support for "infinitely" large sets of tiddlers > advantages: > - ability to write TW interfaces to other systems
> - server-end searches > advantages: > - pretty much required, if we want lazy loading > - servers can do all sorts of indexing etc to speed things up > - maybe even link to external search engines. TWGoogle, anyone?
> Maybe we could just start with a set of well documented JavaScript function > signatures (no implementations required), like an "interface definition" in > Java. Any serverside implementation can/must then implement these functions > accordingly. I.e. to use a different ServerSide implementation one just > needs to swap these functions. E.g. we may come up with a class IBackend for > this purpose.
I think we need both a standardised JavaScript API and a standardised HTTP interface. From the perspective of a TW programmer the JS api is obviously important, but it's the HTTP API that's crucial when you're considering integration with other systems (mashups and so on).
> BTW: This was not yet mentioned but I think the API we will come up should > be used both for the ServerSide implementations but also for the "Local > Filesystem" implementation.
This relates to what I was saying about 'static serversides'. I'm advocating supporting static servers that are accessed both through HTTP (ie readonly web access via plain HTTP and read/write access via WebDAV) and through the file:// protocol. In both those scenarios I'd like to be able to support progressive loading of content.
> On 11/7/06, Daniel Baird < danielba...@gmail.com> wrote:
> > I've realised that my motivation for asking the question originally > > was mainly because I thought we should try and list the things we'd > > like a serverside to be able to do. I think that's what Tom Otovos > > and I had realised -- that potential serverside //user// developers, > > people working on stuff like TiddlySnip, GTD adaptations, plugin > > management, etc should try and collaborate a bit more with the > > serverside //service// developers.
> > The technical discussion that is going on now is really important, but > > I think it'd be worthwhile also spending some time talking about plain > > old requirements. I don't want to see a printed spec in a ring binder > > or anything, but we should try and roughly agree on the things that > > should be common to all serversides, the things that would be nice to > > have, etc.
> > As in, if a team of a hundred programmers appeared and announced that > > they were going to spend 6 months writing an all-new server side for > > us, what would we ask for?
> > I'll start:
> > - clear, basic API between client side (eg plugins) and server side > > advantages: > > - can mix-and-match servers and plugins > > - easy to write a plugin to do non-traditional server stuff -- for > > example, load a new sudoku game or TW page layout from a public server
> > - support for lazy-loading of tiddler content (eg ability to get > > tiddler metadata only) > > advantages: > > - efficiencies all over > > - mainly fast loading of TW
> > - support for "infinitely" large sets of tiddlers > > advantages: > > - ability to write TW interfaces to other systems
> > - server-end searches > > advantages: > > - pretty much required, if we want lazy loading > > - servers can do all sorts of indexing etc to speed things up > > - maybe even link to external search engines. TWGoogle, anyone?
> > Comments?
> > -- > > Daniel Baird > > http://tiddlyspot.com (free, effortless TiddlyWiki hosting) > > http://danielbaird.com (TiddlyW;nks! :: Whiteboard Koala :: Blog :: > > Things That Suck)
> I think we need both a standardised JavaScript API and a standardised > HTTP interface.
I absolutely agree.
I was just suggesting to first start with the JavaScript API. Once we have this API the HTTP interface more or less directly derived from that, I think.
> > Maybe we could just start with a set of well documented JavaScript > function > > signatures (no implementations required), like an "interface definition" > in > > Java. Any serverside implementation can/must then implement these > functions > > accordingly. I.e. to use a different ServerSide implementation one just > > needs to swap these functions. E.g. we may come up with a class IBackend > for > > this purpose.
> I think we need both a standardised JavaScript API and a standardised > HTTP interface. From the perspective of a TW programmer the JS api is > obviously important, but it's the HTTP API that's crucial when you're > considering integration with other systems (mashups and so on).
> > BTW: This was not yet mentioned but I think the API we will come up > should > > be used both for the ServerSide implementations but also for the "Local > > Filesystem" implementation.
> This relates to what I was saying about 'static serversides'. I'm > advocating supporting static servers that are accessed both through > HTTP (ie readonly web access via plain HTTP and read/write access via > WebDAV) and through the file:// protocol. In both those scenarios I'd > like to be able to support progressive loading of content.
> > On 11/7/06, Daniel Baird < danielba...@gmail.com> wrote:
> > > I've realised that my motivation for asking the question originally > > > was mainly because I thought we should try and list the things we'd > > > like a serverside to be able to do. I think that's what Tom Otovos > > > and I had realised -- that potential serverside //user// developers, > > > people working on stuff like TiddlySnip, GTD adaptations, plugin > > > management, etc should try and collaborate a bit more with the > > > serverside //service// developers.
> > > The technical discussion that is going on now is really important, but > > > I think it'd be worthwhile also spending some time talking about plain > > > old requirements. I don't want to see a printed spec in a ring binder > > > or anything, but we should try and roughly agree on the things that > > > should be common to all serversides, the things that would be nice to > > > have, etc.
> > > As in, if a team of a hundred programmers appeared and announced that > > > they were going to spend 6 months writing an all-new server side for > > > us, what would we ask for?
> > > I'll start:
> > > - clear, basic API between client side (eg plugins) and server side > > > advantages: > > > - can mix-and-match servers and plugins > > > - easy to write a plugin to do non-traditional server stuff -- for > > > example, load a new sudoku game or TW page layout from a public server
> > > - support for lazy-loading of tiddler content (eg ability to get > > > tiddler metadata only) > > > advantages: > > > - efficiencies all over > > > - mainly fast loading of TW
> > > - support for "infinitely" large sets of tiddlers > > > advantages: > > > - ability to write TW interfaces to other systems
> > > - server-end searches > > > advantages: > > > - pretty much required, if we want lazy loading > > > - servers can do all sorts of indexing etc to speed things up > > > - maybe even link to external search engines. TWGoogle, anyone?
> > > Comments?
> > > -- > > > Daniel Baird > > > http://tiddlyspot.com (free, effortless TiddlyWiki hosting) > > > http://danielbaird.com (TiddlyW;nks! :: Whiteboard Koala :: Blog :: > > > Things That Suck)
To stay on topic, I don't think it is necessary to have a separate server-side development group but it would be nice to have so they are all "group" as a list.
To me, the first thing I wanted to agreed on is the format transfer between standalone TW and server-side. e.g. to standardise format such as saving should pass on a tiddler div, delete should pass on a tiddler title etc. this is the most important thing than having standard server-side plugin (like having standard HTML rather than browser?).
I believe I am most interested in load-on-demand (lazy/progressive loading?) for large TW, efficient. speed.
PS: Jeremy, is it possible to have a setValue function that allows to flag whether the value is temporary (not saved) or not?
> To stay on topic, I don't think it is necessary to have a separate > server-side development group but it would be nice to have so they are > all "group" as a list.
> To me, the first thing I wanted to agreed on is the format transfer > between standalone TW and server-side. e.g. to standardise format such > as saving should pass on a tiddler div, delete should pass on a tiddler > title etc. this is the most important thing than having standard > server-side plugin (like having standard HTML rather than browser?).
> I believe I am most interested in load-on-demand (lazy/progressive > loading?) for large TW, efficient. speed.
> PS: Jeremy, is it possible to have a setValue function that allows to > flag whether the value is temporary (not saved) or not?
After re-reading the post, I want to make some comments again.
Regarding TW having server-side built-in, I believe it would be better to have a separate plugin that could hijack the required functions and make required queries to servers. This would help reduce TW's file size which I am slightly worried.
I am more favored on having server-side plugin working transparently, such that all plugins available currently in standalone TW would also work for server-side. Searches could be hijack with a separate server-side call.
I don't mind marrying the server-side API, in fact this is what I've planned. However there are cases that both server-side works differently (such as ZW have locks for protection, ccT checks for previous modified date when saving) and it could be difficult in some cases to merge.
I also fancy the idea of having a static server. I have also think of cases where one host their TW on googlepage and their PHP backend on a separate server to reduce bandwidth (shhh, don't let google know). A proxy would be required for this which I hope to implement later (let me know if there is one already available).
I have read through the REST protocol but the only thing top my mind is that they use stateless transfer and stick to HTTP protocol. I am quite happy with these and would try to stick with them. There are things I have missed out I am sure and I have to blame on my lack of understand in this area.
Sorry to be a little late to this party, but I would like to chime in on a couple of things.
First, I think the details of REST vs. whatever is important, but not nearly so (initially) as the interface to TW. I strongly favour TW abstractions that make server-side or local-only operation a choice made (possibly) by the existence of appropriate plugins. The main reason for this is that I think storage of the TW data should be largely transparent to TW itself, in order that core TW development can proceed undisturbed. There are some abstractions already there in 2.1, but I have not double-clicked on it enough to decide if it is enough. So stated another way, there shouldn't be a "Server Side TW" and "Traditional TW", just TW.
Second, I favour keeping this discussion in TiddlyWikiDev because it allows some people to peripherally participate without necessarily subscribing to a new group. I watch too many groups as it is, and it is easier to just flag certain discussion threads here.
Lastly, I strongly agree with Daniel. Let's come to some form of agreement on what this thing needs to do to meet all our needs (not just us GTD guys), 80-20 the list if it is too big, then move on to the technical details. To that end, let me float this idea: what about scheduling a Skype meeting where we can talk "face to face" and accelerate the requirements gathering?
I agree with Tomo and friends. Let's keep it here, and let's keep server side stuff in plugins. I'm just completing a rather unusual server side application of TW - an astronomical observatory control system. It runs from a server, with ASP files driving AJAX displays and forms in tiddlers. There's much more, and I'll be posting something in the near future on all of this.
The bottom line is, I have done everything using plugins, and I haven't run into any brick walls, despite pushing the TW technology to the edge of the envelope and beyond... the only issue I needed to deal with is the storage of options in separate cookies, I ran out really fast. I had to hijack the core to put options as 'crumbs' into a single cookie. I did that with plugins too! Ugly ones, but nonetheless... MarkupPostHead is great!
I really liked the idea proposed earlier in this thread that storing and fetching should just be storing and fetching, and local disk, TiddlySaver, remote server, etc should all look the same from the core code's point of view.
And, I'm completely happy to keep discussions in this group. I wasn't exactly proposing a new Google Group specifically, more thinking about asking people to put their hands up as interested parties, and maybe note somewhere what areas they're interested in.
Then if someone gets to the point of, say, having a bunch of function signatures to show around, they can make sure a representative of each stakeholder area gets to have a say -- core developers, application developers, server devs working with PHP / Java / Python / Zope / Ruby etc etc.
;D
On 11/10/06, Bob Denny <dc3de...@gmail.com> wrote:
> I agree with Tomo and friends. Let's keep it here, and let's keep > server side stuff in plugins. I'm just completing a rather unusual > server side application of TW - an astronomical observatory control > system. It runs from a server, with ASP files driving AJAX displays and > forms in tiddlers. There's much more, and I'll be posting something in > the near future on all of this.
> The bottom line is, I have done everything using plugins, and I haven't > run into any brick walls, despite pushing the TW technology to the edge > of the envelope and beyond... the only issue I needed to deal with is > the storage of options in separate cookies, I ran out really fast. I > had to hijack the core to put options as 'crumbs' into a single cookie. > I did that with plugins too! Ugly ones, but nonetheless... > MarkupPostHead is great!
GData uses REST to provide read/write semantics for XML resources (specifically Atom and RSS). Also, the optimistic concurrency support should help with multi-user scenarios.
