Plugin Library: API concept
FND
now that the first prototype of the plugin library's backend is done,
it's time to think about exposing the aggregated data.
(To get an idea of what data we're gathering, take a look at the
database diagram*.)
There are two obvious options for accessing that data:
* a web interface for browsing and searching
* direct integration with TiddlyWiki (e.g. via the importing wizard)
No matter which option we choose to implement first, both require an API
to retrieve the plugins from the database.
Here are some early thoughts on that API:
* REST exposure; everything's accessible via a URL
* results might be returned as JSON or pseudo-HTML (TW store format)
** returned are complete plugin tiddlers - any further processing should
be performed client-side ("stupid server")
* assuming a base URL like http://plugins.tiddlywiki.org/api/ ...
** /titles/%s - returns plugin %s (potential problem: would return more
than a single item if several plugins carry the same name)
** /categories/%s/ - returns all plugins of category %s
** /tags/%s/ - returns all plugins carrying the tag(s) %s
** /ratings/%s/ - returns all plugins with rating %s
** ?%f=%s - filters results by attribute %f (e.g. author) and value %s
(this might be dropped, as it goes against the stupid-server principle)
This concept seems incomplete, even inconsistent to some extent - so I'd
greatly appreciate getting some feedback!
-- F.
* http://tinyurl.com/3mtnwn
(http://svn.tiddlywiki.org/Trunk/contributors/FND/tools/pluginLibrary/docs/databaseDesign.txt)
paul.s...@gmail.com
> There are two obvious options for accessing that data:
> * a web interface for browsing and searching
> * direct integration with TiddlyWiki (e.g. via the importing wizard)
> No matter which option we choose to implement first, both require an API
> to retrieve the plugins from the database.
adaptor, no?
>
> Here are some early thoughts on that API:
> * REST exposure; everything's accessible via a URL
> * results might be returned as JSON or pseudo-HTML (TW store format)
> ** returned are complete plugin tiddlers - any further processing should
> be performed client-side ("stupid server")
> * assuming a base URL likehttp://plugins.tiddlywiki.org/api/...
> ** /titles/%s - returns plugin %s (potential problem: would return more
> than a single item if several plugins carry the same name)
> ** /categories/%s/ - returns all plugins of category %s
> ** /tags/%s/ - returns all plugins carrying the tag(s) %s
> ** /ratings/%s/ - returns all plugins with rating %s
> ** ?%f=%s - filters results by attribute %f (e.g. author) and value %s
> (this might be dropped, as it goes against the stupid-server principle)
resource
to the other without having to understand the naming scheme. In other
words, use URIs within the results to point to other resources, i.e.
/tags returns a list of URIs rather than 'ids' which are then have to
be plugged into the %s of another URI..
> This concept seems incomplete, even inconsistent to some extent - so I'd
> greatly appreciate getting some feedback!
TiddlyWeb
and getting Chris to help add any extensions you need.
Paul
--
http://blog.whatfettle.com
BidiX
First, I've to say that your post found me in documenting the RESTful
API for TiddlyHome 2.0 [1] (For now TiddlyHome 2.0 is a work in
progress, a pre-beta version is still available at http://TiddlyHome.appspot.com/).
For this I implemented an abstract model in Google Apps based on
WebApp extension [2]. I'll try to share with you in your Plugin
Directory context what I'm experimenting.
So please take this as a constructive contribution.
On Jun 13, 1:34 pm, FND <F...@gmx.net> wrote:
> There are two obvious options for accessing that data:
> * a web interface for browsing and searching
> * direct integration with TiddlyWiki (e.g. via the importing wizard)
> No matter which option we choose to implement first, both require an API
> to retrieve the plugins from the database.
first goal for the API is to be called from whatever application. The
browser and the HTML interface is only a limited use of your
application. What about an application written in whatever language
(python for example) that GET a plugin via a "fetchurl" API to do
something (code reformatting, optimization, cooking etc...).
>
> Here are some early thoughts on that API:
> * REST exposure; everything's accessible via a URL
think you have two resources in your application :
* Plugin: /plugins/<plugin_id>
* Repository: /repositories/<repository_id>
may be also Author ?
Your API should also provides functions to create, replace, update,
delete a Plugin, a Repository and certainly a Category etc ... For
that you should certainly use GET, POST, PUT, DELETE in some way.
> * results might be returned as JSON or pseudo-HTML (TW store format)
> ** returned are complete plugin tiddlers - any further processing should
> be performed client-side ("stupid server")
type: text/html, text/plain ... Suffix like in TiddlyWeb
(.html, .tw, .xml, .js) is certainly a good direction.
> * assuming a base URL likehttp://plugins.tiddlywiki.org/api/...
> ** /titles/%s - returns plugin %s (potential problem: would return more
> than a single item if several plugins carry the same name)
> ** /categories/%s/ - returns all plugins of category %s
> ** /tags/%s/ - returns all plugins carrying the tag(s) %s
> ** /ratings/%s/ - returns all plugins with rating %s
> ** ?%f=%s - filters results by attribute %f (e.g. author) and value %s
> (this might be dropped, as it goes against the stupid-server principle)
>
repository-author like
/authors/Eric/categories/Formatting/plugins or /authors/Eric/
categories/Formatting/Bookmarklets
> This concept seems incomplete, even inconsistent to some extent - so I'd
> greatly appreciate getting some feedback!
>
> -- F.
>
> *http://tinyurl.com/3mtnwn
Hope That Helps,
-- BidiX
http://BidiX.info/
[1] http://tinyurl.com/5gx3bh (http://bidix.appspot.com/BidiX/
namespaces/TiddlyHome/tiddlers/TiddlyHome%20Restful%20API.html)
[2] http://tinyurl.com/4regwn (http://bidix.appspot.com/BidiX/
namespaces/TiddlyHome/tiddlers/A%20Model%20for%20a%20RESTful
%20API.html)
Martin Budden
I think that before you spend much time on the API you need to ensure
that the backend is working. To do that I would make a first cut a web
page that just lists all the plugins held in the library and their
metadata. I'd just write some php that directly queries the database
and displays the information, and not expose the API at all. Once we
have confidence that the backend is working and that the database
design is robust we can then think about exposing an API.
Martin
2008/6/13 FND <FN...@gmx.net>:
FND
for me to process this, but I'll report any progress here.
> Once we have confidence that the backend is working and that the
> database design is robust we can then think about exposing an API.
Well, I did of course check the results (both internal objects and
database contents) during development. Based on that, as well as on the
extensive discussions we had at Osmosoft Towers, I'm reasonably
confident that there won't be any significant changes to the backend
structure anymore.
However, I have yet to do extensive testing (ideally including unit
tests), and I'm still waiting for someone to review the code (which
would be quite important).
Note: I've decided that SVN support is secondary for now (i.e. I'm only
processing TiddlyWiki documents) - getting a basic prototype up and
running has priority at the moment.
-- F.
FND
I've been thinking the same thing; when I was writing up the API
concept, I realized that TiddlyWeb does pretty much exactly that already.
However, using TiddlyWeb isn't a panacea.
My main concern is figuring out which queries are gonna be relevant,
which has implications for scaling. For example, I don't think "all
plugins by author x" will be a frequent query, whereas full-text search
is probably gonna be quite popular.
So I guess we won't be able to do without the custom query string,
except if we always serve *all* plugins, thus moving all filtering to
the client. But having to transfer that much data at once might be quite
an annoyance.
> using an existing format, such as tiddlers, will save writing an
> adaptor, no?
I suppose returning tiddlers in (pure-)store format should make it
possible to use the existing FileAdaptor, as it's essentially the same
as accessing a static TW document somewhere...
>> * results might be returned as JSON or pseudo-HTML (TW store format)
>
> subject to content-negotiation, or supplying a .json, .html extension?
I quite like the idea of using extensions here (even though we'll
probably end up - or at least start - with only a single format).
-- F.
FND
> API for TiddlyHome 2.0
Nice coincidence - I'm glad we can benefit from your having pondered
these issues already... (Those TiddlyHome Restful API pages were quite
insightful.)
> please take this as a constructive contribution.
I happily do! (I'm not hiding the fact that I'm an ignorant fool, so
constructive criticism is very much welcome.)
> You should be more generic. An API is "Programming Interface", the
> first goal for the API is to be called from whatever application.
Sure - but I can expect the respective application to be able to request
a URL, no? And the return format - whether JSON or (X)HTML - should be
pretty easy to process with almost any modern language.
> I think you have two resources in your application :
> * Plugin: /plugins/<plugin_id>
> * Repository: /repositories/<repository_id>
> may be also Author ?
Actually, I think we only have plugins as basic resources - everything
else is just a filter/bag/selection/... of those.
> Your API should also provides functions to create, replace, update,
> delete a Plugin, a Repository and certainly a Category etc ... For
> that you should certainly use GET, POST, PUT, DELETE in some way.
Well, the plugin data is aggregated using a PHP script (via cron). I
don't think that any such data should be modified by other applications.
However, there are supposed to be "community features" like commenting,
rating and tagging at some point...
Will have to look into the details of implementing GET, POST, PUT,
DELETE in this context.
> You certainly need also to explain some relations between plugin-
> repository-author like
> /authors/Eric/categories/Formatting/plugins or /authors/Eric/
> categories/Formatting/Bookmarklets
I've been thinking about this as well - but I'm not sure whether
combining selectors like that is semantically correct
("/foo/lorem/bar/ipsum" is essentially the same as
"?foo=lorem&bar=ipsum", and is thus identical to "/bar/ipsum/foo/lorem").
I might be missing the point here though (see above... ).
-- F.
chris...@gmail.com
On Jun 17, 10:37 am, FND <F...@gmx.net> wrote:
> > You certainly need also to explain some relations between plugin-
> > repository-author like
> > /authors/Eric/categories/Formatting/plugins or /authors/Eric/
> > categories/Formatting/Bookmarklets
>
> I've been thinking about this as well - but I'm not sure whether
> combining selectors like that is semantically correct
> ("/foo/lorem/bar/ipsum" is essentially the same as
> "?foo=lorem&bar=ipsum", and is thus identical to "/bar/ipsum/foo/lorem").
> I might be missing the point here though (see above... ).
represent a hierarchical path. Practically speaking humans tend to
want it to. Therefore what I tend to (try to) do in URL design is come
up with a feasible set of hierarchies that allow a curious person to
browse the resources that are available. So while /foo/lorem/bar/ipsum
etc are all the same thing, they don't encourage the same use.
To do this it is important to think about what the primary resources
or entities are in your system and be explicit about distinguishing
those entities from information that could be considered attributes of
those entities. One way to tell what is an attribute (in this context)
is that attributes are usually harder to fit in a hierarchy.
Things that don't fit you may wish to put in a query string, as a sort
of filter on the results at the current location.
In the plugin library situation you have a bit of a pickle, I think,
because you really want to enable serendipitous browsing of the
tiddlers such that people stumble onto the good stuff. You've got your
primary entity of the plugin/tiddler but generally speaking people
won't be looking via the primary entity. Instead they'll be looking by
attributes and once they find things they like, before they do a
download, the thing that matters is the container of the target: the
thing which has information about the entity (where you can read and
leave comments etc).
This suggests that while your _web service_ may have a primary entity
of a tiddler, your _web application_, the library of plugins, may want
for a different focus. If you want to encourage browsability, then you
want several browsable entry points that eventually lead to the
tiddler. Perhaps things like /authors/{author}/plugins and /categories/
{category}/plugins and date related paths.
Here's a crazy idea: Have you considered using a blog engine, one that
supports tags, as the presentation engine for the plugin store? Each
plugin gets an automatically created blog entry, including a link to
the canonical store (perhaps a bag in tiddlyweb?). You get tag and
date browsing, comments and search for free.
FND
also added to my confusion in terms of the actual implementation... )
> To do this it is important to think about what the primary resources
> or entities are in your system and be explicit about distinguishing
> those entities from information that could be considered attributes
That's my problem - pretty much everything (categories, tags, authors
etc.) is really an attribute here.
> This suggests that while your _web service_ may have a primary entity
> of a tiddler, your _web application_, the library of plugins, may want
> for a different focus. If you want to encourage browsability, then you
> want several browsable entry points that eventually lead to the
> tiddler.
Sounds reasonable.
However, I'm not sure there are clearly identifiable entry points,
condering many users probably won't even have a vague idea of what
they're looking for.
How about using tags as the most basic entity for grouping/filtering:
/tags/foo/bar/baz/
This would return all plugins with tags "foo" AND "bar" AND "baz".
Categories, then, could simply be aliases for certain tag combinations:
/categories/qux/
/tags/foo/bar/baz/
In addition, we'd of course have this:
all plugins: /
specific plugin: /plugins/<name>.<format>
(As mentioned before, using the name as plugin identifier is a bit
dangerous because it's not necessarily unique - but I still prefer that
to using some arbitrary, opaque internal ID.)
Should we also allow combining several arbitrary plugins like so:
/plugins/<name>.<format> <separator> <name>.<format>
> Here's a crazy idea: Have you considered using a blog engine
I hadn't considered that - but it's not a bad idea.
However, it's probably not gonna work out - pros and cons:
+ built-in categorizing
+ built-in commenting (possibly also rating, or even user tagging)
(though we want this for TW anyway - I believe it's on Simon's list)
- limited flexibility - think custom attributes or structured content
(e.g. fields, slices)
- extra processing required to serve plugins ready for importing
- all blog engines are crap (we might as well use MediaWiki... )
- TiddlyWiki seems suitable for and capable of doing this - so why put
effort into external platforms
-- F.