Hypermedia API and Collection+JSON in web2py

2,527 views
Skip to first unread message

Massimo Di Pierro

unread,
Jun 22, 2014, 4:45:06 PM6/22/14
to web...@googlegroups.com
I added Hypermedia API support to web2py using Collection+JSON. Experimental.
Collection+JSON is a standard for self documenting RESTful API.

Example
=========

Let's say you have a model:
 
    db.define_table('thing',Field('name'))

and in controller default.py you add

    def api():
         from gluon.contrib.hypermedia import Collection
         rules = {
                'thing': {
                    'GET':{'query':None,'fields':['id', 'name']},
                    'POST':{'query':None,'fields':['name']},
                    'PUT':{'query':None,'fields':['name']},
                    'DELETE':{'query':None},
               }}
        return Collection(db).process(request,response,rules)

And now by magic your table "thing" is fully exposed using the Collection+JSON API. The API is self documenting and supports GET, POST, PUT, DELETE, etc.

For example you can do things like:

The API are completely self documenting as explained here http://amundsen.com/media-types/collection/

It is customizable
==============

       rules = {
                'thing': {
                    'GET':{'query':None,'fields':['id', 'name']},
                    'POST':{'query':None,'fields':['name']},
                    'PUT':{'query':None,'fields':['name']},
                    'DELETE':{'query':None},
               }}

Let you specify which tables are exposed, which methods are available and which fields are exposed for each method. The query property lets you specify optional filters for example  { 'query':db.thing.name.startswith('A'),....} will only exposed things starting with letter A. Fields can be conditional and different for different users or for the same user in different stages of a workflow (the communication is stateless, but the server is not).

Supports complex queries
=====================
http:/...../{table}
http:/...../{table}/{id}
http:/...../{table}?{field}=value
http:/...../{table}?{field}.gt=value # field>value
http:/...../{table}?{field}.le=value # field<=value
...
http:/...../{table}?_orderby={field}
http:/...../{table}?_limitby=value
http:/...../{table}?_offset=value
...
and combinations there of. They are mapped directly into DAL queries. More examples are in the API response itself.

The bigger picture
===============

This API provide enough information to generate forms and tables and grid completely client side. Recently we stumbled against the problem of moving from Bootstrap 2 to Bootstrap 3 because so much of the form and grid logic is server side. My plan is to move most of the logic in the JS library and allow users to customize them  for different CSS frameworks.

Eventually (in dreams) I would like to have a very slim framework based on bottle+dal+validators+auth+collection and have client side only templates (based on jquery, sugar, and ractive.js) that can generate forms and grids based the collection API. This framework could ship with web2py and allow you to program using same web interface that we all love. There are many design decisions to make to get there. Your suggestions are welcome.

How can you help?
===============
1) test it.
2) there are many existing client side tools for Collection+JSON. Try them with web2py.
3) blog about it and suggest improvements.


I said, it is experimental
===================

Collection+JSON has limits:
- it is very verbose JSON. This is my my implementation has  compact=True option that breaks the protocol but makes much smaller JSON messages.
- it does not convey field type information and constraints. This is why I extended to do so but more work is needed because DAL types do not map into HTML5 input types (this of list:string or list:reference).

More extensions of the protocol are required. Extensions are allowed. Yet they may change the API in the near future.

Massimo

samuel bonill

unread,
Jun 22, 2014, 7:20:36 PM6/22/14
to web...@googlegroups.com
it's good, I use db.parse_as_rest for generate the representation of resources, Collection+JSON help much.

other thing, I would like generate a resource with relationship .... for example


patterns = [ ("posts/{post.id}", {"coments": {"author": "auth_user"}}]  # My implementation
....
parser = db.parse_as_rest(patterns, args, kwargs)

GET: http://example.com/default/index/posts/1.json

{
    "content": [{
        "id": "1",
        "title": "web2py",
        "text": "testing ...",
        "picture": "http://web2py.s3.amazonaws.com/post/my_s3_file.jpg",
        "coments": [{
"id": 1,
 "text": "my coment", "author": {"id":3685, "name": "Lebron james" ...},
}, ... ]
}]
 }

the problem with the traditional RESTful apps on web2py is that for get the information
of our restful example planted before, you need do three request to the api, like this case.

patterns = [ "post/{post.id}",
"post/{post.id}/coments[coments]/{coments.id}",
"post/{post.id}/coments[coments]/{coments.id}/author[auth_user]"
]
....
parser = db.parse_as_rest(patterns, args, kwargs)

#1 GET: http://example.com/default/index/post/1.json
#2 GET: http://example.com/default/index/post/1/coments/1.json
#3 GET: http://example.com/default/index/post/1/coments/1/author.json

sorry my english... regards

Massimo Di Pierro

unread,
Jun 23, 2014, 1:01:27 AM6/23/14
to web...@googlegroups.com
This is a automatic in Colleciton+JSON. For example:

# assume a model that described things and their attributes
db.define_table('thing',Field('name'))
db.define_table('attr',Field('thing','reference thing'),Field('name'))

and you expose both:

def api():
    from gluon.contrib.hypermedia import Collection
    rules = {
        'thing': {
            'GET':{'query':None,'fields':['id', 'name']},
            'POST':{'query':None,'fields':['name']},
            'PUT':{'query':None,'fields':['name']},
            'DELETE':{'query':None},
            },
        'attr': {
            'GET':{'query':None,'fields':['id', 'name', 'thing']},
            'POST':{'query':None,'fields':['name', 'thing']},
            'PUT':{'query':None,'fields':['name', 'thing']},
            'DELETE':{'query':None},
            },
        }
    return Collection(db).process(request,response,rules)

{"collection": {"version": "1.0", "href": "/super/collections/api/thing", "items": [{"href": "http://127.0.0.1:8000/super/collections/api/thing/1/chair", "data": [{"prompt": "Id", "name": "id", "value": 1}, {"prompt": "Name", "name": "name", "value": "Chair"}], "links": [{"href": "http://127.0.0.1:8000/super/collections/api/attr?thing=1",.....

The links field tells you how to get the attributes of the thing Chair.

For many2many and links to images you have to do a little bit more programming. For example:

Given:
db.define_table('thing',Field('name'),Field('image','upload'))

You can define:
rules = {'thing':{'GET':{'query':None, 'fields':None, 'links':{'picture':lambda row: URL('download',args=row.image,scheme=True)}}}}

Problem is that collection+JSON does not say anything about image uploads (POST) and does not say anything about many2many relations.

Massimo

Massimo Di Pierro

unread,
Jun 23, 2014, 1:14:17 AM6/23/14
to web...@googlegroups.com
Let me add that Collection+JSON is a standard but that does not make it perfect. I find many limitations. I managed to overcome some but utilizing extensions.
I also made some of my own extensions. Extensions are allowed and compatible.

Yet it needs more extensions. I did not push this too much because I did not want to depart too much much from the standard.

Collection+JSON is also very verbose. I made a Collection(compact=True) option that makes it much less verbose but will break the specs.

samuel bonill

unread,
Jun 23, 2014, 10:50:32 AM6/23/14
to web...@googlegroups.com
Thanks massimo, this is the point, generate many2many relations, with one request get all the information relations to a resource.  I'm developing a plugin to generate great RESTful API based in many2many relations. inspired by the  Instagram API.

for example :


GET: http://example.com/default/index/posts/1.json


{
    "data": [{

        "id": "1",
        "title": "web2py",
        "text": "testing ...",
        "picture": "http://web2py.s3.amazonaws.com/post/my_s3_file.jpg",
        "coments": [{
"id": 1,
 "text": "my coment", "author": {"id":3685, "name": "Lebron james" ...},
}, ... ]
}]
 
}

I'm trying do this possible, this is my implementation :



--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/WN9yzLIfi6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

samuel bonill

unread,
Jun 23, 2014, 10:57:38 AM6/23/14
to web...@googlegroups.com
patterns = [ ("posts/{post.id}", {"coments": {"author": "auth_user"}}]  # My implementation 
....

parser = Rest.generate(patterns, args, kwargs)

This automatically generate, the post with comments associate and user associated to a comment...

What do you think about my idea ?

Cliff Kachinske

unread,
Jun 25, 2014, 12:58:01 PM6/25/14
to web...@googlegroups.com
Massimo,

I like the way your thoughts are headed.

doesn't the many2many issue goes away if one uses, for example, MongoDB?

Massimo Di Pierro

unread,
Jun 27, 2014, 1:57:14 AM6/27/14
to web...@googlegroups.com
No. At least not in the way web2py uses it.

Falko Webpgr

unread,
Jun 27, 2014, 9:59:01 AM6/27/14
to web...@googlegroups.com
So I created a REST API doc plugin for web2py quite a while ago. Here is what it looks like on the user side

I also posted the plugin for this on here, a while ago. One of the goals back then was to automatically create a WSDL2.0 or WADL file from it, but the student I set to work on it unfortunately never finished it. 

It is very nice to see that something is happening in this direction. Do I understand this right that Collection+JSON is similar to a WSDL file?
The problem with WSDL I always saw was that you could not document the output JSON files. I also still do not know if there is a Format like DTD or XML Schema for JSON. I think this is one of the major obstacles why JSON is not adapted in bio-science.

Cliff Kachinske

unread,
Aug 13, 2014, 5:32:46 PM8/13/14
to web...@googlegroups.com
Massimo,

The new recipe you propose is "bottle+dal+validators+auth+collection"

Could it be "bottle+<whatever_you_like>+validators+auth+collection?  

I have found myself needing to execute complex SQL queries. In this situation, in my opinion, psycopg2 would serve me just as well as dal.


On Sunday, June 22, 2014 4:45:06 PM UTC-4, Massimo Di Pierro wrote:

Massimo Di Pierro

unread,
Aug 14, 2014, 1:12:30 PM8/14/14
to web...@googlegroups.com
I think if you require the dal from the equation, there is almost no overlap any more with web2py.
Reply all
Reply to author
Forward
0 new messages