new framework for json apis

[Gregg Caines]

Hey all... I've been working on a new framework for restful json apis for a while, and was wondering if anyone might be interested in taking a look, giving feedback, etc.

http://percolatorjs.com 

It's definitely a bit weird, but of course I think it's the best way to create JSON APIs.  I didn't really have any desire to duplicate what's already been done.

It's also not production ready ( because I'm not using it in production myself for another month or so), so I'm not really trying to spread it far and wide at this point, but I'd love to hear any comments/criticism/questions from the big brains here about what I have so far.

Thanks,

G

[Scott Elcomb]

On Sun, Dec 16, 2012 at 1:38 PM, Gregg Caines <cai...@gmail.com> wrote:
> Hey all... I've been working on a new framework for restful json apis for a
> while, and was wondering if anyone might be interested in taking a look,
> giving feedback, etc.
>
> http://percolatorjs.com

Is it fair to assume that percolatorjs is an implementation of your
application/vnd.hyper+JSON draft at
<http://caines.ca/blog/programming/hyperjson-a-first-draft/> ?

--
Scott Elcomb
@psema4 on Twitter / Identi.ca / Github & more

Atomic OS: Self Contained Microsystems
http://code.google.com/p/atomos/

Member of the Pirate Party of Canada
http://www.pirateparty.ca/

[Jonathan Chayce Dickinson]

Good work! Vannevar Bush would be proud of you :).

A client library to consume this would be nice...

Jonathan

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Tiger Nassau Inc]

Does not seem to use standard REST calls - that is a no start - needs to be as simple as php slim, then would be really nice
Sent from my LG Mobile

[Gregg Caines]

Kinda-sorta-almost.  That draft makes it look more complicated than it is though.  And nothing forces you to follow that format either.  You can still make old-school arbitrary json apis like twitter and facebook do, if you really want to.

[Gregg Caines]

I assume you mean the standard HTTP methods?  I agree... That would indeed be a no-starter if I'd left those out.  :)

Check the docs at http://percolatorjs.com/documentation.html#server-route .  They show the use of multiple methods on one resource.  You can add others as well as you'd no doubt expect.  

This is different from express/sinatra in that I try to promote keeping every method of a single route together in a single place.  This makes for better code organization and re-use, and lets the framework handle some HTTP errors more correctly.

I still need to document better and make this stuff more obvious, so thanks for the feedback.

G

[Gregg Caines]

Totally agree, but I'm not even done the server-side yet. :)  I'm not even entirely sure what the client library should do yet (beyond the standard http stuff like request or super-agent do), and the only consumers of the API I've written with it are in ruby and erlang, so I'm even a little language agnostic for client libs.

[Jake Verbaten]

Using `onX(cb)` instead of `.on("x", cb)` is going against the grain of node for no reason.

I recommend you change that api.

There doesn't seem to be any support for streams which sucks.

And other then the Hyper+JSON and JSONmodule thing there doesn't seem to be something here that isn't covered by other libraries.

Which means I recommend you take those two concepts and make them seperate open source libraries

[Gregg Caines]

Ohh good questions...

> There doesn't seem to be any support for streams which sucks.

Check out http://percolatorjs.com/documentation.html#context-req and http://percolatorjs.com/documentation.html#context-res .  Those are the unaltered request and response objects, so you can still stream as well you ever could from the node standard library.  There's no synchronous middleware system like connect here either, so you basically get the stream as soon as node does too.

> Using `onX(cb)` instead of `.on("x", cb)` is going against the grain of node for no reason.

I used the word 'on' in onRequest(), onJson() and onBody(), but those are not really for streams or even general events and should only ever have 1 handler/listener, so I didn't feel the need to try to emulate the streams or events API.  Let me know if you think they're more 'eventy' than I do though.  I could certainly name them setRequestPreprocessor(), withJson(), and withBody() too if that leads to less confusion with event/stream APIs.  I'll have to throw this around in my brain a bit.

> And other then the Hyper+JSON and JSONmodule thing there doesn't seem to be something here that isn't covered by other libraries.

I'm going to go out on a limb and assume you're basically saying this could've been implemented on top of express.  That's not the case though once you get into the details:  

* Percolator comes with completely different routing that lets you build re-usable components (like that JSONModule).  Building that on top of express (where two methods of the same resource know nothing of one another) would have been gnarly and not really got me much more than node's standard http server does.  The standard lib does a heck of a lot.  

* I didn't care to have a synchronous middleware system like connect, because I've yet to find a case for a middleware system like that that can't be handled by a 1-liner lazily, and on-demand ($.onJson() is a good example of that -- 80% of my requests never need a bodyParser, and the ones that do get it with 1 line).  

* I didn't want to have any dynamic html generating functionality at all (though it's easily added to an app).  If your app is more website-y and not a single-page-app-y, then express is simply the better choice.  If you have a single page app with static assets backed by an API, then you're in Percolator's sweet spot.  I generally think PJAX should die in a fire though, so I don't care to contribute to it.  I said this was an opinionated framework, right?  :)

There are a whole lot of things outside of json apis that express can do that Percolator will never do either, so I'd never try to compare this to express in general (express is better at basically everything else, even in my biased opinion).  Percolator was more inspired by webmachine ( https://github.com/basho/webmachine/wiki ) and its use-cases than sinatra though, so it naturally doesn't use express.

Thanks for the input!

[Jake Verbaten]

> I used the word 'on' in onRequest(), onJson() and onBody(), but those are not really for streams or even general events and should only ever have 1 handler/listene

req.on("end", handler) is an event that only happens once and and will only have a single handler in most cases. Just make request, json and body events and make your $ thing an event emitter.

> assume you're basically saying this could've been implemented on top of express.

Hell no. You can implement it on top of http. 

CRUDCollection (its annoying to rename your concepts a day after you publish your ideas on the mailing list) could be something returns a (req, res) function I can pass to any router I want. For example

var module = CRUDCollection({
  ...
})

arbitaryRouter.addRoute("/myCollection", module.handler)

arbitaryRouter.addRoute("/myCollection/:memberid", module.wildcard)

> I didn't care to have a synchronous middleware system 

Agreed middleware systems are poor.

What's im trying to suggest is don't write a JSON framework. Write modules that make writing a JSON API server easier. Then later build whatever opinionated framework you want on top of it.

I've listed a few of those modules ( https://github.com/Raynos/routil#routil ), [npm-www](https://github.com/isaacs/npm-www) demonstrates a bunch more of those modules.

[Gregg Caines]

> req.on("end", handler) is an event that only happens once and and will only have a single handler in most cases. Just make request, json and body events and make your $ thing an event emitter.

I think you've sold me on that.  Good points.

> CRUDCollection (its annoying to rename your concepts a day after you publish your ideas on the mailing list) could be something returns a (req, res) function I can pass to any router I want. For example

Sorry. It was actually renamed before I published to the list and I missed updating the one spot in the docs that you found.  You helped me find a documentation bug if it makes you feel any better. :) 

> What's im trying to suggest is don't write a JSON framework. Write modules that make writing a JSON API server easier. Then later build whatever opinionated framework you want on top of it.

Alright so there are a couple of problems that I ran into with this approach mainly revolving around modularity.  Percolator lets you create node modules out of the objects that handle the requests for methods of a single 'resource'.  These modules need more than req and res though if they're going to share functionality server-wide.  Functionality like error handling, server configuration information, inevitable global variables, etc.  There are a number of aspects that can't just be require()ed in and must be passed at runtime.  I'd considered a third parameter for that stuff, or monkey-patching req and res like express does, but neither seemed very pure from a node perspective, so I went with what would be easiest on the app developer, and would ultimately make creating a consistent http api easiest.  

One good example in CRUDCollection is the triple-duty that fetch() provides.  That method is called for GET, PUT, and DELETE to determine a 404 scenario automatically before the handlers for GET, PUT and DELETE are even called.  The system-wide error handler is called, with the current request url to put out a nice 404 message in the case that the thing could not be retrieved.  If it could be retrieved, you don't have to make that database call again in GET to send it to the user, it's simply available as $.fetched .  

These might seem like small things, but these small things are basically death-by-1000-cuts when trying to write a real API.  See the difference in length between the express version and the Percolator version for example ( http://percolatorjs.com/examples.html ).  There's no one place where the express version is obviously wordy, and yet it ends up being twice as long.  What's additionally not shown is that the linksCollection in the percolator example could have been require()ed in from a separate file because it doesn't rely on closures to share state like the express version does.

I'm interested to hear if you have a better a solution for those problems though.  Thanks for the feedback again.

G

[shaun etherton]

Hey Gregg

FWIW. I like it so far and have started to use it.

I'm mainly experimenting with it at this point, but your framework is a good fit for what i was looking for.

Thank you.

> > > > > > On Sun, Dec 16, 2012 at 9:24 PM, Scott Elcomb <pse...@gmail.com (mailto:pse...@gmail.com)> wrote:

> > > > > > > On Sun, Dec 16, 2012 at 1:38 PM, Gregg Caines <cai...@gmail.com (mailto:cai...@gmail.com)> wrote:
> > > > > > > > Hey all... I've been working on a new framework for restful json apis for a
> > > > > > > > while, and was wondering if anyone might be interested in taking a look,
> > > > > > > > giving feedback, etc.
> > > > > > > >
> > > > > > > > http://percolatorjs.com
> > > > > > >
> > > > > > > Is it fair to assume that percolatorjs is an implementation of your
> > > > > > > application/vnd.hyper+JSON draft at
> > > > > > > <http://caines.ca/blog/programming/hyperjson-a-first-draft/> ?
> > > > > > >
> > > > > > > --
> > > > > > > Scott Elcomb
> > > > > > > @psema4 on Twitter / Identi.ca / Github & more
> > > > > > >
> > > > > > > Atomic OS: Self Contained Microsystems
> > > > > > > http://code.google.com/p/atomos/
> > > > > > >
> > > > > > > Member of the Pirate Party of Canada
> > > > > > > http://www.pirateparty.ca/
> > > > > > >
> > > > > > > --
> > > > > > > Job Board: http://jobs.nodejs.org/
> > > > > > > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "nodejs" group.

> > > > > > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > > > > > To unsubscribe from this group, send email to

> > > > > > > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > > > > > > For more options, visit this group at
> > > > > > > http://groups.google.com/group/nodejs?hl=en?hl=en
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Job Board: http://jobs.nodejs.org/
> > > > > > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "nodejs" group.

> > > > > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > > > > To unsubscribe from this group, send email to

> > > > > > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > > > > > For more options, visit this group at
> > > > > > http://groups.google.com/group/nodejs?hl=en?hl=en
> > > > >
> > > > >
> > > > > --
> > > > > Job Board: http://jobs.nodejs.org/
> > > > > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "nodejs" group.

> > > > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > > > To unsubscribe from this group, send email to

> > > > > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > > > > For more options, visit this group at
> > > > > http://groups.google.com/group/nodejs?hl=en?hl=en
> > > >
> > > >
> > > > --
> > > > Job Board: http://jobs.nodejs.org/
> > > > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > > You received this message because you are subscribed to the Google
> > > > Groups "nodejs" group.

> > > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > > To unsubscribe from this group, send email to

> > > > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > > > For more options, visit this group at
> > > > http://groups.google.com/group/nodejs?hl=en?hl=en
> > >
> > >
> > > --
> > > Job Board: http://jobs.nodejs.org/
> > > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > You received this message because you are subscribed to the Google
> > > Groups "nodejs" group.

> > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > To unsubscribe from this group, send email to

> > > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > > For more options, visit this group at
> > > http://groups.google.com/group/nodejs?hl=en?hl=en
> >
> >
> > --
> > Job Board: http://jobs.nodejs.org/
> > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > You received this message because you are subscribed to the Google
> > Groups "nodejs" group.

> > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > To unsubscribe from this group, send email to

> > nodejs+un...@googlegroups.com (mailto:nodejs%2Bunsu...@googlegroups.com)

> > For more options, visit this group at
> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.

> To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com (mailto:nodejs+un...@googlegroups.com)

[Gregg Caines]

Very cool, Shaun...  Feel free to give me a shout anytime if you have questions, or find bugs.

G

[Jake Verbaten]

The problem of how do I write a generic request handler but still let the user decide how to handle errors is frustratingly hard.

The best solution I have come up with is.

router.addRoute("/thing", require("route-handler")({ error: myErrorFramework }))

which is annoying at best. you hard couple the signature of the error callback to a particular neutral signature that your error framework handles.

But really it's not that bad. You configure your application wide error handling system, then configure all your routes in routes.js and pass in the error handler to each route handler.

When building things like these I've only found error handling to be that annoying thing that you need everywhere and only be configured once.

[Gregg Caines]

I mostly agree, but I personally think it's worse than you make it out to be.  Take a look at the Percolator and express examples at http://percolatorjs.com/examples.html and you'll see that most of the reason that the express version is ~2x longer is due to error handling.  I find setting up error-handlers to be boring as hell and frankly don't want to do it anymore.  Also see the absolutely brilliant webmachine state machine diagram at https://raw.github.com/wiki/basho/webmachine/images/http-headers-status-v3.png to see all the kinds of errors that can occur, many before you even route.  And when a good REST API responds with a 404, for example, it should do so in exactly the same way every time regardless of whether the 404 is found in the router (no route exists) or in the resource (the requested item doesn't exist in the database).  

Real-world apps need to share a lot more across response handlers than just error-handling though.  In just my current app I need to share these common services amoung the different response handlers:  statsd, the database, a logger, and a message queue. 

And any time you want to share any centralized resource, it either needs to be:

1. explicitly passed (breaking the function(req, res) signature)

OR

2. monkey-patched on either of the req or res objects

OR

3. you're stuck with closures implicitly pulling in external state.  

I threw out #3 immediately because relying on closures and external state keeps me from splitting up my response handlers into separate modules/files.  I also found #2 to be kind of unsanitary, and only superficially more consistent with the node http server than #1.  Let's face it, if req and res aren't exactly what node gives back, then you've lost backward compatibility anyway.  It's not enough for them to be named the same and contain mostly the same stuff if the code inside makes use of any of the differences (and the entire point is to make use of the differences).

Let me know if you have better ideas for solving these problems though...  

G

 

[Jake Verbaten]

> that the express version is ~2x longer is due to error handling. 

The reason express is more verbose is because you've inlined the code. I'm not saying use express, that would be silly. im not saying percolator could be an express plugin that would be silly.

I'm saying perculator can be a set of modules on their own that do useful small things independently and when combined in a light framework wrapper solves your JSON api problems.

> And any time you want to share any centralized resource,

You require it. This trivially allows you to split it into files.

If a third party module needs access to a centralized resource then that third party should have an api like:

module.exports = function (db, statsd, logger, queue, errorHandler) {
    return function (req, res) {

    }
}

or

var config = require("my-config")
    , db = require("my-db")(config.db, "collection")
    , stats = require("my-stats")(config.statsServer)

    , error = require("my-error")(config.errorSetup)
    , logger = require("my-logger")(config.loggerSetup)
    , queue = require("my-queue")(config.queueSetup)

module.exports = function (req, res) {

}

I've actually split out all our configuration logic into a npm config module that all my modules depend upon.

I've split out the DB into a npm module that anything can require.

My logger module just monkey patches console.log and sends them somewhere.

The queue could trivially be a module, my redis connections are a module.

The stats system and error handling are more annoying to modularize.

[Gregg Caines]

Alright we might be in more of an agree-to-disagree situation.  While I considered those solutions, I mostly felt that they are more verbose and complicated than what I've gone with, and still don't actually buy you much that I can see.  One of my main goals with Percolator was eliminating complexity and verbosity like that.  It seems like a small thing when looking at one route, but when you've got 10 routes, it's enough of a pain that I thought the other way was worth doing.

With that said, Percolator's router and http support come from a dependency on a pretty clean and unopinionated http abstraction I wrote:  https://github.com/cainus/oneone .  Because of that, the req/res interface is still supported (it uses argument.length to figure out what ducks to expect), but I just don't use it because of the issues we've talked about.  Some of the higher level percolator stuff won't be available if you use that interface, but you've always got the solutions that you've described if you really want that.

You're right though - There are definitely some libs I could pull out of this for separate use, particularly the json and error representations, and all that url-wrangling that $.uri buys you.  And if someone wanted to go the req/res direction, they could still get that functionality, if I make those separate modules.  I'll see what I can do with that over the next few months.

G