JSONP Rest API

[Stephen Major]

So we are setting out on our next big project and came across SilkJS. We would like to build out our backend entirely in a v8 javascript and your library seems more appealing then node,js 

For our project the front ends will vary from extjs for the web, to windows 8 metro, android, ios etc. So it seems only logical that our backend is a JSON Rest API... we would like to easily scale out the backend by simply adding more nodes behind a load balancer. SilkJS seems perfect for this as each node shouldnt have to be aware of one another to perform their duties.

Is it easy to implement session state across Redis clusters in SilkJS?

How difficult would it be to implement a Rest API in SilkJS like restify for node.js?

http://mcavage.github.com/node-restify/

Would be nice to see a good permissions system coupled into the API as well, what would be the best way to approach that?

Also would like to make the API be an Oauth2 provider, imagine mobile clients where the user performed authentication via a pin number much like the paypal android client, this way they were not constantly sending user name and password over the wire.

I think this library has some great potential for developing backend services, the addition of these would make time to production a snap for many.

[Stephen Major]

I guess I already found the answer to this... you dont yet support TLS/SSL :( which pretty much removes the usefulness as a API service

well if you ever do support tls/ssl is there a chance youll be adding support for the questions listed in the last post?

[Jai Gupta]

I guess I already found the answer to this... you dont yet support TLS/SSL :( which pretty much removes the usefulness as a API service

well if you ever do support tls/ssl is there a chance youll be adding support for the questions listed in the last post?

 You can use stunnel in between for tls/ssl.

[Michael Schwartz]

To get SSL, use Apache or ngnix as a proxy.  A simple diagram:

Apache + mod_proxy <--> SilkJS

I would far more trust one of those well hardened servers to do the SSL and GZIP functionality than something I'd implement myself.  If you have questions about the configuration for Apache's mod_proxy, see this:

https://github.com/mschwartz/SilkJS/blob/master/extras/silkjs-vhost

(It's in the extras directory when you clone the SilkJS repository)

As far as implementing RESTful services, SilkJS was born to be the back end for RIA/AJAX applications.  It is designed to expose and perform RESTful or JSONP-like services with outstanding performance.  

To give you an idea of the performance, I was tasked for almost 4 years with architecting a CMS for PINT.  I inherited a 250,000 line PHP monster that was traditional MVC model.  I implemented the front end using ExtJS and exposed a large number of services for it in PHP and saw 100ms to 300ms typical service request times.  I ported the back end to JavaScript using Helma, a Rhino/Jetty based server side JS platform and saw the service request times drop to 10ms to 30ms times.  I ported that code to SilkJS and saw sub 1ms to 4ms times.

While SilkJS can trivially implement a RESTful service API, my preference is a JSON RPC model using POST exclusively.  POST assures no proxy between the user and the server will interfere, and allows for much larger payloads from client to server (the max length of a URL with query string is something to consider).

As far as redis goes:

https://github.com/mschwartz/SilkJS-redis

This is an add-on module for SilkJS that implements a driver and API for accessing redis.

Optionally use the ./build.js script to install the redis server in /usr/local, then use the ./build.js script to install the module.

Once the module is installed, use it as in the redis-cli.js program:

https://github.com/mschwartz/SilkJS-redis/blob/master/bin/redis-cli.js

If you require pipelining support, let me know.

Cheers

[Michael Schwartz]

I wanted to address your questions directly.

SilkJS does not implement sessions out of the box.  However, here's a fully working implementation of sessions for SilkJS:

https://github.com/mschwartz/SilkJS-Sessions

I mentioned in my previous response that there is Redis support via the SilkJS-redis module.  You'd modify the Sessions logic to store your session data in Redis.

oAuth may be problematic for mobile applications.  If you're using something like Sencha Touch, the web application is a single page one with client side JavaScript adding and removing elements from the DOM dynamically.  oAuth requires a complete redirect to some provider's WWW page for authentication and to ask the user to allow the application to access the user's data from the provider's API.

That said, I'd never send passwords across the wire.  On the client, I would md5 or otherwise encrypt the user's credentials (primarily password) and send that once to the server via REST or RPC to obtain either a cookie or authentication token.  Since you're planning on using sessions, you could store in the session whether the user is logged in.

On Aug 15, 2012, at 4:15 AM, Stephen Major <sma...@exsyshost.com> wrote:

[Stephen Major]

thanks for the great and informative replies.

it seems to me adding an additional layer such as reverse proxy is less than ideal, it adds another POF and more of a headache to deploy then just having a silkjs service that can be rapidly spun up on thousands of cloud instances.

i like silkjs better than node.js but right now without ssl/tls support i am more inclined to go with golang and possibly node.

though i am definitly going to be following this project as i think it has the ability to ne better than node.

on another note i see where you are coming from in regards to oauth... our web front end will be developed with extjs and mobile apps with extjs touch. the reason we were exploring the oauth idea is mainly windows 8 metro SSO as well as windows phone 8 and the paypal android pin auth method was appealing because if a phone is lost or stolen we would not want the thief to access billing information.

all that aside a pin is a nice speedy way to login instead of entering user name and pass each time which for secure passwords can be a pain from mobile devices. paypal app for android does this and so does the new login for windows 8...

thoughts?

[Michael Schwartz]

You might want to try Apache as proxy anyhow.  You can trivially spin up thousands of cloud instances if your AMI has both installed and configured.

On oAuth:

A paypal app talking to paypal services may trivially bypass oAuth and use some other scheme.

I also think people confuse oAuth with SSO or Open ID - the latter two may be what you really want/need.

I am not sure how a provider like Google would allow any application or WWW site to intercept Google's users' credentials.

What you might do is store a user/password combination in local storage (on a phone) and use the PIN # to identify which of several possible user/password combinations to use.  If you get some sort of auth token from a remote provider, and they don't expire it (like Google doesn't expire for 2 weeks or so, I think), you can just use that token for days…

[Stephen Major]

You are mistaken OAuth consumer for OAuth provider... what I was requesting is that an API can easily be developed with access controls as an OAuth provider... google, twitter, facebook have nothing to do with this... in fact paypal is not the only one to implement pin over OAuth if you read the tiwtter Oauth API documentation their OOB OAuth implementation uses pin access as well... this is because mobile devices cannot be trusted and as such a Oauth exchange takes place and an authentication token issued to the client but for verification each time instead of user name and pass being sent, shared secret, token and pin number is sent.

SSO in metro 8 style apps is coupled with OAuth if you read their docs there are changes that take place on the OAuth server istelf to provide SSO functionality for the metro style app. They do this to solve the device cannot be trusted model.

Here is Oauth provider implemented in node.js https://github.com/ammmir/node-oauth2-provider

the management of the apache layer is avoidable with node and golang as they have ssl coupled in nicely so its not really a solution, would love to see it added to SilkJs

[Michael Schwartz]

I understand what you're saying about oAuth provider.

The protocol still requires a redirect by the provider to some URL on the host requesting credentials validation.

[Stephen Major]

okay but in your previous reply you stated that you dont think a provider such as google would allow an intercept, there is no need for an intercept if your own api is the provider as it has nothing to do with 3rd parties like google or yahoo or twitter.

Check out twitters implementation on their oauth under the oauth method of OOB (out of band)  this method uses the pin authentication just like paypal uses.

I am saying implementing the same thing twitter has done to help secure the api

[Michael Schwartz]

Yes, it can be done.  It should be no more difficult to implement OAuth for SIlkJS as server than for any other server-side stack.  

What part of implementing it looks difficult to you?

[Stephen Major]

My extent of js knowledge is hacking extjs and jquery... Really looking into getting started with silkjs, but.it would definitely.cut.back on learning curve and time to production if there was a boilerplate for developing rest api's with ssl and authentication with permissions system.

This would greatly help those of us coming from other languages like php

Many are looking for solutions like this because of obviouse performance issues with php and even more so with the.php frmework.

Silkjs just makes sense for high performance external api's and everyone will need a good.permission and authentication system to make that happen

[Michael Schwartz]

So you want me to write this API?  I might just do it.

You want to help?

On Sep 16, 2012, at 11:13 AM, Stephen Major <sma...@exsyshost.com> wrote:

 

[Stephen Major]

I will be as much help as i can, i am new to the whole server side js idea.

So.i am facing a bit of a learning curve.

I think.this will make it easy for.people to get up and running with silkjs once.complete though. So yhey can focus o. Their apps and not something basic and prone to security mistakes like auth and permissions.

When i get some time i will draft up some design.goals, open to suggestions of coarse

[Michael Schwartz]

Every time I look at oAuth I see something that is quite custom for the site that it's running on.  For example:

1) The site has to generate a login form.  The client site generally opens the URL to this form in an IFRAME or a popup window.

- this implies that somehow our API needs some generic means of generating this login form

2) The login form may be a lot more complex than just asking for credentials. (Scopes)

- google, for example, allows access to user profile, email, contacts, docs, etc.  Permission for each is part of the oAuth login request.  The form it presents may have checkboxes for each permission requested, so the user can permit some and refuse some.

3) A SilkJS, or PHP, or any other type of server-side implementation needs to store user credentials and authorizations/permissions in some sort of backend store.

- SilkJS is pretty much agnostic about what backend you might want to implement.  It has API for FileSystem, SQLite3, MySQL, Oracle, and Redis.  A generic oAuth implementation cannot guess which backend is used or what the structure (e.g. table columns in SQL database) are named and how a password might be encrypted in the DB.

[Michael Schwartz]

I just sent an email with thoughts about oAuth server.

Now some thoughts about REST and JSONP.

I am not at all a fan of REST.  I have had to consume 3rd party APIs that are REST.  The problems I have with it are several, including:

1) There is a limited number of HTTP methods (e.g. get, post, put, delete, etc.).  The logical difference between put and post is rather subtle.  And often the actual method you want to invoke isn't one of the ones provided, so you have to do something that is a sort of hack.

2) GET can be cached by any number of caching appliances between the user's browser and the server.  These appliances may or may not obey any headers you might set.

3) The length of a URL is limited.  The implementation of the server determines the limit, as well as the browser.  The generally accepted safe max length is 2000 characters.  Now, what if you want to implement a GET that is something of a query?  I can see those easily being longer than 2000 characters.

So what I've been doing for the past several years, and with quite a bit of success, is an RPC scheme.  This RPC scheme exclusively uses POST, and the "method" (would be GET/PUT/POST in REST) is actually a function to be called on the server side.  POST never is cached.  POST is not limited in length in any way.  ExtJS supports POST via Ext.Ajax.request() and jQuery supports it via $.ajax.

For either ExtJS or jQuery, I implement a single function in a file called rpc.js:

function rpc(method, params) { … }

The function does the AJAX request using POST.  The method parameter is a string, like "Users.login" or "Comments.getComment" - think of it as Class.method.  This allows you to organize your API by class (namespace, however you want to see it).  The Users class on the server side would have methods like "login" and "changePassword" and "getUser" and whatever else makes sense.

The implementation on the server side takes advantages of one of the neat features of JavaScript - inspection.  The Users class might look like:

// file: UsersService.js

UsersService = {

login: function() {

// do login logic

var username = req.data.username;

var md5Password = req.data.password;

// call some function to validate the user credentials

if (validUserCredentials(username, md5Password)) {

Json.success(); // see below for discussion about Json class

}

else {

Json.failure('Invalid credentials');

}

},

logout: function() {

// do logout function

}

};

The rpc() function always hits the same URI on the server, /Service.  The server-side code for it might look like:

Your bootstrap.js would look like this:

// file bootstrap.js

Config.documentRoot = 'docroot'; // serve static and dynamic files from current directory + '/docroot'

include('UsersService.js');

// This function called whenever the server is hit with URI = /Service

function Service_action() {

var methodParts = req.data.method.split('.');

var klass = methodParts[0], 

               method = methodParts[1];

if (global[klass] && global[klass][method]) {

global[klass][method]();

}

Json.failure('No such method: ' + req.data.method);

}

The client side implementation of rpc():

// rpc implementation using Ext.Ajax.request.  Could use $.ajax in jQuery based client code with trivial modification.

function rpc(method, config) {

var params = Ext.apply({ method: config.method }, config.params);

Ext.Ajax.request({

method: 'POST',

params: params,

success: function(response) {

if (config.success) {

config.success(Ext.decode(response.responseText);

}

}

});

}

Usage of RPC:

rpc('Users.login', {

params: { username: myform.username.getValue(), password: Ext.util.md5(myform.password.getValue()) },

success: function(o) {

if (!o.success) {

alert('roc failed, message = ' + o.message);

}

else {

alert('log in successful');

}

}

});

The source to the Json class for SilkJS is here: https://github.com/mschwartz/SilkJS/blob/master/modules/Json.js

Json class supports JSONP - if the POST contains callback=nnn, then the response is suitably wrapped.

Note that I use SilkJS's include() function.  It supports the CommonJS require() specification as well.

Got any questions?  I can continue with this sort of tutorial for you.  Since it's being posted to the silkjs google group, it'll be saved for posterity.

[Stephen Major]

I will be as much help as i can, i am new to the whole server side js idea.

So.i am facing a bit of a learning curve.

I think.this will make it easy for.people to get up and running with silkjs once.complete though. So yhey can focus o. Their apps and not something basic like

[Stephen Major]

Not to start the rest flame war but honestly most implementations are not fully restfull which is one of the reasons for the many debates on the subject.

With that said it seems to be standard practice to use rpc as the communication interface for services in the backend. It is what we will use for our clusters of different services to communicate with one another.

But it is also seems to be a standard to expose your public facing api as a rest api.

I dont want to get into a debate here because it is hashed out in so many threads around the internet as it is, but there are a number of reasons for doing so.

Please explain how you handle versioning with your rpc method to prevent breaking possibly millions of third party written and outdated clients which you have no control over their codebase? How about discoverability?

Im not flaming just.curious if you have solved these issues which seem trivial with rest

[Michael Schwartz]

To be brief.

You don't change existing API in ways that break functionality (RPC).  Add a new method to do something different if you need something different.  It IS ok to change the guts of a function as long as it doesn't break the RPC API.

I've never considered discoverability to be a feature I ever cared about.  If you really want to implement something along the lines of how DLLs or require() works (require returns an exports object) via RPC, it's as trivial as:

/Discover (the URI)

klass: 'Users'

function Discover_action() {

var klass = req.data.klass; // 'Users' in this example

if (!global[klass]) {

Json.failure('No such klass');

}

var funcs = [];

for (var func in global.klass) {

if (Util.isFunction(global.klass[func])) {

funcs.push(fund);

}

}

Json.success(funcs);

[Michael Schwartz]

Also of interest to you, perhaps is this repo:

https://github.com/mschwartz/SilkJS-extjs4

SilkJS backend, ExtJS 4 front end.

The back end defines a database in MySQL and provides RPC services, including for those that ExtJS does under the hood to load stores (like when you click on the paging toolbar).

To give you an idea of how concise code for SilkJS can be and how powerful the API is, here's the server-side services to do CRUD for any generic table you might define.

function Server_action() {
	switch (req.data.method) {
		case 'listUsers':
			Json.success(Schema.list('Users', {}, function(o) {
				o = Schema.clean('Users', o);
			}));
		case 'editUser':
			var example = Json.decode(req.data.example);
			example.userId = example.userId || 0;
			Schema.putOne('Users', example);
			Json.success();
		case 'deleteUsers':
			var examples = Json.decode(req.data.examples);
			forEach(examples, function(example) {
				Schema.remove('Users',example);
			});
			Json.success();
	}
}

Additionally, the ExtJS grid and store is dynamically generated from the database Schema definition (see Server.js).  That is, a single ExtJS 4 grid extension can handle CRUD for an infinite variety of DB tables, even in the same application.

This example is quite a bit simpler than the RPC scheme I've described, but only for illustration purposes.

Note that editUser does both create and update of a User record.

[Stephen Major]

Versioning with REST seams to be what everyone else is doing with their public api's it is less restrictive.

I have to wonder why almost every solution that was designed to be scalable uses REST on the public.facing.api and rpc on the backend.

Google, facebook, twitter

"The downside to.rpc would be that you will be defining your own semantics for operating on resources (which your client developers will need to learn) when you could use the operations that already exist.HTTP defines the operations/actions/verbs that can be used. If you use this, you can provide a uniform interface that api users can call without being forced to learn your new plans.Doing so would allow you to achieve one of the helpful constraints in REST."

http://stackoverflow.com/questions/4575463/is-xml-rpc-bad-used-as-a-protocol-for-a-public-api-implementation

Skynet in golang which is designed to help developers quickly get up to speed with developing scalable, self healing, applications with automatic service discovery and recovery uses rpc on the backend for setvice to service communication but yet they suggest using REST for the public facing api

https://github.com/bketelsen/skynet

Like i said we are not.interested in debating which is better, we are only interested in helping.create a boilerplatr for silkjs whoch others can quickly get upbto speed for their REST api's

It is okay that you like rpc, to each is their own, but there are plenty who will want a REST api, its not like i am suggesting a boilerplate that is only useful to a subset of.people, on the contrary i am suggesting one that is useful to the largest group of.people.

Here you can see one is available for nodejs http://mcavage.github.com/node-restify/

[Eric Dykstra]

Mike,

The RPC scheme above is sweet. I've only added a couple of methods, but it is going to work really well for calls from my ExtJS MVC controllers.

Then it occurred to me that I should be able to load ExtJS stores using RPC too. However, I am not sure how to go about it.

I have this model:

Ext.define('ACS.model.Volunteer', {

    extend:                 'Ext.data.Model',

    fields:                  Schema.Volunteers.fields.concat(Schema.Persons.......,

    idProperty:             'volunteer_id'

});

and this store:

Ext.define('ACS.store.Volunteers', {

    autoLoad:                false,

    extend:                 'Ext.data.Store',

    model:                  'ACS.model.Volunteer',

    proxy:                   {

        actionMethods:       {

            create:         'POST',

            read:           'POST',

            update:         'POST',

            destroy:        'POST'

        },

        reader:              {

            root:           'records',

            totalProperty:  'totalCount',

            type:           'json'

        },

        type:               'ajax',

        url:                '/volunteers_list' 

    },

    remoteSort:              true

});

The /volunteers_list is a function in the server.js file called (of course) volunteers_list_action().

Can I re-factor the ExtJS store above, to make a RPC instead of using  volunteers_list_action() in server.js?

Thanks,

Eric

[Michael Schwartz]

Of course!

SilkJS and ExtJS go together very well.

Reader.  root should be list, totalProperty count (double check the Schema class, for Schema.list method).

http://stackoverflow.com/questions/9679059/how-to-pass-parameter-to-ext-data-store

That URL is the basics of how you use RPC.  baseParams or extraParams or in the store's beforeload handler

In fact, the SchemaGrid component does it here:

https://github.com/mschwartz/SilkJS-extjs4/blob/master/docroot/client/Ext.ux.SchemaGrid.js#L18

And you can see here that listUsers, listWhatever, etc., is the method config value set when instantiating that grid.

https://github.com/mschwartz/SilkJS-extjs4/blob/master/docroot/client/Viewport.js#L9

And of course, listUsers:

https://github.com/mschwartz/SilkJS-extjs4/blob/master/Server.js#L39

You don't have to use _action() methods.  They're a rough equivalent to a Java server's Servlet.

Instead of _action(), you can just put the body of your _action() method in a file called, say, "Users.sjs" in the docroot and refer to it by URL: http://my.example.com/Users.sjs in your Reader.

Now you can edit the .sjs all you want and the server will reload/recompile it when it changes on disk.  In other words, _action() means you have to edit then restart the server to see your change take effect.  Not with .sjs!

--
You received this message because you are subscribed to the Google Groups "SilkJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to silkjs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Eric Dykstra]

Mike,

Thanks.

This ended up working just fine, even though I was sure it failed at one point.

Ext.define('ACS.store.Volunteers', {

    autoLoad:                false,

    extend:                 'Ext.data.Store',

    model:                  'ACS.model.Volunteer',

    proxy:                   {

        actionMethods:       {

            create:         'POST',

            read:           'POST',

            update:         'POST',

            destroy:        'POST'

        },

        extraParams:         {            

            method:         'Volunteers.list'

        },

        reader:              {

            root:           'records',

            totalProperty:  'totalCount',

            type:           'json'

        },

        type:               'ajax',

        url:                '/rpc'

    },

    remoteSort:              true

});

I think what was tripping me up, was that I thought I had restarted the server, but had not. So - that's my next step, which should be trivial at this point - move it to a *.sjs file.

Thanks.