Re: [nodejs] Maybe of interest: Why we need a NodeJS application server or: writing boilerplate code considered harmful

[Angel Java Lopez]

Hi people!

Panyasan, very interesting... but I don't see the need of most of your items. Maybe you want not an "application server" (ooops.... I still remember J2EE... argg... ;-) but an "opinionated web framework" (like Ruby on Rails wo/model code generation) (in the comments to your post, Geddy was mentioned).

Let's review:

It should also not force a different programming model on developers (like Opa or Meteor), but let you code in plain old asynchronous javascript. No to-javascript-compilation (except optionally). No magic. Just the right tools.

It exists. Node.js + Javascript ;-)

It should have an integrated API for client and server.

Can you elaborate this?

It should provide a static HTTP server, REST routing, and bidirectional, realtime messaging and broadcasting (such as Socket.io).

Then require('socket.io'); Why "static" for HTTP server? There is middleware for Connect/Express. REST routing: it's not my area, but there are some modules.

It should offer async startup/plugin/configuration system like Cloud9′sArchitect.

Easy one (maybe some boilerplate for async support), or someone writing it as a module, then require it;

It should provide an out-of-the box system and API for user & group management, registration, access control, Password storage/retrieval/update etc, preferrably with a set of built-in templates that can be used for managing the most generic configuration tasks. With this, a pluggable system to use third-party authentication providers.

??? A bit: invoicing to users included, paypal system, etc. ;-) Ok, again, it could be resolved with a module.

It should also provide an integrated system of data modeling and persistence. I really do not care about database technology. I simply want to store, edit and retrieve my model data.

I work in many technologies, and all these or not needed, or provided by a library and tools.

t could also have a toolset that would allow you to deploy your application instantly to a cloud provider such as Heroku or Nodejitsu.

Again, Node.js

Sorry if it sounds harsh, I don't manage English expressions and variants, it's not my mother tongue. But IMNSHO, we don't need all that, unless you like Ruby on Rails and alike (maybe Geddy). 

Other opinions?

I think that that is the key point of your post: RoR-like or not (Node.js community (or part of community) actually "prefers" Sinatra-like). "Application Server" term confused me. In my jargon, it's related to things where the business logic, application resides, exposed to many kind of clients (web, desktop, no web, whatever the future will bring to us, etc....) SAP uses that term too, a la J2EE (sorry for mention it again ;-)

Angel "Java" Lopez

@ajlopez

gh:ajlopez

On Fri, Dec 14, 2012 at 10:51 AM, panyasan <cmbou...@gmail.com> wrote:

This is not strictly a technical question on node, so please excuse if this is the wrong forum, but if you're interested, I'd like to hear your opinions on the following blog post:

http://panyasan.wordpress.com/2012/12/10/why-we-need-a-nodejs-application-server-or-writing-boilerplate-considered-harmful

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Mark Hahn]

I'm sure many would appreciate such a comprehensive large framework.
I and many others in the node community would not.

One characteristic of the node "philosophy" is freedom. The freedom
to plug existing small modules together like a lego set. The freedom
to easily write our own modules. The freedom to swap out modules for
others as better ones come along. The freedom to make our
architecture unique while not writing code from scratch or even using
boilerplate.

If I had to live within the constraints of J2EE or ROR or even
Express, I would find another job. My architecture migrates quickly
from project to project with each one more awesome than the last. Any
existing framework would be outdated within a year as far as I am
concerned.

It is a new world..

[Ben Noordhuis]

On Sat, Dec 15, 2012 at 7:52 PM, Mark Hahn <ma...@hahnca.com> wrote:
> I'm sure many would appreciate such a comprehensive large framework.
> I and many others in the node community would not.
>
> One characteristic of the node "philosophy" is freedom. The freedom
> to plug existing small modules together like a lego set. The freedom
> to easily write our own modules. The freedom to swap out modules for
> others as better ones come along. The freedom to make our
> architecture unique while not writing code from scratch or even using
> boilerplate.
>
> If I had to live within the constraints of J2EE or ROR or even
> Express, I would find another job. My architecture migrates quickly
> from project to project with each one more awesome than the last. Any
> existing framework would be outdated within a year as far as I am
> concerned.
>
> It is a new world..

A node.js "application server" is something Bert Belder and I have
been discussing.

The proliferation (and wildly varying quality) of modules and
frameworks seems to be holding back node.js to some degree. I talk to
a lot of developers and it's one of the most common complaints:
"There's too much choice, we don't know what to use."

The idea is to have a curated list of modules with appropriate
stability and support guarantees. If you find a bug, we'll make sure
it gets fixed in a timely manner; you won't be at the mercy of the
module author.

That gives the developer a stable, known-good base to start out with
while preserving the freedom to use whatever he wants. It should also
take away the cold feet that some businesses have.

[Mark Hahn]

As I said, many would appreciate it.

The reason I wrote the long diatribe was because I believe
node/javascript offers a unique base for the freedoms I mention. If
the community as a whole adopted something and all the new useful
modules required it, that would be a great shame.

It would hinder forward architecture progress like ROR did. ROR can't
come close to the architecture features I'm using now. And you seem
to be asking for an ROR equivalent for node. That would slow down the
awesome progress node has been enabling.

In other words, be careful what you ask for. You just might get it.

[Mark Hahn]

Clarification: I said: "you seem to be asking for an ROR equivalent
for node". I didn't mean your new framework would have the old
features of ROR. I just meant that no matter how modern your new
architecture would be, it would become stale quickly in the fast-paced
world that node enables.

[Alexey Petrushin]

There's very simple way ti navigate in lots of modules - go to github, select JS lang, sort by popularity, read first 5-10 pages, in 90% cases You'll found what You want.

[Jake Verbaten]

> The idea is to have a curated list of modules with appropriate
stability and support guarantees.  If you find a bug, we'll make sure
it gets fixed in a timely manner; you won't be at the mercy of the
module author.

A priviliged list of approved modules is dangerous. It stales development.

Instead fix the npm discoverability problem to make it trivial to search for a list that meets the criterias.

The discoverability problem is a lot harder but far more worthwhile and it will solve that very problem.

[wouter.s...@gmail.com]

I agree that maintaining a list of primus inter pares modules is very tricky path to follow

[panyasan]

Hi Ben, hi everybody,

while I appreciate all the answers including the critical ones, I think only Ben really understood what I was aiming at. Probably the term "application server" was not appropriate. I didn't ask for a big monolithic system. My claim was that web applications have a certain set of features in common (some of which are in the list in my post), which are really generic ("the boilerplate"). It makes absolutely no sense to implement them again and again, maintain them, write tests, etc. when they could really be bundled together and maintained by a community of people who need this functionality. If you're a professional, full-time developer, you're probably best of with creating this kind of stuff yourself. But my argument was that a lot of potential is lost because of the fragmentation ("the freedom to choose") in the node module ecosystem. Discovering modules is not the problem in my opinion - maintaining them in a longer-term perspective is....

[greelgorke]

for me, it's more than freedom, but also an approach comes close to true modularisation: unix-philosophy. it's in node its rebuild with one-function-modules a la substack + simple but powerfull and flexible interface convention: callbacks + streams. if i think i need an application server, i'd rather go back to jee, ror or even .net if it fits my task.

but we for sure need some help in finding the right modules.

[Jake Verbaten]

> Discovering modules is not the problem in my opinion - maintaining them in a longer-term perspective is....

Discovery is a problem. You can't discovery modules which are actively maintained and have LTS

--

[Martin Wawrusch]

I think everyone here understood you very well, a lot of us have just been there and done that, in different languages, and seen the negative consequences of it. And there are a lot of great modules out there that are perfectly maintained and can be easily combined to achieve what you want to achieve. Express, Mongoose, Passport, Request and many others come to mind. The real problem is discovery, buy we don't bitch about it because it is our job to fix it if it annoys us. Node, imho, follows Nike's claim: "Just do it". If you feel the need for boilerplate code then write it or hire someone to do it. BTW, there are a couple attempts to do so out there, so you could contribute to those projects.

@Mark excellent comments as always. If you ever come to LA chat me up and I'll invite you to a  beer (or more :-)).

On Sat, Dec 15, 2012 at 1:36 PM, panyasan <cmbou...@gmail.com> wrote:

--

[greelgorke]

in fact we have allready several "stacks" for web development, which are something like ou wish: the connect-stack including express and comatible stuff like passport. flatiron is another one. there are even frameworks on top of them preselecting stuff, like all the ror-clones: geddy, tower, railways etc. all the stuff is allready there, with freedom and preselection.
but as said before, node don't claim to be chimera making everything possible. there are some classes of usecases where node just fits, and classical web development is not that case. we dont even have this problem. our problem is to know, which case is better solved by node, and then to find the right modules.

i think that a thing like an application server is just one solution to the problem: (1)how to find the right libraries, (2)integrate them and (3)ensure they evolve and are maintained.
1: improve the search and combine it with ranking including usage statistics, reputation and activity of the project. there is workinprogress on it.
2: simplify apis, split them into smaller one, follow conventions and patterns like callbacks and streams. in the end of the day we all depend on core libs, so make them as smalles denominator in your api. and we do it already.
3: this is the community thing, want to ensure maintanace and progress? contribute and give the community something back! you can do it with workforce or sponsoring. this is how node evolved. but for sure we need more communication at creating of packages. too foten happens that two or more libs arise only because the maintainers cant work together, because of detail discussion.

[Ted Young]

My experience (ROR, etc) is that large frameworks and plugin systems actually increase code rot and maintenance dramatically.  Rails versions much faster than ruby, and even stable, community-seal-of-approval plugins that do basic things get left in the dust.  For example, acts_as_authenticated and acts_as_attachment were de facto modules written by core ROR community members, and I was forced to haul them out and rewire several apps in order to continue basic maintenance and development when they got deprecated.  In other words, the new stuff in these systems does not work with the old stuff, because the system itself has to evolve.  Conversely, the php apps I've created out of libraries and my own code require almost no maintenance to continue upgrading, because there is no middle layer of code between me and the language constantly shifting around under me.  

 

Better than large systems are patterns and coding standards that are widely disseminated, to help ensure libraries can easily inter-op with each other.  The event and stream API's (well, hopefully the stream api ;) ) are examples of these in node.  I'm excited that npm will crack the discoverability problem, I think that would be a first for a package management system.  But if various community institutions want to maintain lists of recommended modules, I don't see anything wrong with that, though I think node-core should avoid an "official" list.

Panyasan, out of curiosity, what is wrong with express and the various auth libraries, etc?  All of the pieces you describe exist as libraries, how exactly does this not fit the bill of having generic pieces without being a monolithic system?

Ted

[Jake Verbaten]

Its difficult to choose modules.

 - what's the best module

 - which one doesn't require maintenance yourself.

 - which one will have long term support.

I've found I had to simply write a lot of modules myself and stop using other peopes modules because they wern't stable enough or properly maintained.

[panyasan]

@everybody: Thanks for taking time to comment on my post! I really appreciate your input. I still think that there are some misunderstandings, which is my fault, because it seems that I haven't made my intention clear enough. As a summary, let me try to restate my argument - not because I hope to convince you, but to show you that there is a segment of developers that would profit from what I was trying to outline. This segment -- part-time and occasional programmers -- is a minority, yes, but it would be able to contribute valuable stuff in return, when being relieved of "The Boilerplate (c)". ;-) 

I did not mean to say that we need something new to replace the existing stuff (express+middleware, mongoose, passport, etc., etc., etc.) with a monolithic new architecture. On the very contrary, my argument was that it could be worthwhile to make something that you, as full-time developers, do for every new of your projects -- the wiring together of all these different parts -- into a collaboratively maintained project, which I labeled (or maybe: misnamed) "application server".  The claim is -- and here maybe you would strongly disagree -- that there is a certain amount of application logic that is the very same for 90% of applications, at least the ones I would like to write (see my post). 

It is not that I am too lazy to do this. I have done it in the past and learned my lesson (see my post). It is that I have too little time to be spending it on a) writing it in the first place, b) writing tests to make sure it is stable, and c) most importantly, maintaining it beyond the time I will be paid (or have free time) to develop the main app. Contracting it out to someone else wouldn't help, because the problem simply shifts. As soon as you stop paying this person (and nobody would be paying her indefinitely), the backend will become obsolete, the libraries used will go out of maintenance, etc, and nobody is there to fix that. 

So, to sum up: the disagreement is not over whether the node community has already created great tools, which individually will do what I need. It most certainly has. I understand what you're saying is "if you're not ready to do everything yourself (i.e. wire the different modules together), don't be in the business of writing node applications". That is of course a valid opinion, but as I said, I think that a lot of valuable things could be created if one only had to worry about the "real stuff", i.e, the business logic specific to the application, and not to the generic parts. I love coding with JavaScript and NodeJS and really think the JavaScript/NodeJS community and ecosystem are awesome. That is why, even if I had the time for it, I wouldn't want to learn Ruby or switch to Java.

  

[panyasan]

Am Sonntag, 16. Dezember 2012 07:54:01 UTC+1 schrieb tedsuo:

Panyasan, out of curiosity, what is wrong with express and the various auth libraries, etc?  All of the pieces you describe exist as libraries, how exactly does this not fit the bill of having generic pieces without being a monolithic system?

Hi Ted, as I write in my last response, there is nothing wrong with the various libraries. A few months ago, I started a "tutorial" (it was actually more of a documentation of my trial-and-error attempts) on how to build a complete application structure with these libraries. You can find it on my blog. It was a great experience to see the creativity and productivity of the node community, evidenced by the sheer amount of modules on NPM. I looked at a ACL library, which didn't have a mongodb backend, asked the author, and he added one in no time. What more can you ask for? That's just great. But after a while, I got stuck. There were a lot of loose ends, the stuff didn't naturally fit each other, and I had to spend more and more time on the "glue code". That's not what I expected, and I simply had no time for it. But I am repeating myself. I hope I made myself clearer in the other response. What I need is a higher-level API which expose only functionality, and doesn't directly make me dependent on the libraries. Maybe that's a stupid thing to wish for, but that's how it its for me (remember, I am NOT a professional). In a sense, what I would like to have is a "sandbox" in the sense of Nicolas Z. Zakas' "scalable javascript architecture" (I talk about it in the first part of the tutorial).

[Isaac Schlueter]

Yeah, I haven't found module maintenance to be a big issue, at least
for small modules. If a program is small enough, you can just finish
it, and all that's left are minor bugs that are trivial to fix when
they're found.

If a module is small, it doesn't really matter how good the "long term
support" is. You just fix it if there's any problems, and it's easy
enough to do so.

It's hard, though, to find the good modules: the ones that do a single
thing well, are no bigger than necessary, and are well-covered with
tests and documentation. That's the discovery problem.

[Jake Verbaten]

> On the very contrary, my argument was that it could be worthwhile to make something that you, as full-time developers, do for every new of your projects -- the wiring together of all these different parts -- into a collaboratively maintained project, which I labeled (or maybe: misnamed) "application server".

That wiring is different for every author. It's different for every project I've done as an author. It's different for every base framework.

If you want this you fit right into the target audience of meteor and should probably be using that. Alternatively consider using ruby on rails, it does the things you want reasonably well.

> There were a lot of loose ends, the stuff didn't naturally fit each other, and I had to spend more and more time on the "glue code". 

Your also forgetting that application architecture is application specific. A generic boilerplate won't fit most apps. the reason things dont naturally fit is because you didn't architect your application base based on the application, you tried to use something generic.

[panyasan]

I looked at Meteor and at first thought that it might be a good fit. However, as I said in my post, I want to stick with the NodeJS model of doing things and not use a completely different (synchronous) paradigm. But that (as well as not wanting to use RoR) my personal choice, nothing I need to bore you with. Please tell me if I am getting on your nerves, after all, this seems to be a technical forum, not an application design forum. I am not saying I am right, I'm just responding to your posts and trying to defend my claims.

What I find interesting to debate is if your general claim (each project is radically different) really is true for everybody. It is true for you of course, and probably for the majority of people posting here. But when I look at my - admittedly anecdotal - experience, I can say that all of the major applications that I build (or would like to build) need exactly the same kind of backend. To make it less theoretical, here are the applications:

• A bibliographic data manager, used for collecting, managing and publishing bibliographic references. Target audience: research groups, university departments
• A twitter-like intranet communication tool with a calender, used in an organization to document business processes across different companies. Target audience: company staff, administrators.
  
• A staff scheduling program, used for organizing service shifts at conferences: Target audience: students interested in working a shift.
• Planned, but not implemented: a web-based IMAP administration/backup/copy tool. Target audience: Company email administrators
  
• Planned, but not implemented: a conference organization tool specific to the organization of session-based conferences. Target audience: Academics
  
• Planned, but not implemented: a newsletter tool with very fine-grained distribution options. Target audience: Company PR administrators.
  

That's only three existing projects, so my evidence is tiny compared to the number of projects you or others have already done. But they including the not implemented ones are all very different and have different audiences, and their backends have a lot in common, at least, however, the following three elements:

• They manage the users that sign up and login and out. Therefore you need the logic to manage the user data (name, email, other properties), and a GUI administration tool to do the management. That is very generic, to be sure, and doesn't cover all the different properties that you might want users to have. But for rapid application development, that is enough to get you going.
  
• They manage access control, i.e., you define resources on one hand, and roles and permissions, on the other, and then assign roles to users, which in turn have permissions to access resources. That is a generic, but very powerful paradigm, which fits - at least in my view - all of the use cases that I can think of. This access control system can be displayed and edited - I have already done that in PHP - in a visual editor. Again, very generic, but if you work with named resources (which can be a database or a button on a widget on the client), it seems to me to be the right tool.
• Data modelling and persistence. I don't need to talk about this, it's been solved by many different libraries (such as mongoose), so I am just adding it for the record. Full-fledged systems like Wakanda even provide visual editors for models, but that's not what I need.

So again, what you're saying is that trying to solve thins generically is the wrong approach. It might be true for the majority of production-grade apps, but I would know exactly how to hook the application-specific requirements into this one-size-fits-all system. Ok, but enought of this. Thanks again for your thoughtful comments.

[Kevin Jones]

On Monday, December 17, 2012 10:52:06 AM UTC, panyasan wrote:

What I find interesting to debate is if your general claim (each project is radically different) really is true for everybody. It is true for you of course, and probably for the majority of people posting here. But when I look at my - admittedly anecdotal - experience, I can say that all of the major applications that I build (or would like to build) need exactly the same kind of backend. 

Think this hits the important point, people have different needs & expectations that are not always reflected in a single community. Back in the day I think it was fairly obvious Java app-servers were based on flawed fundamentals and should never have taken off, the fact they did speaks to needs beyond having the best solution for the problem at hand.

Today though I wonder if you looking for the wrong answer, is the new application server not a PaaS API?

[klr...@gmail.com]

I was looking at a similar decision albeit on a much smaller scale but with quite a lot of existing non js backend software. In my eternal quest for simplicity eventually I decided to rewrite it all using Zappjs which is a sort of wrapper with express, node, socketio, coffeescript, coffeekup-views. Being risk shy I stuck to my existing db structure but  replaced it with the Mariadb drop in so I can do some async queries where it makes sense and hopefully avoid any license issues. Although I have Linux and dev & debugging  background as a total js newcomer I may have missed some of the finer points. But customers seem to be happy with the first pieces I rolled out and I am quick on my feet changing and publishing almost every week and rewriting the other stuff little by little as time permits. Personally I found it more efficient to integrate bits from existing modules into my code to keep it all very compact and readable for me and with the lowest possible overhead, of course. One also learns more and thus gains long term productivity. No regrets, all is stable, with a good response times so far and quite easy to manage after the usual steep js+async newbie learning curve. After all it's all about productivity and flexibility and freedom. Hope this comments helps you. And hats off and warm thanks to the maintainers of Node and the surrounding tools!!! You're my heroes.

Karl

[panyasan]

Am Montag, 17. Dezember 2012 12:24:56 UTC+1 schrieb Kevin Jones:

Today though I wonder if you looking for the wrong answer, is the new application server not a PaaS API?

Definitely what I am looking for looks more like an API than one of the old Java App Servers. It becomes increasingly clear to me that my use of the word "application server" pushed the wrong buttons... Thanks for pointing that out.

[James Tikalsky]

> What I find interesting to debate is if your general claim (each project is radically different) really is true for everybody

Well, for node.js apps, they *are* radically different, because we (the node.js community) are still in the early adopter phase of things. (To compare with Java, it's 1998... think about what app servers looked like in 1998.) We're building lots and lots of apps, all kinds of different stuff, and we're learning what works and what doesn't. As Kevin pointed out, the PaaS providers will start to tailor their offerings with specialized configurations, once they know what we typically build and deploy. In addition, we'll see some more opinionated frameworks appear (RoR being a good example: it's opinion was to get things up and running quickly). Probably some of the things being built on top of express.js will emerge as "framework for building type x application" solutions.

[panyasan]

Good points, thanks. Let's see how things will develop. I just saw this (Firebase security API) and I think - in combination with the other stuff - this what I am after, only that I would like to have it as a set of node modules ... Happy holidays everybody, and keep up the great work on Node & its modules!