Developing SOFEA

[Justin Meyer]

SOFEA/TSA application developers are going to face a new set of
challenges. We should have some ideas on how to address them.

I'd like to start a discussion on best practices for the engineering
issues developers will face as they push more logic onto the client.

I'm hoping that by addressing these concerns with practical and tried
solutions, we'll be able to make SOFEA/TSA a little less scary.


The following are a few issues that I've come up with. Please share
your experience and the techniques you use.

DOM Building - How do you build HTML?

Event Handling - How do you organize event handling, and attaching
event handlers?

Ajax/Service - Do you have a layer that handles data transport? What
does that look like?

File Structure - Do you have a standard file structure? What does it
look like?

Programming Paradigms & Architectural Patterns - Does Aspect Oriented
Programming, Functional Programming, MVC, SOA have a home in your
application? Do you use simulated classes or stick with prototype
based inheritance? What layers do you use to abstract application
technologies. Do you use OpenAjax to publish/subscribe?

Testing - What do you use to test?

Compression / Build - How you you combine and load JS for faster
download.

Development Methods - Waterfall, Agile, V, Story based development,
Requirement based development.

Logging - How do you log data from the client. Do you report errors?

Tools/Libraries - Which tools and libraries do you use? Obviously
Firebug, but is there something good for IE?

[Justin Meyer]

[Mic Cvilic]

Hi Justin,

Here below are the answers to your mail and about what we are doing/use at beebole.

Cheers,

Mic

On 23 Oct 2008, at 23:57, Justin Meyer wrote:

SOFEA/TSA application developers are going to face a new set of
challenges.  We should have some ideas on how to address them.

I'd like to start a discussion on best practices for the engineering
issues developers will face as they push more logic onto the client.

I'm hoping that by addressing these concerns with practical and tried
solutions, we'll be able to make SOFEA/TSA a little less scary.


The following are a few issues that I've come up with.  Please share
your experience and the techniques you use.

DOM Building - How do you build HTML?

For our project we developed a JS library called PURE that populate pure HTML's based on JSON data.

It's very fast and we've put it open source. Feedback, ideas are highly appreciated ;)

http://beebole.com/pure

Event Handling - How do you organize event handling, and attaching
event handlers?

Attaching events during the HTML rendering, again with PURE. 

And avoid slow DOM parsing/attaching of the HTML again after rendering.

We cared the code of PURE to avoid IE leaks while inserting and removing HTMLs.

See the example 4 in the zip file at docs/allExamples.html from:

http://github.com/pure/pure/downloads/master

Ajax/Service - Do you have a layer that handles data transport?  What
does that look like?

jQuery ajax calls for POST.

Or JSONP(GET) to avoid the cross domain scripting limitation.

Especially useful while developing on a local http server while the services are in a remote web server. 

Or to mashup external contents.

File Structure - Do you have a standard file structure?  What does it
look like?

Programming Paradigms & Architectural Patterns - Does Aspect Oriented
Programming, Functional Programming, MVC, SOA have a home in your
application?  Do you use simulated classes or stick with prototype
based inheritance?  What layers do you use to abstract application
technologies.  Do you use OpenAjax to publish/subscribe?

Our business backend is ERLANG + MNESIA -> functional + some powerful goodies

The web server is MOCHIWEB + NGINX

The front-end is JS + HTML (jQuery and PURE libraries)

We don't use inheritance, objects, but try to be functional with JS too.

We build a single object like jQuery and implement functions that can be chained.

The browser never handle any business logic, for security reason and avoid to duplicate the business logic.

The server never tells how to represent data.

Not really an MVC, but a strong separation between UI and business logic.

The front-end and the backend communicate through JSON services (The POX way but in JS).

The only exception is when the target is not the browser (spreadsheets), here the backend sends HTML tables.

I don't know if it deserves the label SOFEA, but summarised in a few words: 

- we build single page web apps

- the browser handle the UI exclusively

- the server handle the business logic exclusively

- both server and browser talk to each other in JSON

It is very similar to what we were doing on SAP projects in the past.

Replacing XML by JSON and XSLT by HTML, and are very happy by the change.

Testing - What do you use to test?

A home made HTML/JS test suite to test massively ERLANG services.

We'll wait anxiously to read what do you use to test the frontend ;)

Compression / Build - How you you combine and load JS for faster
download.

Dean Edwards packer: http://dean.edwards.name/packer/ 

And bundle all js in a single file gzipped.

CSS compression with YUI compressor and bundled in a single file.

Development Methods - Waterfall, Agile, V, Story based development,
Requirement based development.

No

Logging - How do you log data from the client.  Do you report errors?

Not for now. Only backend logs.

Tools/Libraries - Which tools and libraries do you use?  Obviously
Firebug, but is there something good for IE?

http://aptana.com (IDE based on eclipse integrating Firebug) with a free community version.

Microsoft Visual Web Developer Express Edition 2008. 

It's free, and you get a very nice web IDE and a powerful JS debugger when working with WINDOWS + IE

SUN Virtual Box to get various OS and browsers running on the same computer at the same time, for quick HTML/CSS testing.

-- 
BeeBuzz, the voice of BeeBole : http://beebole.com/blog

[Justin Meyer]

Thank you so much for sharing!  I'm going to go through your response again, but some preliminary comments: 

Could you describe for the group a little more about Pure?  How does the template system work.  I looked in the example code but it looked like black magic.

For testing, I use JavaScriptMVC.  Actually, I use JavaScriptMVC for pretty much everything.  But it's testing is really solid.  I would argue it's superior to any other library.  And this is really the only JMVC feature I would feel comfortable saying that about.    

You can see a quick demo here: http://javascriptmvc.com/learningcenter/test/demo.html.  Make sure you allow popups.  You can read about it here: http://javascriptmvc.com/learningcenter/test/index.html.

In my opinion, it has the nicest syntax, the most complete event simulation, and it is the easiest to use.  If you use JavaScriptMVC's controllers, it will write part of the tests for you.  It also supports Ajax fixtures which make test-driven development possible for JavaScript.

But, I've just recently (1 hour ago) added what could be it's best features - command line testing.  Using a beefed up version of Resig's simulated FF2 environment for Rhino, any javascriptmvc application will run its unit tests automatically in a command line.

This is great for integrating with commit or build scripts.  Obviously, this is not a test across all browsers, no does it match FF2 perfectly.  However, it provides a good and most importantly RAPID sanity check that can be integrated into other scripts. 

Sorry for the JMVC plug.  I'll keep it to a minimum, but testing is one of those features I've tried hard to solve which I think is critical for TSA development.  Solenium has too much overhead to get started.

p.s.  You might catch me saying that JMVC's build process and event delegation are better than anything else's too.  But not as much as testing.  

--
Justin Meyer

Jupiter IT Solutions
847-924-6039
justin...@gmail.com
AOL: jusbarmey

[Kris Zyp]

> But, I've just recently (1 hour ago) added what could be it's best
> features - command line testing. Using a beefed up version of Resig's
> simulated FF2 environment for Rhino, any javascriptmvc application
> will run its unit tests automatically in a command line.

Have you tried HTMLUnit? It is built on Rhino and provides pretty
extensive browser environment emulation, and I think even has decent
support for emulating the different browsers out there.
Kris

[Mic Cvilic]

Hi Justin,

Are you developing JMVC ?

I'll have a look at the testing feature you talk about, as we are cruelly missing such tool. Thanks.

About PURE...

When we switched from XML to JSON, we lost the speedy client side XSLT.

So we started a javascript rendering engine to replace it.

The first version was DOM based but too slow for production use. The today's version compares with the speed of XSLT.

But we ended with something interesting:

The HTML stays HTML. Without funny tags or slicing.

This brings 2 main benefits:

1) Any portion of your document can be recycled and become a template, again and again.

2) The HTML is now part of the versionning during the development and maintenance. And not anymore an initial mockup that is then broken in pieces.

As a side effect, you can see your HTML in a browser or in a WYSIWYG tool during the whole duration of the project (personally a big win for me).

The good idea, I think was to use the W3C selectors, to make an unobtrusive link between HTML elements and the JSON data.

This link we call a directive is a JSON too that has the structure { "W3C selector" : "action", ... }

It means: "select" an HTML element and apply the "action" to it.

Although it was quite pretentious, we found that the acronym PURE Unobtrusive Rendering Engine was a good name for what it was ;)

Later we enabled an automatic link between CSS classes and JSON properties, but this was more to follow a trend than using it in production.

This automatic mix between JSON and HTML is may be the black magic you talk about.

A simple example with directives:

- - - - - - - - - - - - - -

The HTML:

<p>Hello <span>Name</span></p>

The JSON data:

var data = { "who" : "Mary" };

The directive:

var directive = { "p span" : "who" };

The command(in jQuery syntax):

$('p').render(data, directive);

The result:

<p>Hello <span>Mary</span></p>

The same example with automatic mapping:

- - - - - - - - - - - - - -

The HTML:

<p>Hello <span class="who">Name</span></p>

The JSON data:

var data = { "who" : "Mary" };

The command(in jQuery syntax):

$('p').autoRender(data);

The result:

<p>Hello <span class="who">Mary</span></p>

You can get various examples and a detailed documentation at:

http://github.com/pure/pure/wikis

Any comments, ideas, improvements will be highly appreciated.

Thanks,

Mic

[Justin Meyer]

No, I haven't.  Thanks for sharing.  I'll read up about it and share it with the group.

[Justin Meyer]

Mic,

  Yes, I am developing JMVC.  The documentation is lacking for the new feature, but post a note on the forum when you want to learn about it and I'll write up instructions.

More on PURE, and I apologize if I am being too lazy to look this up myself.  

Does it use a CSS query selector to fill in the data?

Can it iterate through a list and use control structures (if / else)

Does it use eval in any way?  JavaScriptMVC has taken over EJS (which uses eval), but people have been looking for template system that doesn't.  

Thanks!

[Mic Cvilic]

Hi Justin,

May be we should move this discussion on the PURE or JMVC groups, to avoid spamming sofea.

Yes it use CSS selectors to locate the HTML element you want an action to.

With one twist for attributes, the selector "input[value]" means in W3C language, select an "input" that has an attribute "value".

While it means select the attribute "value" from the "input".

About iterations.

In automatic mode, PURE detects if the JSON property is an array and then repeat the selected element for each entry in this array.

In manual mode, a directive is something like { 'tbody tr' : 'row <- rows' }.

Iteration example

To keep the number of special notation to a minimum we enabled the use of JS functions as directive:

i.e:

{ 'div.error[style]' : 

function(arg){

return arg.context.error ? 'display:block' : 'display:none';

}

} 

It can be an anonymous function as above, or any available JS function.

Here you have the freedom of javascript.

Another option available is the inclusion of other templates within another one.

If the template is itself, it gets recursive.

Recursive call example

And can be as well the solution for conditionals.

PURE is compiling the HTML + directives into a javascript function.

If the compilation is on the fly, there is an eval executed on the page.

There is another option (That I'll use on our project when it will be in production). 

You can compile all your HTML to JS functions, then get those functions with a runtime.

You can save the runtime in a JS file and add a <script> tag to it on your page.

No eval are run in your page when you render. You don't have to link the pure.js file in production.

You save some overhead compilation time. 

The JS file will be cached and you can bundle it with others JS and gzip it.

Here's some information about compilation

[Kris Zyp]

One particular way that I have used HTMLUnit, that might be of interest
to the group, is that I have used it for doing server side rendering of
browser-targeted JavaScript-based templating (like JMVC and PURE) for
search engines. Obviously one of the issues with pure client side UI
generation and templating is that search engines don't have any HTML
(since they don't understand JavaScript) to index. In the past, I have
setup a system where I check the user agent of incoming requests and if
it is a user agent that doesn't support JavaScript, than HTMLUnit will
act as the browser, execute all the JavaScript (and browser-side
templating) on the server, and send the rendering HTML to the user agent
(search engine usually). Too see an example you can hit
http://www.xucia.com. You will find that there is virtually no HTML, it
is all rendered in JavaScript. However, you can see it from the
perspective of search engine by forcing server side rendering with
HTMLUnit: http://www.xucia.com/?mode=html&noscript=true. In this case
all the HTML is fully rendered. The templating engine I used on these
sites is abandoned technology, but I think it is a similar paradigm to
the templating engines like PURE, JMVC, and Dojo's DTL engine, which
could probably all leverage this.
Kris

> justin...@gmail.com <mailto:justin...@gmail.com>
> AOL: jusbarmey
>
> >

[Justin Meyer]

Wow, I've not thought of it for search.  Are you using a Rhino based server?  Can HTMLUnit be easily detached so a load balancer could direct to the Rhino server where appropriate, but to the other server (Rails/Java/Php, etc) for normal requests.

Great idea.

Justin

AOL: jusbarmey

[Kris Zyp]

In the situation where I used HTMLUnit (for the server used on
www.xucia.com) it was integrated directly into the server (which was
Java based). But I do think it would be way better to have a standalone
HTMLUnit server that could do this rendering and be used like you
describe where something directs the requests and the main server
doesn't need to be Java. It seems like such a tool would be beneficial
to SOFEA/TSA developers since it would deal with one of the common
objections to this approach. I have had others mention interest in such
a system, but I don't know that this is high enough on my list of todos
that I will be getting to it anytime soon, so if anyone wants to do it,
feel free ;).
Thanks,
Kris

> <http://www.xucia.com/?mode=html&noscript=true>. In this case

> all the HTML is fully rendered. The templating engine I used on these
> sites is abandoned technology, but I think it is a similar paradigm to
> the templating engines like PURE, JMVC, and Dojo's DTL engine, which
> could probably all leverage this.
> Kris
>
> Justin Meyer wrote:
> > No, I haven't. Thanks for sharing. I'll read up about it and share
> > it with the group.
> >
> > On Sat, Oct 25, 2008 at 8:14 AM, Kris Zyp <kri...@gmail.com
> <mailto:kri...@gmail.com>

> > <mailto:kri...@gmail.com <mailto:kri...@gmail.com>>> wrote:
> >
> >
> >
> > > But, I've just recently (1 hour ago) added what could be
> it's best
> > > features - command line testing. Using a beefed up version of
> > Resig's
> > > simulated FF2 environment for Rhino, any javascriptmvc
> application
> > > will run its unit tests automatically in a command line.
> > Have you tried HTMLUnit? It is built on Rhino and provides
> pretty
> > extensive browser environment emulation, and I think even
> has decent
> > support for emulating the different browsers out there.
> > Kris
> >
> >
> >
> >
> >
> > --
> > Justin Meyer
> >
> > Jupiter IT Solutions
> > 847-924-6039
> > justin...@gmail.com <mailto:justin...@gmail.com>

> <mailto:justin...@gmail.com <mailto:justin...@gmail.com>>

> > AOL: jusbarmey
> >
> > >
>
>
>
>
>
> --
> Justin Meyer
>
> Jupiter IT Solutions
> 847-924-6039

> justin...@gmail.com <mailto:justin...@gmail.com>
> AOL: jusbarmey
>
> >