jQuery

[Sam Minnee]

Hi everyone,

JavaScript in SilverStripe is in disrepair.  There are a lot of good concepts, but we're held back by being stuck with a forked, old version of Prototype.  SilverStripe developers need to be able to create leading-edge websites and web applications, and we can't let the framework continue to be a hindrance.

Most of the people who have hacked at a deep level with SilverStripe, particularly with the CMS interface, would agree.  The solution that has previously been discuss has been an upgrade to prototype 1.5 or 1.6.  However, this is an important decision, and I don't think it's wise to go with the first option that occurs to us.  Mark Rickerby, Ingo Schommer and I have given a lot of thought over the last few weeks to the future of our JavaScript development.  

We think that jQuery is the best base platform for development going forward.

We considered Dojo, YUI, ExtJS, Mootools, Protoype/Scriptaculous, and jQuery.  However, it quickly became apparent that the two real contenders were Protoype/Scriptaculous and jQuery.  (Note that Prototype/Scriptaculous work as a team, kind of like GNU/Linux) 

These are the main reasons why we think that jQuery is the way to go:

 * There is a lot more active development happening on jQuery when compared to Prototype/Scriptaculous.

 * jQuery has a more organised and active plugin-development community, with a plugin repository on the jQuery website.

 * jQuery is a more focus on UI development, whereas Prototype seems more focused on language extensions.  UI development is what we, as a project, need.

 * jQuery is about 1/3 the size of Prototype/Scriptaculous, when factoring in gzipping & minification.

 * jQuery client-side performance is better.

 * Since we're currently running on a hacked, old version of prototype, it's going to be easier to incrementally roll out a migration to jQuery than it would to incrementally roll out an upgrade of prototype.  And incremental roll-out is going to important in actually seeing this through to fruition.

The decision to "move to jQuery" will mean the following things:

 * Core SilverStripe technology will be rewritten based on jQuery, incrementally.

 * New front-end systems will be created in jQuery.

We also need to be clear about what it doesn't mean:

 * We're not going to force you to use jQuery to write your applications.

 * Existing applications that you've written that rely on prototype won't break horribly.

We would like to take this opportunity to get feedback from the wider development community about this decision, particularly from those of you who have heavily customised the CMS interface and form validation system.  A major concern is to ensure that your custom modules can be made to work with upgraded versions of the CMS as easily as possible.  To do this, we need to know the ways in which everyone has hooked into the javascript parts of the CMS and form validation system.

In addition to the move to jQuery, we would like to put some effort into improving the "sapphire side" of the javascript infrastructure:

 * A nested requirements parser for javascript files, so that you don't need to list every single dependency as a separate Requirements::javascript() call.

 * A javascript concatenator/minifier, so that all of the script files get mashed into a single download.

 * More standardised support for transmitting JSON and HTML-snippets to the client, so that we don't need to make this up from scratch every time.

Sam

Sam Minnée | Chief Technology Officer

SilverStripe

http://www.silverstripe.com

Office: +64 4 978 7334

Skype: sam.minnee

[Jamie Neil]

Sam Minnee wrote:
> We considered Dojo, YUI, ExtJS, Mootools, Protoype/Scriptaculous, and
> jQuery. However, it quickly became apparent that the two real
> contenders were Protoype/Scriptaculous and jQuery. (Note that
> Prototype/Scriptaculous work as a team, kind of like GNU/Linux)

Can I ask why you discounted the other frameworks (particularly YUI)?

--
Jamie Neil <ja...@gpmd.co.uk>
GPMD <http://www.gpmd.co.uk>

[Sam Minnee]

YUI, Dojo and ExtJS provide more mature widget libraries, but our
primary requirement was a robust framework for building our own
interfaces using unobtrusive javascript. We needed good libraries
that provided basic features really well:

* ajax interaction
* progressive enhancement by applying behaviours with CSS selectors
* drag & drop
* visual effects
* form validation

If you look at some of the leading-edge web 2 UIs out there, very few
of them get their impact by dropping in sophisticated widgets.
Instead, they carefully applying the above elements to the particular
interface challenge at hand.

Although YUI and Dojo have some of these facilities, Protoype and
jQuery were more focused on building the sort of library that we were
after. ExtJS has the richest set of UI widgets; however, it is
difficult to use in an unobtrusive manner. That said, ExtJS can work
on top of jQuery, and we're going to look at the possibility of using
these rich widgets in places where unobtrusive javascript isn't so
important, such as within the CMS.

[Michael Gall]

Alright, I'm pretty much completely against the rewriting of anything.

I don't consider the state of Javascript to be in too bad a condition. I have just over the past week or so been reapplying the upgrade of Prototype 1.5 to the 2.2.2 branch and the amount of code changed is very minor. The tree controls are the biggest problem, but in saying that it isn't too major.

This said, the upgrade to 1.6 (I haven't tried this) should not be too bad. It doesn't break _too much_ compatibility and in general will be replacement for 1.5. The biggest issue as far as I'm concerned is Behaviour.js, it hasn't been actively maintained and I think we would need to give it some love and care, but this would be a welcome contribution to the javascript community I'm sure.

There is a few points I'd like to address:

 * There is a lot more active development happening on jQuery when compared to Prototype/Scriptaculous.

I'm not sure this is entirely true. Prototype (let's just include Scriptaculous in that) does move slower, but in our case this is not necessarily a bad thing.

 * jQuery has a more organised and active plugin-development community, with a plugin repository on the jQuery website.

Sure. This is definitely a good thing.

 * jQuery is a more focus on UI development, whereas Prototype seems more focused on language extensions.  UI development is what we, as a project, need.

I don't agree with this at all. I see Silverstripe as a platform and to this end I think Prototype is far better suited for this. My belief is that apps built with jQuery don't end up being as extensible unless extensions are used extensively.

 * jQuery is about 1/3 the size of Prototype/Scriptaculous, when factoring in gzipping & minification.

I'm not entirely sure this is a huge issue, but obviously worth putting on the ledger.

 * jQuery client-side performance is better.

Sure.

 * Since we're currently running on a hacked, old version of prototype, it's going to be easier to incrementally roll out a migration to jQuery than it would to incrementally roll out an upgrade of prototype.  And incremental roll-out is going to important in actually seeing this through to fruition.

I think this is 2 issues that need to be addressed. First of all, the fact that Prototype has been forked (and tinyMCE) is constantly a pain in the ongoing development. I think this needs to be addressed first and foremost - 3rd party libraries must be left as 3rd party maintained, and any modifications need to be made separately and however the library suggests it be extended.

 

In addition to the move to jQuery, we would like to put some effort into improving the "sapphire side" of the javascript infrastructure:

 * A nested requirements parser for javascript files, so that you don't need to list every single dependency as a separate Requirements::javascript() call.

 * A javascript concatenator/minifier, so that all of the script files get mashed into a single download.

Obviously this is a differenet issue, buut definitely something important to do going forward.

 * More standardised support for transmitting JSON and HTML-snippets to the client, so that we don't need to make this up from scratch every time.

I think this needs to be assessed as part of the library. Automating this could pay off majorly for the platform.

Michael Gall

--

http://wakeless.net

[Nathan Cox]

I won't pretend I'm an expert on JS libraries or SilverStripe's existing javascript but I'd love to see SS using jQuery...it's always felt better to work with than Prototype.  Flyingmonkey.co.nz's contact form even runs off jQuery using a custom jValidator class on the PHP end.  Of course you can have different libraries running the front end and the admin site but having it all jQuery at some point would be good.

Nathan

[Sam Minnee]

> Alright, I'm pretty much completely against the rewriting of anything.
> I don't consider the state of Javascript to be in too bad a condition. I
> have just over the past week or so been reapplying the upgrade of Prototype
> 1.5 to the 2.2.2 branch and the amount of code changed is very minor. The
> tree controls are the biggest problem, but in saying that it isn't too
> major.

Rewriting isn't a task to be taken lightly, it's true. The question
is "will the benefits justify the significant cost?"

The most important thing to decide is what is going to be better for
the next 2-3 years, not what's going to be better for the next 2-3
months. If the better solution is going to take more up-front work,
it is still quite likely the better way to go.

The real question, I think, is not "how much work will it be to
upgrade to Prototype 1.6", but "Is jQuery going to be better". At the
heart of it, it seems as though you are pushing Prototype over jQuery
because SilverStripe is a platform.

I completely agree that SilverStripe is a platform, in two senses:
(1) A collection of existing applications that can be easily tailored
to a variety of situations. The CMS, GenericDataAdmin, and complex
field types such as ComplexTableField fall into this category.
(2) A toolbox to rapidly put together bespoke applications. The
Sapphire framework falls into this category.

You can make a strong case that (1) in particular would benefit from
Prototype over jQuery, because it provides a way of specifying a set
of behaviours that can then be customised using inheritance. But
there are two counters to this argument:

1) In our experience, it's been more a theoretical benefit than a real
benefit. We tried this in the CMS and it's just a mess. It fails to
provide a rapid, intuitive means of re-creating "CMS-like"
functionality. A more successful strategy in JS-land is a component-
based architecture, where you customise the experience by re-arranging
the components rather than overloading their behaviour.

2) Low Pro for jQuery (http://www.danwebb.net/2008/1/31/low-pro-for-
jquery) provides a solution to this problem, meaning that we can still
allow for tailoring-by-subclassing while using jQuery. We thereby get
the benefits of using jQuery (which I believe are substantial) without
sacrificing major disadvantages.

But, quite apart from the jQuery vs Prototype discussion, is this idea
of re-structuring the CMS as a set of loosely-coupled, reu components.

[Sam Minnee]

> But, quite apart from the jQuery vs Prototype discussion, is this idea
> of re-structuring the CMS as a set of loosely-coupled, reu components.

Gar! I clicked the send button mid-sentence! :-P As I was saying...

Quite apart from the jQuery vs Prototype discussion, is this idea of
re-structuring the CMS as a set of loosely-coupled, reuseable
components. There are hints of this:

* it was successfully applied to the tabstrip
* we started doing this with the tree, but then got lost in the
business of just making the CMS work under a very tight deadline and
so lost sight of reusability
* there is a bunch of "general form interactions" that are, frankly,
poor partial implementations of ideas that have been much more
reliably implemented in standard libraries like livequery

We're trying to make a collection of tools for building web
applications more easily. Pursuing this model is going to be very
helpful in achieving this goal. It's also going to involve a lot of
rewriting whether we stay with Prototype or jQuery.

The other problem spot in the current system is form validation. This
has been causing measurable pain on complex projects that we're trying
to do. Rather than put effort into patching our home-grown validation
system, we want to migrate to a system that has got 3rd party
developers maintaining it.

[Michael Gall]

The question of whether the tradeoffs of moving to jQuery rather than staying and fixing the current implementations basically boils down to the question, is jQuery _THAT_ much more productive than Prototype. To make the equation work in favour of jQuery it really does need to be far more productive due to the huge amount of code that is written already, and developers having learnt the current front-end APIs and having to relearn.

There are 2 major problems on the JS side of the CMS, first is the fork of Prototype. (and the fork of TinyMCE), this is obviously the biggest headache for upgrading, but I'm confident we could work around it quite quickly. There are a few functions that have been added that we can implement on top of Prototype 1.6 (or 1.5 if we so desired). The other problem is a exactly as you've described, half finished libraries that have been hacked in (mainly the Tree). Don't get me wrong, I'm not having a crack, cause it works and it's shipped.

What I'm saying is changing the library is NOT a golden bullet. There is nothing that you can implement using jQuery that can't be implemented using Prototype. As far as my knowledge goes the thing that isn't missing, but doesn't exist in the core of jQuery is the Class inheritance structure. Often the equivalent jQuery code "feels" more succinct, but in reality the difference is often only 1 command. I don't want to start a pissing context between the 2 syntaxes. So I've deleted my conceited example.

--

Is there a plan for the form validation, I haven't really considered this within SS so can't really make a huge comment. Form validation is something that has been done a hundred times and using a popular library is definitely important so as it is maintained into the future.

Michael

[Ingo Schommer]

hey michael,

you're right, the libraries themselves are not as different to validate serious rewriting efforts.

one big plus we had on the board for jQuery is a rather large centralized plugin repository,

which enables us to build (mostly website-related) widgets in a quick and consistent manner.

its ui-project (http://ui.jquery.com/) is currently more refined than prototypes similiar efforts (http://prototype-ui.com).

the bigger picture here is that we start off with a plan for rearchitecting the cms-specific javascript,

and website-functionality like form validation into more self-contained components

that can be used to build great web applications (like we started with the tree control).

this is no "side-effect" of choosing jquery over prototypejs.

in that light, choosing a new base library is a secondary choice (and comparable workload on the long run).

we're gonna see more use of external components within the cms module, where we're also considering

higher level frameworks such as extjs/yui/dojo. this means we're not actually rewriting

every piece of javascript we currently have, rather than trying to replace it with

existing opensourced components.

ingo

-------

Ingo Schommer | Senior Developer

SilverStripe

http://silverstripe.com

Phone: +64 4 978 7330 ext 42

Skype: chillu23

Level 3, 97-99 Courtenay Place

Wellington, New Zealand

[Sam Minnee]

Thanks for that, Ingo. I'm broadly in agreement with what you've
said.

There is another concern, however: SilverStripe have made the decision
to convert/upgrade the core modules, but Michael might not want to
have to upgrade his CMS customisations just yet.

Ideally, we would want existing prototype-based extensions to the CMS
to continue working with a minimum number of changes. jQuery's "no
conflict" feature is a good starting point for this. For this to be
feasible, we need to know what hooks into the CMS JavaScript have been
made.

Michael - care to elaborate? :-) In essence, we want to document all
of the undocumented APIs that have been used.

> SilverStripehttp://silverstripe.com

[Mat Weir]

Are you guys looking at having this changeover completed before GSOC 2008
begins?

Mat Weir

[Sam Minnee]

We are certainly aiming to have all the new infrastructure in place
for the gsoccers to make use of.

[Michael Gall]

one big plus we had on the board for jQuery is a rather large centralized plugin repository,

which enables us to build (mostly website-related) widgets in a quick and consistent manner.

its ui-project (http://ui.jquery.com/) is currently more refined than prototypes similiar efforts (http://prototype-ui.com).

I think it's important to put this in perspective. Sure, ui.jquery.com is better organized at the moment, but from a quick browse through Ajaxian, there seems to be about the same amount of developments in both the worlds of jQuery and Prototype. This coupled with the fact that your stated outlook is for the next 2-3 years, it's reasonable to suggest that something like prototype-ui.com will become an extremely good central repository. I think it's important to mention that a good portion of the ui.jquery.com stuff is in Scriptaculous.

the bigger picture here is that we start off with a plan for rearchitecting the cms-specific javascript,

and website-functionality like form validation into more self-contained components

that can be used to build great web applications (like we started with the tree control).

this is no "side-effect" of choosing jquery over prototypejs.

I don't quite get the "no side-effect" comment here?

in that light, choosing a new base library is a secondary choice (and comparable workload on the long run).

we're gonna see more use of external components within the cms module, where we're also considering

higher level frameworks such as extjs/yui/dojo. this means we're not actually rewriting

every piece of javascript we currently have, rather than trying to replace it with

existing opensourced components.

So what exactly is the plan? How are you planning on implementing these changes?

Michael
 

[Mark Rickerby]

Thanks for your feedback Michael!

On 18/03/2008, at 11:23 AM, Michael Gall wrote:

one big plus we had on the board for jQuery is a rather large centralized plugin repository,

which enables us to build (mostly website-related) widgets in a quick and consistent manner.

its ui-project (http://ui.jquery.com/) is currently more refined than prototypes similiar efforts (http://prototype-ui.com).

I think it's important to put this in perspective. Sure, ui.jquery.com is better organized at the moment, but from a quick browse through Ajaxian, there seems to be about the same amount of developments in both the worlds of jQuery and Prototype. This coupled with the fact that your stated outlook is for the next 2-3 years, it's reasonable to suggest that something like prototype-ui.com will become an extremely good central repository. I think it's important to mention that a good portion of the ui.jquery.com stuff is in Scriptaculous.

The point is that "will eventually become" is not quite enough for our purposes, we need solid evidence and demos. While we are indeed taking at a long wave view, to effectively proceed, we need an external UI project to be organized right now, not in a few months or years time.

jQuery is a good fit as an open source ecosystem and its programming model is at the right level of abstraction for our purposes. We believe the majority of our users will benefit from the tighter integration that Sam is talking about.

However, we want to be clear that developers using SilverStripe can choose any JS framework that suits their needs. We're not trying to lock anyone down to a specific javascript layer - but we do intend to provide significant 'add-on' functionality that will be jQuery based. You could think of it as a choice between duplo, basic, and technic lego. We'll provide lots of duplo and basic blocks to quickly put great apps together, but these blocks will be totally compatible with the more complicated moveable parts of the technic, should you want to go deeper into customization or unique behavior.

the bigger picture here is that we start off with a plan for rearchitecting the cms-specific javascript,

and website-functionality like form validation into more self-contained components

that can be used to build great web applications (like we started with the tree control).

this is no "side-effect" of choosing jquery over prototypejs.

I don't quite get the "no side-effect" comment here?

Modules and Sapphire components that have no specific integration in the CMS are logical places to start. We can add jQuery support to these components without affecting the old components already running on Prototype.

 this means we're not actually rewriting

every piece of javascript we currently have, rather than trying to replace it with

existing opensourced components.

So what exactly is the plan? How are you planning on implementing these changes?

We're taking things one step at a time :)

Regards,

Mark

Mark Rickerby | Core Developer

SilverStripe

Office: +64 4 978 7330

Email: ma...@silverstripe.com

Web: http://www.silverstripe.com

Skype: maetl_