GSOC - a Library to compile and compress Javascript and CSS files

[kavith Thiranga]

Hi All,

I have submitted my proposal[1] for the idea[2] of implementing a Library to compile and compress Javascript and CSS files. And I have implemented a prototype library with a number of working functions[3]. I am waiting for your feedback. Thank you.

[1] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/kaviththiranga/1
[2] http://docs.joomla.org/GSOC_2012_Project_Ideas#Project:_Library_to_Compile_and_Compress_Javascript_and_CSS_Files
[3] https://github.com/kaviththiranga/joomla-platform/tree/JCompress/libraries/joomla/compress

[kavith Thiranga]

Hi All,
I am still waiting for some feedback as I have some time to make modifications if needed. I would really appreciate any comments or feedback.
Thank you.

[Matt Thomas]

Hello Kavith,

I'm not a platform developer, so I can't fully comment on some of the technical specifics of the code.

Overall, the proposal look good. It is well thought out and addresses the core issue. Following are a few thoughts that came to mind while reading it:

• What are your thoughts on caching the combined/minified files?
• Would this library allow for excluding files from the combination/minification?
• Would this library support the developer to indicate the order of files when combined/minified?
• Would you only combine/minify files that use JDocument, or would you also search for files that use <link> and <script> tags? For example, see lines 37 - 40 of https://github.com/joomla/joomla-cms/blob/master/templates/beez_20/index.php#L37

These are not requirements, or changes that you need to make, but a few thoughts that came to mind.

Also, you may want to look at http://jproven.com/extensions/jbetolo and http://blank.vc/ as they have some of these features in them. Their code may inspire  something within yours.

I hope that helps.

Best,

Matt Thomas

Founder betweenbrain™

Lead Developer Construct Template Development Framework

Phone: 203.632.9322
Twitter: @betweenbrain

Github: https://github.com/betweenbrain

[Jeremy Wilken]

Thank you for your proposal! I have a couple of thoughts to add to Matt's.

1. Caching: You will want to consider how to use caching for performance, and explain how you plan to handle that. For example, different pages load different css and javascript, so you have to have a means to manage the cache for various pages.
2. There are several system plugins out there that attempt to do this, some better than others, that you should look at just for ideas on how its been done so far. I haven't found any of them to be perfect, but it can be good research.
3. Matt asked about getting files that are also embedded directly in html output (not registered in JDocument). Ideally all files should be registered through JDocument, but often in templates those are placed carefully. For example, jQuery is often set above <jdoc:include> and then a jQuery.noConflict(); statement is executed in case Mootools is also being loaded. It would be a challenge to properly grab the css and js files from the html output, so if you consider this I would suggest thinking about having it be an option (a standard setting, and a more 'aggressive' setting). Sometimes script tags need to be loaded at the bottom of the page for example, so that makes it a much more difficult idea to implement.

[kavith Thiranga]

Hi Matt,
  
  First of all, thank you very much for your feedback. I will clarify my ideas regarding your points.

•    I am thinking of cashing the files as it will increase the performance, But I haven't yet come up with a plan on doing it.
•    Yes. This library allows excluding files as my plan is to give the chance to developer to decide whether to compress/combine or not (This can be changed upon mentors' ideas). For an example,

     developers can compress a js/css using JCompress::compressFile() method and use url of compressed_file when using   
     addScript() and addStyleSheet() methods on JDocument object.

class TestApp extends JApplicationWeb{

...
    
$original_file = JPATH_SITE.'/css/main.css';
$compressed_file = JPATH_SITE.'/css/main_compressed.css';

JCompress::compressFile($original_file, $compressed_file, array("overwirte" => true) );


$this->document->addStyleSheet($compressed_file);

...

}

Or they can use JCompress::compressCss() / JCompress::compressJs() with addStyleDeclaration() / addScriptDeclaration() methods of JDocument object if they use small stylesheets/javascripts and prefer not to write files.

...
$this->document->addStyleDeclaration( JCompress::compressCss($uncompressed_code));
....

I will update my proposal also regarding these ideas. Hope you got my idea.
Thank you.

--
Kavith Thiranga Lokuhewage,
Undergraduate,
BSc (Hons) Software Engineering,
Staffordshire University, UK.

[kavith Thiranga]

Hi Jeremy,

Thanks for your feedback. As I have mentioned in my last reply to Matt, I have not yet come up with a plan on cashing. As you proposed I will look in to system plugins you mentioned about and do a research on that.

Regarding your 3rd point- Because of the difficulty in implementation of automatic compression, I decided to implement in a way, that developer can manually decide which code to be compressed/combined. You can see my approach on that in my last reply. The idea you mentioned about making it a setting also makes sense. I will research on that also. Thanks all for your valuable feedback. It is really helpful for me to improve my proposal.

regards,

[piotr_cz]

Proposal looks good to me.
I'd be interested in knowing same things as Matt and Jeremy:

- Integration with JDocument.
I'd research options to automatically compress all assets when using
solely native methods $this->document->addStylesheet('/styles.css').
There would be switch in Joomla global configuration (compression on/
off, ratio, exlcusions, etc). This way compression would be
transparent and available for extensions without changes in code.

You can also take a look at JHtml::script and JHtml::stylesheet which
are more intelligent wrappers for addScript and addStylesheet.

As for non-platform ways to add assets (manually using script and link
tags in template), I guess It's worth to make a small research if it
does make sense to bother at all.
I mean, when developer opts to add assets this way, this theoretically
means that he/she has some reasons to go around native methods (ie.
loading assets asynchronously with js loader like require.js or
labjs).
What I've usually seen is simply the convenience of using familiar
html way over framework methods but well, in my opinion this is wrong.


It would be nice to have an option to load javascript files at the
bottom of html, but then there is a risk that developer may include js
code inline in template html which is dependent on some libraries.


- As for caching,
I've managed to integrate Minify (http://code.google.com/p/minify/) as
a Joomla library. Seems like caching is pretty decent there (caching
files based on required files and proper header responses).


This is quite interesting project and in my opinion Joomla was missing
this functionality for long time. There were few attempts but I had
problem with each.
I think you are doing well so far.

On Apr 4, 6:16 pm, kavith Thiranga <rc404m...@gmail.com> wrote:
> Hi Matt,
>
>   First of all, thank you very much for your feedback. I will clarify my
> ideas regarding your points.
>

>    -    I am thinking of cashing the files as it will increase the

>    performance, But I haven't yet come up with a plan on doing it.

>    -    Yes. This library allows excluding files as my plan is to give the

> >    - What are your thoughts on caching the combined/minified files?
> >    - Would this library allow for excluding files from the
> >    combination/minification?
> >    - Would this library support the developer to indicate the order of
> >    files when combined/minified?
> >    - Would you only combine/minify files that use JDocument, or would you

> >    also search for files that use <link> and <script> tags? For example, see
> >    lines 37 - 40 of

> >    https://github.com/joomla/joomla-cms/blob/master/templates/beez_20/in...

>
> > These are not requirements, or changes that you need to make, but a few
> > thoughts that came to mind.
>
> > Also, you may want to look athttp://jproven.com/extensions/jbetoloand

> >http://blank.vc/as they have some of these features in them. Their code

> > may inspire  something within yours.
>
> > I hope that helps.
>
> > Best,
>
> > Matt Thomas

> > Founder betweenbrain <http://betweenbrain.com/>™
> > Lead Developer Construct Template Development Framework<http://construct-framework.com/>

> > Phone: 203.632.9322
> > Twitter: @betweenbrain
> > Github:https://github.com/betweenbrain
>

[Matt Thomas]

Hi Kavith,

I like the idea of $this->document->addStyleDeclaration( JCompress::compressCss($uncompressed_code)); for granular control over which files gets compressed. As piotr_cz mentioned, automatically compressing all assets when using native methods would be fantastic. You could do something in global config on the CMS side, but being a platform library, you would need to have some sort of method to invoke it globally as well.

If it were me, this is how I'd approach it, and in this order:

1. Implement compression of assets that use JDocument.
2. Implement combining of previously compressed assets that use JDocument.
3. Look at caching of assets previously compressed and combined.
4. Evaluate compressing and combining assets loaded using non-platform method.
5. Evaluate the possibility of loading specifically declared JavaScript at the end of the document.

I would start by focusing on JDocument [#1, #2] as it handles the ordering of files and is the native method that everyone, in my opinion, should be using. I would approach the project in this order to ensure that you can achieve some good "wins" right away. In my opinion, if you only succeed to accomplish #1 and #2, you have done well. 

Caching [#3] will be more difficult to achieve, as Jeremy mentioned, that different pages load different assets. 

Targeting of assets that don't use native methods [#4] will take some time to find a performant way of achieving this. You may even need to implement a way to turning this on or off for performance reason.

#5 is an added bonus that some of us would love, but may require rethinking the way assets are loaded as JDocument, as far as I know, doesn't support a method to "tag" which assets get output where.

Hope that helps!

Best,

Matt Thomas

Founder betweenbrain™

Lead Developer Construct Template Development Framework

Phone: 203.632.9322
Twitter: @betweenbrain

Github: https://github.com/betweenbrain

[kavith Thiranga]

Hi Matt, Jeremy and piotr_cz,

Thank you all for your valuable feedback. According to your ideas, I have understood that my approach of using library has several drawbacks.

I am going to add some new contents to proposal under topic "Better approach on using library" . I will notify you after I am done.
Thank you.

Kavith

[kavith Thiranga]

Hi all,

It will be really helpful if someone can suggest me a way of globally invoking compress function, so it will be able to compress assets registered on JDocument using addStyleSheet() / addScript() methods. Something like overriding addStyleSheet() / addScript() methods or getting all registered assets via JDocument fields and compress them.
In addition I am thinking of keeping  my previously described approach also enabled in case, the developers want to load assets using traditional HTML way (using link tags)  rather than using native methods. So developers can manually compress/combine when loading assets in html way. What are your opinions on that.
I would appreciate some early replies as I am running out of time.
Thank you.

[kavith Thiranga]

Hi all,

I have updated my proposal and added some suggestions with your ideas. Please have a look.

Thank you.

[Jeremy Wilken]

I like the idea of $this->document->addStyleDeclaration( JCompress::compressCss($uncompressed_code)); for granular control over which files gets compressed. As piotr_cz mentioned, automatically compressing all assets when using native methods would be fantastic. You could do something in global config on the CMS side, but being a platform library, you would need to have some sort of method to invoke it globally as well.

If you tie compression into the JDocument class, then you could add an optional parameter to the methods like addStyleDeclaration($code, $compress = true) {} so its on or off by default. You could place a configuration item into the global configuration perhaps that toggles compression (locate it with the cache for example). Then you check the global setting first, and handle it appropriately.

From what I see, this I think this should be part of the JDocument package rather than its own, but that would be up to the assigned mentor to advise on properly. If you have it as a separate package, then we start to worry about dependencies if you were to use my approach.

If someone adds a <script> or <link> tag to the page output, you would have to wait until the page is rendered and then parse it to find them. The tricky thing about that is you then have to make assumptions about why the html code was used, so this is why I suggest it not be done by default. If you load a script or stylesheet through the html tags, Joomla probably shouldn't try to do anything about that. It is up to the developers to properly compress in that situation. You could provide the option to try to intercept those scripts or stylesheets and compress them, but it should certainly not be on by default.

[CirTap]

Am 05.04.2012 15:00, schrieb Matt Thomas:
>
> I would start by focusing on JDocument [#1, #2] as it handles the ordering
> of files and is the native method that everyone, in my opinion, should be
> using. I would approach the project in this order to ensure that you can
> achieve some good "wins" right away. In my opinion, if you only succeed to
> accomplish #1 and #2, you have done well.

Actually the place to look for is JDocumentRenderHead::render() if we're
talking about HTML output (JDocumentHTML).
That's where the <head> element and everything inside is built.

You can "hijack" this class by pre-registering your own version of
JDocumentRenderHead with JLoader and override render() to do "The Right
Thing"(tm).
Then inspect the data in the $head array and do some magic with it.

From the CMS perspective, render() has to return a string that fits
where <jdoc:include type="head" /> appears. If JDocumentHTML(!) builds
the template it'll then call your renderer class instead.

That's at least how I tackled the issue and also invented a
JDocumentRenderTow (<jdoc:include type="toe" />) that takes care of
rendering scripts that should appear at the end of <body>, i.e.
analytics and stuff like that.

My 2ct.

Have fun,
CirTap

[kavith Thiranga]

Hi Jeremy and CirTam,

Thanks for your feedback. I will research on what you have suggested and mean time I will update my proposal with those rough ideas about implementation as I haven't got enough time to describe it more deeply. I will notify you after I have updated.

Thank you very much.

[kavith Thiranga]

Hi all,
I have updated my proposal. Please have a look under the topic - Better improved approach of using the library.

Thank you.

[piotr_cz]

Great discussion, everybody.

I really think that there should be a global switch for compression.
The decision about compression (and it's options) should be made by
website administrator, not by each extension developer. Jeremy pointed
out to good example: cache settings.
In my experimental implementation I'm loading Joomla configuration
file and using some of it's settings to configure Minify (debug,
caching, cachetime). When debug is on, compression is disabled to
allow easier debugging of javascript files (actually same thing is
happening now: with Joomla debug set to on, platform tries to load -
uncompressed files).

Still I'd like to have direct access to compression methods in case
I'd like to load assets asynchronously (for example using javascript
client).

I would keep JCompress as own library package, but modify JDocument to
use it when compression is on. You may create own joomla-platorm
branch.
JCompress could be used in future for other data beyond Javascript and
Css.

[Matt Thomas]

Hi Kavith,

It looks very good to me. It has sufficient detail to tell me what you plan on doing, as well as the challenges you face. This project would be a win for me if you can succeed at implementing a good compression and combination library for assets that use native methods. It can always be expanded and refactored once the foundation is laid.

@piotr_cz - keep in ind that this is a platform project, so global config, or some other interface may not exist. That is not to say that one couldn't be developed to tap into his library for the cms.

Best,

Matt Thomas

Founder betweenbrain™

Lead Developer Construct Template Development Framework

Phone: 203.632.9322
Twitter: @betweenbrain

Github: https://github.com/betweenbrain

[kavith Thiranga]

Thanks all of you for your valuable feedback to improve my proposal.
Thank you very much.

[kavith Thiranga]

Hi all,

Even though, application period is over, I really think we can continue the discussion regarding this idea. I am looking forward to your feedback and ideas towards this. And also, I really appreciate those who gave their valuable suggestions to improve my proposal.

Thank you.