[Dojo-interest] How to create smallest possible dojo build?

[Arech]

Hi!

I had been using Dojo starting from v1.1 and like it very much, but at some
point about v1.4 I had to stop to use it due to my client's complaints on
its size and speed.
Now when v1.7 released, it's stated by release PR, that it is possible to
make Dojo's core to be smaller than 4Kb (gzipped). That's nice, but how to
achieve it? I've read all related tutorials on the site, but they are all
about v1.6 and knows nothing about "Dojo nano". Then, I've tried to do a
custom build, but the result had the same size, as script on /download page,
- 117kb or about 34kb gzipped, - and that's completely unacceptable (I even
started to wonder, if they included famous JavaScript x86 emulator in the
core...).

So, the question is, how to create the smallest possible dojo build? How to
make that "dojo nano 4kb gzipped"?

Thanks

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3554327.html
Sent from the Dojo Toolkit mailing list archive at Nabble.com.
________________________________________________________
Dojotoolkit: http://dojotoolkit.org
Reference Guide: http://dojotoolkit.org/reference-guide
API Documentation: http://dojotoolkit.org/api
Tutorials: http://dojotoolkit.org/documentation

Dojo-i...@mail.dojotoolkit.org
http://mail.dojotoolkit.org/mailman/listinfo/dojo-interest

[ben hockey]

4kb is some nice marketing - which is actually not quite right, after
squeezing every last byte out it's over 4k but under 5k. it is possible
to build a file called dojo.js that gzips to ~4kb BUT all it will do is
load other modules. it is not possible to build an entire application
in 4kb! also just to clarify... dojo has a collection of modules called
"base" and a collection called "core". this 4kb does not include any of
those modules - it is just the "loader".

117kb is about what i'd expect for a custom build of something that has
a few layout widgets and some form widgets. as an example, if you do a
build of dojox/mobile i think it comes in around 150k.

see http://comments.gmane.org/gmane.comp.web.dojo.devel/15941 for some
discussion and details of a profile that is needed to produce the "nano"
build. the BIG key to making any of your builds smaller with 1.7 (as
discussed in the previous link) is to use

layerOptimize: 'closure'

this removes dead code that is not removed with shrinksafe. one of the
features planned for 1.8 is a parser that will build an AST and (among
other things) allow us to remove dead code without needing to
specifically use closure for optimization.

ben...

[ben hockey]

i need to correct myself - i just noticed that 3,971 bytes was the
smallest it got down to.

ben...

On 12/2/2011 6:00 AM, Arech wrote:

[Arech]

Wow, Ben, thanks a lot for the answer! Discussion on linked page looks very
promising and I'll try to use any helpful info from there.

BTW: looks like you know a little about plans and roadmap of Dojo
developing. Want to complain a little:)) The only thing that stops me from
using and engage others to use it is that the Dojo's core is very heavy and
contains a lot of unneeded at 90% of time stuff. This heaviness slows down
sites especially on lightweight netbooks or tablet PCs. I'm not alone with
that problem - every time I say to someone, that I'm a big Dojo fan, I hear
something like "You really like Dojo? Why? It is very slow and heavy".

I wonder, are Dojo-do'ers comfortable with that? Do they think, it's not a
serious problem?
Why not to make 2 more special loaders in addition to heavy dojo.js (117kb
minimized is absolutely too much) : the first one is lighter version of
dojo.js with only small subset of most commonly used functionality like used
in jQuery's core or even lighter (say to get into 30-50kb unzipped at max),
and the second one is super-light loader, like require.js, just to allow one
to do basic cross-browser event processing and issue require() requests.
With a Dojo's brilliant custom build technique, it is possible to bake in
resulting JS only the things, that site really needs, keeping the result as
small and as fast as really possible. But in order to achieve this now, one
had to create a deep (and risky) hack into build process to strip
unnecessary code.

Does anybody in Dojo going to address this problems? Should we expect in
future releases of Dojo to be more lightweight and modular, or the core will
continue to grow larger?
Thanks)

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3557112.html

[ben hockey]

these problems have all been addressed - in fact almost all of them
were addressed previous to dojo 1.7.

dojo has always had what is called "base" - this comes in around 30k
gzipped and is comparable to jQuery's core in size and features.
jQuery claims to be 31k gzipped and minified. dojo also has "core"
components which are the other dojo.* modules not included in base and
these are in addition to the ~30k base.

for some reason, there is a perception that dojo is slow and heavy but
it's really not a fair perception. we are competitive in size and
speed when you compare the same feature set across the popular
libraries. my belief is that this perception comes from looking at
feature rich web apps based on dijit - accessibility,
internationalization, validation, widgets, etc don't come for free :)
jQuery UI is a library that is similar to dijit. i just downloaded
jquery ui with all its features selected and it comes in at ~300k
minified when you account for jQuery core as well.

dojo's developers understand this perception of dojo as big and slow
and as such we are always looking for opportunities to demonstrate
that this perception is unfounded. in dojo 1.7, we now also have the
option of building a super-light loader (smaller than requirejs afaik)
which has a feature set comparable to requirejs.

also with dojo 1.7, you now have the opportunity to work with dojo
"baseless" - this is what we call the nano build and it is just a
loader. from there, you explicitly declare your dependencies and you
only get what you need. the vast majority of dojo and dijit were
refactored during 1.7 so that if you include a particular module, you
get the minimum dependencies needed for that module.

at the end of the day, even though we have made the code VERY modular,
if you build a project that includes a few dijit widgets then you
can't avoid ending up with a resulting file that is in the 100k+
range. however, if you just want something to match requirejs, you
can have it and if you want something to match jQuery core, you can
have it and if you want to build applications with many feature rich
widgets you can have that too. each has its own price though :)

i should mention that dojo mobile is specifically built with lighter
weight widgets and great lengths were taken to strip back every little
piece used in mobile so that applications can be built which will work
efficiently on mobile devices. if you are targeting mobile devices,
this is what you should be using.

all of this is to say that all of the concerns you've raised are
understood and effort has been made to address them. in the next
couple of weeks you'll see a new dojo website which will have a great
deal more information to help you convince others that dojo is the
best toolkit available for them. we will continue to add more
features to dojo but they will only be included in your build if you
use them.

thanks,

ben...

[Arech]

Thanks a lot for rich answer, Ben! I forgot about dijit stuff... Rich widgets
are the most common cause of the slowness... You've really inspired me to
give Dojo new try :)

BTW, I've just tried to build with layerOptimize:'closure' and a rich set of
staticHasFeatures:{} taken from the link you gave earlier. The resulting
dojo.js has about 80Kb unzipped or 28Kb gzipped (with 7zip "ultra"
compression level). dojo/tests/runTest.html page doesn't work at all with
'dojo-sync-loader':0, but I guess it is because test suite is rather legacy
thing and it relies on sync loader?

Well, anyway 80/28kb is much better than ordinary result, but it's still not
4-5Kb gzipped, as you've mentioned))

I'm just curious: what am I doing wrong?)) (don't bother answering if the
building of super-light loader will be covered in a new coming site. Don't
waste your time, I'll wait a few weeks, that's ok)

Here is my profile:

dependencies = {
staticHasFeatures: {
'dom':1,
"host-browser":1,

'host-node':0,
'host-rhino':0,

"dojo-inject-api":1,
// usually, never for a stripped, async loader
'dojo-combo-api':0,
'dojo-trace-api':0,
"dojo-sniff":0,
"dojo-test-sniff":0,
"dojo-undef-api":0,
"config-tlmSiblingOfDojo":0,
"dojo-timeout-api":0,
"dojo-amd-factory-scan":0,
"dojo-requirejs-api":0,
'config-dojo-loader-catches':0,
'dojo-sync-loader':0,
'dojo-log-api':0,
'dojo-publish-privates':0,

// maybe
"dojo-dom-ready-api":1,
"ie-event-behavior":1,
"dojo-config-api":1
}

,layerOptimize:'closure'

,stripConsole: "normal"
,layers : [
{
name : "../app/main.js", //contains just alert('hi'); and it doesn't
baked into dojo.js
dependencies : [ "app.main" ] //so in fact doesn't matter
}
],
prefixes : [
//[ "dijit", "../dijit" ],//not necessary
//[ "dojox", "../dojox" ],//for building dojo.js
[ "app", "../../app"]
]
};

and command line:

call build.bat profile=/%HOMEDIR%\myjs.profile.js action=release
releaseName=rel releaseDir="../../../c/" copyTests=true cssOptimize=comments
mini=false stripConsole=normal

(
- front slash in front of path in 'profile' looks necessary to make build
system to load profile file from arbitrary location
- copyTests=true cssOptimize=comments mini=false - to leave test suite in
the result
)

Thanks a lot!
Aleksey.

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3557358.html

[Arech]

neonstalwart wrote

>
> see http://comments.gmane.org/gmane.comp.web.dojo.devel/15941 for some
> discussion and details of a profile that is needed to produce the "nano"
> build. the BIG key to making any of your builds smaller with 1.7 (as
> discussed in the previous link) is to use
>

> layerOptimize: 'closure'.

I've just finished porting some of my code to v1.7 and tried to build it and
execute.
So, layerOptimize set to 'closure' reduces dojo.js size to 88kb. That is
cool and everything works :)

When I try to apply advanced magic of staticHasFeatures, the size of dojo.js
reduces more. Only setting dojo-sync-loader to 0:

staticHasFeatures: {
'dojo-sync-loader':0
}

makes dojo.js smaller to about 80-81kb. That is perfect, but the result is
unusable. Code:

fails with exception "dojo is not defined" at dojo.connect() point. (Firefox
8.0 @Win7 x64)

Looks like something serious have been changed in v1.7 release (downloaded
it yesterday) since October, when you had a conversation at
gmane.comp.web.dojo.devel

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3557641.html

[Arech]

O-la-la...
dojo.js is not a single file, that must be counted when we are talking about
dojo's core file size... Even baked dojo.js fetches CSS query engine
/dojo/selector/acme.js, which weights additional almost 50kb (or 8.5kb if
you remember to specify optimize:closure option).
And also dojo fetches /dojo/require.js when custom layer is require()'d. It
has 131 byte, but anyway - it is another unnecessary http request...

So, we had to talk not about 87kb (85kb, - sorry, 85 is proper rounding)
(with layerOptimize:'closure'), but about 85+8.5 = 94 kb in a 3 (!!!) http
requests.

This is very bad :(

Do you think the build system work as expected here? Why acme.js and
require.js are not inlined in dojo.js ???

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3557724.html

[ben hockey]

there are many different things you can poke and prod when doing a build.  http://livedocs.dojotoolkit.org/build, http://livedocs.dojotoolkit.org/build/qref, http://livedocs.dojotoolkit.org/build/buildSystem ...

one big thing you could do to reduce the code needed for dojo to work is to use AMD format for your code.  the full benefits of dojo "nano" can only be achieved using AMD.  

with dojo, we have a commitment to being backwards compatible.  in this case it means that if you previously included just dojo.js, you expected that dojo.require and dojo.query would be available without you needing to do anything to explicitly load it.  so... in order to be backwards compatible, if we detect that you have code that was expecting those same conditions then we make it happen - we make dojo.require and dojo.query (and the rest of dojo "base") continue to work.  this explanation is oversimplified but the general idea is what's happening.  if you use AMD then since AMD has never been part of dojo previously, we don't have to be backwards compatible about anything yet so we can make things better for you.  we can avoid loading all of base by default and wait for your modules to tell us what they need.

as an analogy, if we talk about require.js again... require.js doesn't understand dojo.require - the same is true for dojo "nano".  it only understands `define` and `require`.  if you write your code using dojo.require then the loader has to bring in dojo/require.js to support that code.  also, require.js does not understand how to query the DOM - the same is true for dojo "nano".  so, it needs to fetch dojo/selector/acme.js for dojo.query to work.  while we're on the topic, in dojo 1.7, you can also select a lighter query engine if you prefer it http://livedocs.dojotoolkit.org/dojo/query#dojo-query-1-7-only

the WHOLE dojo 1.7 story including what the build can do and how to make it do it and what query selectors are available and everything else new in 1.7 is very long for me to explain but you can believe that if dojo "nano" does not do anything it claims to be possible i will tell you the truth and not marketing hype.  hopefully you have had enough interaction with me by now to know that i will be open and honest with you.

here's the best summary of nano, base and core in 1.7 that i can give you off the top of my head:

- "nano" ~4kb minified, gzip - this is a loader with a very narrow focus.  you must use AMD, you must use it in a browser, you must...  do you get the picture?  it is a loader that can load modules and even then it can only load a subset of what the unbuilt dojo.js can do.  from the thread i linked you too, you can see that we have actually managed to make this under 4kb - it's not just marketing.

- "base" ~30kb minified, gzip (maybe it's 40kb, i haven't looked in a long time but it's 10's of kb but less than 100kb minified, gzip).  this is the fully backwards compatible dojo.js and provides all the same functionality that was provided in previous versions of dojo whenever dojo.js was included.

- "core" this is not a single file so putting a size on it is kind of hard off the top of my head but it's more than base is all i can tell you.  these are the modules in dojo/* that were not previously included whenever dojo.js was included.

now that we have dojo "nano", there is really less of a need to distinguish between base and core because none of it is included by default when you load dojo.js (nano).  although, if you use "dojo" as an AMD dependency, you will load all of base and make nano ineffective.  so, you should make your dependencies more granular by just using "dojo/_base/array" for if you just want array helper functions (dojo.forEach, dojo.some, dojo.every, etc) and avoid loading all of base when you don't need it.

anyhow... there's way too much to cover in detail so if you have some more specific questions, start some new threads for those and i'll try to help you if i can - others will likely help too.  apart from that, http://livedocs.dojotoolkit.org/ is the "development" version of our documentation which is eventually released at http://dojotoolkit.org/reference-guide/ so start looking through livedocs for documentation and look out for a new dojo website within a few weeks.

thanks,

ben...

[Arech]

I did it!!)))

Thanks for a lot of clarifications! I read http://livedocs.dojotoolkit.org/
a few times and then the original thread
http://comments.gmane.org/gmane.comp.web.dojo.devel/15941 and then I
realized that you used special build profile standardCustomBase. That
profile uses customBase: true (4 those, who's interested see
http://livedocs.dojotoolkit.org/build/customBase ) setting for dojo layer
and it strips all unnecessary code - that's what I need. I applied it to my
profile and now all works perfectly, I got that 4kb gzipped loader (its
about 10kb minified and that is perfect!).

--
View this message in context: http://dojo-toolkit.33424.n3.nabble.com/How-to-create-smallest-possible-dojo-build-tp3554327p3559335.html