Modularization

[Ivan Sagalaev]

Hello!

I believe there came the time to move highlight.js away from its ad-hoc
module system usable only in a browser to something more formal. I was
reluctant to do this in the past because nobody was able to formulate
the problem that it will solve. And the argument "everyone does this" is
not exactlypersuasive :-).

However after some personal emails and GitHub comments I finally figured
it out. The problem is: highlight.js is not usable in node.js. You can
do this only by maintaining a separate fork which is suboptimal. Sorry
for taking so long to get it!

I've reviewed a couple of proposed solutions and made myself familiar
with the modern (sad) state of affairs on this and now I want to discuss
our options. I see two of them:

1. AMD and require.js all the way.

Each language and the library are wrapped into `define('name',
[..requirements..], function() {})`. These modules can be used in
node.js[1] and they can be build into a single .js file usable in a
browser using the build tool provided by require.js. I don't see any
problems here but this is probably for the lack of experience. I think
we won't even lose the additional custom minimizations to variable names
that the current build.py does because they can be applied before the
final build process.

I also don't want to lose the current implicit in-browser API: the
library is in the global variable `hljs` and you can dynamically add
additional languages with `hljs.LANGUAGES.name = ...`. Can anyone
explain if it's possible?

Another thing that I'm not sure how to do is using the requirements
inside a module definitions. We now have django.js that defines Django
highlighting by borrowing everything from hljs.LANGUAGES.xml that is
already defined by the time. How does this translate into the require.js
world? Will "xml" module be passed into the "django" module function?

2. Two build targets.

Each language file will contain exactly one anonymous function returning
a language object. The build tool will have two different targets, say,
"browser" and "amd" that will wrap the file content into the appropriate
boiler-plate:

- "browser":

'hljs.LANGUAGES.name = ' + <file> + '(hljs);'

- "amd":

'define("name", ' + <file> + ');'

The module format in this case may not be the AMD endorsed by require.js
but also a common.js, "native" to node.js (why *everything* has to end
with ".js"???). We even can have two different module build targets.

---

I don't have any objections to either of those ways but I'm probably +0
in favor of the first way. What do others think?

[Oleg Efimov]

Nice to hear it!

I don't know about situation in CommonJS world, but for Node.js I think custom builds is not so needed.

It is enougth to have one highlight.js file with all languages for Node.js.

I'll try to create something working with Node.js + require.js + highlight.js at home and tell you about results.

пятница, 13 апреля 2012 г., 12:37:45 UTC+4 пользователь Ivan Sagalaev написал:

[Ivan Sagalaev]

> I've reviewed a couple of proposed solutions and made myself familiar
> with the modern (sad) state of affairs on this and now I want to discuss
> our options. I see two of them:
>
> 1. AMD and require.js all the way.

>...
> 2. Two build targets.

Back to that thing again�

I found a boring evening to play with those two options and now it seems
that the first one is a complete no-go because it requires using
require.js in a browser. I really don't want to do that since my old "no
dependencies" policy worked very well judging by quite a few praises
about perceived simplicity of highlight.js I've seen out there.

The second option, on the other hand, allows us to have everything: a
standalone packed browser build as well as individual modules in any
format we like. I.e. option 2 makes possible option 1 but not vice versa.

What I've done so far is extended build.py to support the --target
option with the single "browser" value for now and converted a few files
to be bare anonymous functions returning a language definition:

function(hljs) {
return { ... }
}

During the build it gets wrapped into `hljs.LANGUAGES.name =
<function>(hljs);` and then everything is built and compressed as
before. There's also one pleasant side effect: since the `hljs` in the
language file is a local variable now the YUIcompressor replaces it with
a single-letter name and the overall compressed size becomes a little
smaller :-).

The plan to go forward is as follows:

1. convert the rest of the language files (this should be simple)
2. implement common.js build target to be able to use the library under
Node (which was the whole point)
3. implement AMD target for those who already uses require.js and would
like to use highlight.js with it

I will certainly do item 1 myself. If someone wants to help with item 2,
say so and I'll publish the branch then. The item 3 is actually optional
so I'll probably not going to do it myself anyway :-). Volunteers are
welcome!

There's also one additional idea to have a build target "jQuery" which
will *remove* the built-in initialization code from the library making
it even smaller. That would require inventing some sort of conditional
comment syntax for source files.

[Oleg Efimov]

Good decision :) I hope to have some time to work on "2" in next days.

понедельник, 28 мая 2012 г., 13:38:20 UTC+4 пользователь Ivan Sagalaev написал:

[Ivan Sagalaev]

So this is in the master now, yay!!! And it really does work under
node.js without a fuss:
https://twitter.com/isagalaev/status/207301074548293633

I ended up with just two build targets: "browser" and "node". The
"browser" is the default one, so if you don't want to know anything
about node.js you don't have to :-).

The "require.js" target is left as an exercise for the adventurous :-)

[Oleg Efimov]

Nice! The only thing i suggest is to build node.js version into
build/node dir to allow multiple builded scripts at one time.

2012/5/29 Ivan Sagalaev <man...@softwaremaniacs.org>:

--
Best regards,
Oleg "Sannis" Efimov

http://sannis.ru

[Ivan Sagalaev]

> allow multiple builded scripts at one time.

Hm… Why? The build/ dir is supposed to be temporary anyway, you would
move everything out of it somewhere else.

[Oleg Efimov]

I need some clarification: the obly thing that I suggest is to remove
`rm -rf ./build` from build script. So we can build browser and node
versions and any next build does not clen previous.

Reasons:
- It is helpful for simultanuous testing browser and node.js versions,
especially for automatically runned tests.
- I want to contribute package.json for node.js version, so it is not
good thing that `npm publish` will remove build/highlight.pack.js

If you accept this, maybe highlight.node.js will be also a good thing?

2012/5/29 Ivan Sagalaev <man...@softwaremaniacs.org>:

>> allow multiple builded scripts at one time.
>
>
> Hm… Why? The build/ dir is supposed to be temporary anyway, you would move
> everything out of it somewhere else.

[Ivan Sagalaev]

> - It is helpful for simultanuous testing browser and node.js versions,
> especially for automatically runned tests.

This is a good example but it's hypothetical. I could argue that there's
little chance that any software will want to test two highlight.js build
targets. To be honest, I can hardly even imagine any software of that
sort :-). I always prefer to make changes when they are warranted by
real use-cases. It's really simple to remove that cleanup or hide it
behind some condition when it is really needed.

> - I want to contribute package.json for node.js version, so it is not
> good thing that `npm publish` will remove build/highlight.pack.js

Could you educate me about "package.json"? All this node.js stuff is
very new to me :-)

Anyway, if you're going to use highlight.js under node.js why you would
want to create a browser build (highlight.pack.js) at all?

> If you accept this, maybe highlight.node.js will be also a good thing?

There's probably some confusion. Node.js is not supposed to use a packed
self-contained version so there's no such thing as highlight.node.js.
The result of the "node" build is the whole directory with all the
language files and the highlight.js.

[Oleg Efimov]

> This is a good example but it's hypothetical. I could argue that there's
> little chance that any software will want to test two highlight.js build
> targets. To be honest, I can hardly even imagine any software of that sort
> :-). I always prefer to make changes when they are warranted by real
> use-cases. It's really simple to remove that cleanup or hide it behind some
> condition when it is really needed.

Yep, this is not the main reasond and i do not insist to do now.

> Could you educate me about "package.json"? All this node.js stuff is very
> new to me :-)
>
> Anyway, if you're going to use highlight.js under node.js why you would want
> to create a browser build (highlight.pack.js) at all?

The common way to use modules in node.js universe is installing them
with npm (http://npmjs.org/). It is bundled with node installation for
now and is a de facto standard.
It allow to describe you application/module with package.json that
includes descriptional information, build scripts and dependencies. It
is close to Python's setup.py or RPM spec files.
So for people that used node.js is better to simply list `highlight`
in their application dependencies rather than clone&build it.

There is 3 projects already places in npm registry that provides hljs:
- https://github.com/pumbur/hljs
- https://github.com/andris9/highlight
- https://github.com/jgallen23/highlight.js

So if you plan to decrease distance between forks, I think it is
better to keep package.json in your repository and grant publish
rights to your. Yes, it looks more simple not to do this, but having 3
different packages in npm registry with different version naming that
also differ with initial highlight.js version is not so good. As I
see, only pumbur's fork have big diff, other two are forked only for
publishing to npm registry.

[Ivan Sagalaev]

> I think it is better to keep package.json in your repository

Oh, yes, I'm all for it!

BTW, can the npm format describe a module/package consisting of several
files?

[Oleg Efimov]

It should only specify the "entry point" - ./build/highlight.js and
the package should contain all requiren files. You can setup
.npmignore (like .gitignore) to specify which files should not be
taked to package archive.

2012/5/29 Ivan Sagalaev <man...@softwaremaniacs.org>:

[Ivan Sagalaev]

Well then we just need to generate package.json in addition to
highlight.js and language files. The file itself can live in the src/.
Are you comfortable with Python to add this code to build.py?

[Oleg Efimov]

I think it should not be a big trouble :) Have you any thoughts on 7.0
release date? I need to play with npm to be sure that it possibl to
place package.json to repo subfolder and successfully publish a
package. Anyway, i can do it only after work :)

2012/5/29 Ivan Sagalaev <man...@softwaremaniacs.org>:

> Well then we just need to generate package.json in addition to highlight.js
> and language files. The file itself can live in the src/. Are you
> comfortable with Python to add this code to build.py?

[Ivan Sagalaev]

There's no deadline for any version release simply because we don't have
any external constraints. So feel free to contribute it whenever you
have time.

[Oleg Efimov]

But I hope this will be in 7.0 release.

I pull package.json support to GitHub, please review it:
https://github.com/isagalaev/highlight.js/pull/95

It successfully publishes if you build node target and run `npm
publish` from build dir. So we need only to decide about package name.

2012/5/29 Ivan Sagalaev <man...@softwaremaniacs.org>:

> There's no deadline for any version release simply because we don't have any
> external constraints. So feel free to contribute it whenever you have time.

[Ivan Sagalaev]

Yep, I'll give it a look later today.

As for the name… Ideally it should be "highlight.js". If dots aren't
allowed then it becomes "highlightjs" which is by the wat the same name
that I used for the recently registered highlightjs.org.

[JGAui]

I'll pull in latest code and publish today.  Who should get access to the npm repo?

[Ivan Sagalaev]

Let it be me to get access.

To clarify: are you going to publish the new version from the master
yourself? We have some more things to do before the new release and I
wanted to publish it then.

[JGAui]

no, I don't have to, just thought it would be an easier transition, but up to you

[JGAui]

what is your npm username?

[Ivan Sagalaev]

Username is isagalaev (freshly registered :-) ).

I didn't want to transfer the ownership right away in case we delay the
release for some reason and in the meantime you might want to suddenly
update something. So I'll ping you then when we've rolled out the new
versions.

Thanks for collaboration!