Migrating Emscripten to use npm mechanism to ship Closure and html-minifier
Jukka Jylänki
the PRs
https://github.com/emscripten-core/emsdk/pull/404,
https://github.com/emscripten-core/emscripten/pull/9989, and
https://github.com/emscripten-core/emscripten/pull/9990
propose migrating Closure compiler and html-minifier to reside outside
the Emscripten tree, and be installed via npm.
The benefits of this change are:
1. Future updates to closure and html-minifier will not bloat up the
size of the Emscripten git repository (each update would increase the
size of the git repository by ~+10% otherwise)
2. Emscripten developers do not need to maintain a CDN and code for
distributing closure/html-minifier, easing CDN and testing burden from
emsdk,
3. Tracking which version of closure and html-minifier we are on
becomes explicit and idiomatic to people familiar with npm community
(can be found in package.json) rather than having to look up git logs,
4. Updating versions become easier, as migrating to a new versions
can be changed into the file package.json
The drawbacks of this change are:
5. developers who follow the "I git cloned all repositories myself"
approach and do not use emsdk need to run "npm install" once in the
Emscripten root directory if they want to use closure or
html-minifier,
6. If npm goes down, it will disrupt emsdk installation
Can people think of other benefits/drawbacks that should influence
this design change?
Cheers,
Jukka
Gabriel Cuvillier
Hello,
On CMake-based projects where I want to keep separate the web app
code (= several vanilla JS files) from the various emscripten
modules that are used (= several emscripten-generated JS/wasm), I
conveniently use Closure as provided by Emscripten directly from
my CMakefile to bundle/optimize the web app code.
=> CMakeList.txt is looking like:
add_executable(<my_emscripten_module_1> file_1.cpp file_2.cpp --closure 1) # => my_emscripten_module_1.js/wasm add_executable(<my_emscripten_module_2> file_3.cpp file_4.cpp --closure 1) # => my_emscripten_module_2.js/wasm add_custom_target(<my_webapp_using_the_emscripten_modules> COMMAND "java" "-jar" "$ENV{EMSDK}/upstream/emscripten/third_party/closure-compiler/compiler.jar" "--js" "<file_1>.js" "--js" "<file_2>.js" "--js_output_file" "<my_webapp_using_the_emscripten_module>.js" "--language_in=ECMASCRIPT_2017" "--language_out=ECMASCRIPT_2017" "--compilation_level" "<BUNDLE|SIMPLE|ADVANCED>" DEPENDS <my_emscripten_module_1> <my_emscripten_module_2) # => my_webapp_using_the_emscripten_modules.js
Then, to build the app, I just have to do:
make <my_webapp_using_the_emscripten_modules>
And pretty much, that's it, the 3 bundles are created: the web
app code (closure optimized/bundled vanilla JS), the emscripten
module 1 (closure optimized generated JS+wasm), and emscripten
module 2 (closure optimized generated JS+wasm)
I find this very convenient to do, as it allows to do everything
from CMake, and without having to use another module bundler
(Webpack, and co.) and the full node/npm ecosystem that I simply
don't need. I admit it feels a bit old-school/an heterodox way of
doing things on the Web (using Make!?), but I find this much more
in spirit with the C++ way of building projects. After all, we
are using Emscripten for a reason, isn't it ?
But I suppose that scenario would be feasible also if Closure was
provided through npm and not through a .jar directly in Emscripten
distribution (I would have to call some npm command for this,
right ?). I can live with a scary 'node_module' folder ;)
Cheers,
gabriel
Jukka Jylänki
would mean that instead of
COMMAND "java" "-jar"
"$ENV{EMSDK}/upstream/emscripten/third_party/closure-compiler/compiler.jar"
COMMAND "$ENV{EMSCRIPTEN}/node_modules/.bin/google-closure-compiler"
I.e. only the location of the closure-compiler file will change. Emsdk
takes care of the npm install line, so emsdk installing a sdk will
still come with Closure preinstalled and available.
However the above path change is in some sense unrelated to this
in-tree -> npm change, but instead a result of Closure getting updated
to a newer version (which we want to do no matter what). In the newer
version of Closure, java is optional. (google-closure-compiler
executable will use it if available)
su 15. jouluk. 2019 klo 17.37 Gabriel Cuvillier
(gabriel....@gmail.com) kirjoitti:
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/3e9f7095-dee6-711e-1623-ae715c81c5ce%40gmail.com.
Gabriel Cuvillier
Thanks for the tip !
Alon Zakai
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B6sJ-2YQX9ku%3DPrwhDP0pnNoRZGjF8WtEnRPPMp0jFXJ-6Fsw%40mail.gmail.com.
Gabriel Cuvillier
If this change does not impact end-users of Emscripten, that just
want to do C++ development targeting WebAssembly, then this is
perfectly fine. This change would just be an EMSDK
"implementation detail", and I agree that from this point of view,
removing the Java dependency is interesting.
However, if the users are expected to use Node/npm at some point, then please, please, don't do this :) Not everyone is willing to / can use the Node/npm ecosystem on the projects they are working on.
An alternative suggestion: instead of having to rely on 3rd party
distribution system, why not "precompile/bundle" closure (as well
as minify-html, or any other 3rd party dep), and use a more
classic CDN-approach to deliver these big files. This would
remove the need for a package management system (npm, or even PIP
if you have the same idea for Python tools) running on users
machine... and potentially breaking for various reasons (network
failures, misconfiguration of PATH, weird interaction with
system-wide installation of Node/npm, etc..)
Cheers,
Gabriel
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpQOBVM6EHiVZwrwGmD_Qub-koAFey%3DLLq_uHXi12E4-RQ%40mail.gmail.com.
Shachar Langbeheim
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/549820ac-3512-3b28-4dca-ec47892848ef%40gmail.com.
Gabriel Cuvillier
Well... Git LFS is one solution, but maybe not the easiest one to
handle for the devs.
But a simple file server with a nice directory structure
containing each precompiled/packaged 3rd party dependency is
another alternative (just have to be correctly tag/name the
folders and files, so that Emsdk can download the correct versions
according to the Emscripten release being pulled). Put it
differently, the simplest thing that could possibly work.
By doing so, there is some "reinvent the wheel" syndrome , as npm/pip/etc. are already addressing such kind of needs (manage packages/dependencies/delivery/etc.). The issue is that they are heavily tied to their own respective language/platforms (be it Node, Python, etc.). I wonder if there exist some kind of package management system that is programming language-neutral (and OS neutral also), and only focusing on having a bunch of files to be installed in a particular location according to some spec in a git branch. Basically, this is the need there.
But of course, I am probably not the right person to take any decision on this :)
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B_KjGbSVvrHo2PTjqyTsAHyVfyURjH7pNc0%2BTp-EeGAjJtOxw%40mail.gmail.com.
Jukka Jylänki
(gabriel....@gmail.com) kirjoitti:
> My two cents on this, from the point of view of a daily Emscripten user on various projects :
>
> If this change does not impact end-users of Emscripten, that just want to do C++ development targeting WebAssembly, then this is perfectly fine. This change would just be an EMSDK "implementation detail", and I agree that from this point of view, removing the Java dependency is interesting.
install still sets up the SDK functionally like it used to.
> An alternative suggestion: instead of having to rely on 3rd party distribution system, why not "precompile/bundle" closure (as well as minify-html, or any other 3rd party dep), and use a more classic CDN-approach to deliver these big files.
in https://github.com/emscripten-core/emsdk/pull/404#issuecomment-564720935,
basically each update of closure would increase the size of the git
repository by ~30MB/+13% overall download size. And since git tracks
all history, that means it will download all old versions when one git
clones. After a couple of updates of Closure, the size of all the
Closures in the tree history would be a larger portion of the overall
Emscripten git repository than Emscripten bits itself.
Git LFS route would work for this only if Closure and html-minifier
and other tools would provide a single file amalgamation of the whole
toolchain they provide, but unfortunately this is not the case.
Closure and html-minifier are split across hundreds of small files, so
Git LFS cannot be applied.
> This would remove the need for a package management system (npm, or even PIP if you have the same idea for Python tools) running on users machine... and potentially breaking for various reasons (network failures, misconfiguration of PATH, weird interaction with system-wide installation of Node/npm, etc..)
it would mean developing and maintaining support into Emsdk + its CDN
backend to serve Closure and html-minifier etc. (which using npm helps
avoid, points #2 and #4)
Emsdk does provide a fixed version of Node.js for all platforms, so
system-wide installations of Node/npm, and also PATH configuration,
should not matter here. Network failures are a possibility, but that
may be a bit of a FUD point - the robustness and reliability of npm
infrastructure is likely better maintained compared to the robustness
of emsdk infra. (there are probably 100x more active users of npm
ecosystem compared to emsdk ecosystem)
Shachar Langbeheim
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B6sJ-2kMqBLJ%3D-3zjRFGTYM8yLOZOFx4ALhXrudWtXu-HyXNQ%40mail.gmail.com.
Jukka Jylänki
scheme would be to avoid downloading from npm, which would keep the
repository more self-contained.
Downsides that come to mind:
- It will require asking users to install git-lfs separately.
- Would require developing a bundled installation script in
Emscripten that non-emsdk users are asked to run to uncompress.
(comparable and does not avoid a npm install)
- For npm developers, running npm install is already familiar and
idiomatic, but a custom script is, custom.
- There may be need to ship platform-specific versions of some npm
tools (closure-compiler-win.zip, closure-compiler-macos.zip,
closure-compiler-linux.zip, closure-compiler-arm.zip), or create fat
zips that have all platforms. For closure specifically there is a JS
version, to could use just that instead.
- More maintenance work compared to the npm approach (e.g. on
packaging and versioning)
- Migrating between versions becomes harder for users. With npm if
user wants to try to update to latest closure that the Emscripten tree
does not currently have, they can edit a single line in package.json
and reissue npm install. With a custom zip users would have to figure
out the directory structure and the install/update mechanism that the
custom scheme uses to replace the tool with a new installation.
Though there is no reason why such a "zip in the repository" approach
(or migrating distribution to emsdk) could not be adopted also later.
I think avoiding npm up front for the fear of it possibly being
unreliable is probably not sensible. Since npm is the least work to
use and maintain, I would recommend seeing that path out until it
fails, and then migrating to some other heavier method if npm is not
feasible.
Gabriel Cuvillier
> It will require asking users to install git-lfs separately
Indeed, that's a good argument against git-lfs: one user-visible
additional requirement.
Shachar Langbeheim
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/982257b1-153e-eb6b-db22-4385f3ae1b19%40gmail.com.
Alon Zakai
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B_KjGYR3xrSVT52Rd1jJ9dFiwHyO-8oeVHqHYHhvk8hgEZxPA%40mail.gmail.com.
Sam Clegg
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpSMQQK4g%2BoLK5r32OXjq7nWr4YCMF52s63sc_V1%2B7aRgg%40mail.gmail.com.