Patching Emscripten: preserving good practices
Beuc
We almost never could use Emscripten without applying a few patches.
It's become increasingly difficult, not only to patch, but to convince the developers that it's legitimate, so I thought I'd detail our use case for good here.
So we've been using Emscripten with patches in emscripten, llvm and SDL2 port [1].
By the time a patch is officially included in a release, we usually need more patches as we face new issues -- or regressions.
Rinse and repeat.
Consequently for our production use (the RenPyWeb build that we publish to our users), we build an identified, versioned release (e.g. 1.38.42) with specific fixes.
New difficulties:
- scarse documentation for recompiling "upstream" (patch in progress #9383)
- no matching tags for llvm and bynarien, commit IDs quite hard to track (discussion in #9317/#9383)
- misleading versioning in tagged commits (#9282)
- grab-from-git port system; EMCC_LOCAL_PORTS considered for "local debugging", not cached, and causing issues (e.g. #9342 #9402)
Rebutals:
- "99% users use emsdk binaries"
Well, the remaining 1% will provide 99% of external contributions, take care of them :)
Also being able to test fixes before they finish the full review&ci process allows spotting issues in advance.
- "users will either use the unmodified binaries, or work on master/tip-of-tree"
No, we use an identified, tagged Emscripten version, because it's more stable than a random commit.
Also when we publish a hot fix, sometimes a couple months after release, we need to reproduce the same build environment
- "emsdk can build from source"
This build process doesn't allow patching, so it's only when one doesn't trust the binaries and wants to rebuild them as-is AFAIU.
(To be honest, I'd like to be able to use the binaries, but Emscripten being bleeding-edge, I figure that'll have to wait :))
I hope this makes clear why we need to checkout emscripten/llvm/bynarien for an existing release, patch it, and build it.
Preferably without reverse-engineering the build process and fighting all the stack.
That's the first time I have to justify this while working on free/open source software.
How do you people deal with this?
[1] For instance, our current release needs these patches, some of them still needing work & official integration:
- https://github.com/emscripten-core/emscripten/issues/9257
Work-around ./configure misdetection
- https://github.com/emscripten-core/emscripten/issues/9097
Fix fullscreen exit causing wrong screen size
- hard-coded change to EMT_STACK_MAX which isn't configurable
- https://github.com/emscripten-core/emscripten-fastcomp/pull/195
Compiler limitation when combining several Emscripten features
- https://github.com/emscripten-ports/SDL2/issues/88
Send SDL_APP_TERMINATING on page exit
- https://github.com/emscripten-ports/SDL2/issues/70
Support Emterpreter/Asincify in SDL2
- Search "Emscripten" in https://renpy.beuc.net/#status_reports for all the other patches that were eventually integrated over the year
Alon Zakai
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/20190906073253.yh5wv677v3l4xn4k%40mail.beuc.net.
Sylvain Beucler
Hi,
(I was hoping other people would give their PoV...)
Generally-speaking, we may need more patches because of:
- the combination of Emscripten features (Python +
uninterruptible main loop -- aka fpcast+async; thankfully I could
get rid of dynamic linking...)
- centrally improving Emscripten portability rather than adapting the codebases (e.g. spending weeks on the 1-line #8751 / LANG support, rather than adding a quick EM_ASM in all my projects)
- working directly with Emscripten rather than using a framework
that provides a layer of work-arounds (e.g. Unity)
Again, at a given point of time we always have a buffer of pending patches (I forgot to mention #6511 and #7631 that were added to our shell.html template, btw), the ones we can upstream are replaced by newer ones: old bugs we eventually tackle like fullscreen support or autosave-on-quit; backports for regressions in a given Emscripten release.
Hopefully one day the web ecosystem will be so mature that we'll
be able to take our distro's years-old compiler, `./configure
--host=browser` without any change, and publish. Meanwhile... we
patch :)
To answer your specific questions:
- This is a game engine, much of the tests are manual, sadly (but I contribute tests when I submit Emscripten patches); we often get integration errors, or errors due to combining multiple Emscripten options.- I upgrade Emscripten to a tagged version ~every other month;
since this is likely to introduce breaking changes, I only do it
when I have the time to deal with the consequences (e.g. when I'm
not in the middle of developing RenPyWeb itself). I avoid ToT for
stability and reproducibility reasons.
- EMT_STACK_MAX is the exception, not the rule ;) That's about the only thing I could have upstream but didn't due to lack of time -- and it's now deprecated.
- Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
Cheers!
Sylvain
Beuc
Come to think of it, there's one thorough automated test that we
have to run at each upgrade:
building dependencies!
https://github.com/renpy/renpyweb
https://github.com/renpy/renpyweb/tree/master/scripts
https://www.beuc.net/python-emscripten/python/dir?ci=tip
Typing a simple:
$ make
will download and build all dependencies, and it's not uncommon that something breaks on Emscripten upgrade.
Cheers!
Sylvain
Alon Zakai
> * Are there tests in your codebase that we could run in upstream emscripten?
Come to think of it, there's one thorough automated test that we have to run at each upgrade:
building dependencies!
https://github.com/renpy/renpyweb
https://github.com/renpy/renpyweb/tree/master/scripts
https://www.beuc.net/python-emscripten/python/dir?ci=tip
Typing a simple:
$ make
will download and build all dependencies, and it's not uncommon that something breaks on Emscripten upgrade.
Cheers!
Sylvain
On 10/09/2019 12:41, Sylvain Beucler wrote:
Hi,
(I was hoping other people would give their PoV...)
Generally-speaking, we may need more patches because of:
- the combination of Emscripten features (Python + uninterruptible main loop -- aka fpcast+async; thankfully I could get rid of dynamic linking...)
- centrally improving Emscripten portability rather than adapting the codebases (e.g. spending weeks on the 1-line #8751 / LANG support, rather than adding a quick EM_ASM in all my projects)
- working directly with Emscripten rather than using a framework that provides a layer of work-arounds (e.g. Unity)
Again, at a given point of time we always have a buffer of pending patches (I forgot to mention #6511 and #7631 that were added to our shell.html template, btw), the ones we can upstream are replaced by newer ones: old bugs we eventually tackle like fullscreen support or autosave-on-quit; backports for regressions in a given Emscripten release.
Hopefully one day the web ecosystem will be so mature that we'll be able to take our distro's years-old compiler, `./configure --host=browser` without any change, and publish. Meanwhile... we patch :)
To answer your specific questions:
- This is a game engine, much of the tests are manual, sadly (but I contribute tests when I submit Emscripten patches); we often get integration errors, or errors due to combining multiple Emscripten options.
- I upgrade Emscripten to a tagged version ~every other month; since this is likely to introduce breaking changes, I only do it when I have the time to deal with the consequences (e.g. when I'm not in the middle of developing RenPyWeb itself). I avoid ToT for stability and reproducibility reasons.
- EMT_STACK_MAX is the exception, not the rule ;) That's about the only thing I could have upstream but didn't due to lack of time -- and it's now deprecated.
- Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/14c22302-1e4c-e5af-bab8-20e55e14151a%40beuc.net.
Beuc
HHi,
On Mon, Sep 16, 2019 at 6:50 AM Beuc <be...@beuc.net> wrote:
> * Are there tests in your codebase that we could run in upstream emscripten?
Come to think of it, there's one thorough automated test that we have to run at each upgrade:
building dependencies!
[...]
It might be useful to set up CI that runs the emscripten tip-of-tree builds on that (emsdk install tot-fastcomp or tot-upstream). Those are literally the very newest code, that passed chromium CI but is not as heavily tested as the actual release tags. You may sometimes see a temporary breakage you can ignore, but it would also catch regressions.
There seem to be a misunderstanding, you asked for test you could
run, not the other way around ;)
It isn't worth it for us to constantly rebuild with ToT: most of
the time we're working on Ren'Py/RenPyWeb itself and can't afford
varying dependencies.
Investigating all temporary and non-temporary breakages sounds
like time I should reserve for properly upgrading and adapting to
a newer, stable Emscripten once every other month.
- Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
Is that still an issue? Are the docs at
(I hadn't seen this question on first read.)
My first attempts with binaryen seems to work, and for those interested Sam Clegg posted a recap:
https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
Cheers!
Beuc
Beuc
I just upgraded 1.38.45->1.38.46 and I got a new build failure
in an unmodified ffmpeg build.
Which is also what happened last time I upgraded Emscripten 1
month ago (different error).
I don't see wasm-ld-related notes in the ChangeLog, but apparently
it got stricter again.
Compiling a few external libs may actually be a good release test
case for the Emscripten project ¯\_(ツ)_/¯
- Beuc
Mehdi Sabwat
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net.
Alon Zakai
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/3f1f8c14-8d50-d66d-f6fb-31c588f42acf%40beuc.net.
Sam Clegg
> Which is also what happened last time I upgraded Emscripten 1 month ago (different error).
> I don't see wasm-ld-related notes in the ChangeLog, but apparently it got stricter again.
> Compiling a few external libs may actually be a good release test case for the Emscripten project ¯\_(ツ)_/¯
>
https://github.com/emscripten-core/emscripten/issues/9565?
The good news is that we now have a unit test to cover that specific
issue. But I agree we also need larger system tests. Perhaps we
could add ffmpeg to our build somehow, perhaps via an improved ports
system.
> - Beuc
>
> On 20/09/2019 20:06, Beuc wrote:
>
> Hi,
>
> On 17/09/2019 00:18, Alon Zakai wrote:
>
> On Mon, Sep 16, 2019 at 6:50 AM Beuc <be...@beuc.net> wrote:
>>
>> > * Are there tests in your codebase that we could run in upstream emscripten?
>>
>> Come to think of it, there's one thorough automated test that we have to run at each upgrade:
>> building dependencies!
>> [...]
>
> It might be useful to set up CI that runs the emscripten tip-of-tree builds on that (emsdk install tot-fastcomp or tot-upstream). Those are literally the very newest code, that passed chromium CI but is not as heavily tested as the actual release tags. You may sometimes see a temporary breakage you can ignore, but it would also catch regressions.
>
> There seem to be a misunderstanding, you asked for test you could run, not the other way around ;)
>
> It isn't worth it for us to constantly rebuild with ToT: most of the time we're working on Ren'Py/RenPyWeb itself and can't afford varying dependencies.
> Investigating all temporary and non-temporary breakages sounds like time I should reserve for properly upgrading and adapting to a newer, stable Emscripten once every other month.
>
> - Tagging: TBH I didn't figure out yet e.g. "what LLVM commit is used for emsdk 1.38.42" from https://chromium.googlesource.com/emscripten-releases/
>
>
> Is that still an issue? Are the docs at
>
> https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten
>
> (I hadn't seen this question on first read.)
>
> My first attempts with binaryen seems to work, and for those interested Sam Clegg posted a recap:
>
> https://github.com/emscripten-core/emscripten/issues/8995#issuecomment-532984238
>
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
Sylvain Beucler
On 08/10/2019 21:12, 'Sam Clegg' via emscripten-discuss wrote:
> On Wed, Oct 2, 2019 at 12:35 PM Beuc <be...@beuc.net> wrote:
>> I just upgraded 1.38.45->1.38.46 and I got a new build failure in an unmodified ffmpeg build.
>> Which is also what happened last time I upgraded Emscripten 1 month ago (different error).
>> I don't see wasm-ld-related notes in the ChangeLog, but apparently it got stricter again.
>> Compiling a few external libs may actually be a good release test case for the Emscripten project ¯\_(ツ)_/¯
>>
> I think are you referring to this bug
> https://github.com/emscripten-core/emscripten/issues/9565?
>
> The good news is that we now have a unit test to cover that specific
> issue. But I agree we also need larger system tests. Perhaps we
> could add ffmpeg to our build somehow, perhaps via an improved ports
> system.
"perhaps via an improved ports system": I wouldn't recommend it, because
IMHO the point of Emscripten is to build libraries as-is (through
emconfigure/emmake), rather "port" them individually. For instance the
SDL2 port is a bad use case for this test, since its build system is
remade from scratch and Emscripten-specific. We want to test a
vanilla/portable external build here.
Cheers!
Sylvain
Sam Clegg
of packages that are known to be buildable with emscripten. I didn't
mean we should do work on the upstream projects to port them.
A bit of background: I've been pushing back on adding more ports
recently because I'm not sure we want to have every open source
package in the emscripten tree itself. So what I mean when I say
improved ports system is some way to list third party packages that
are buildable (along with the build recipe), that are not necessarily
part of the "core" emscripten but should certainly be tested with each
release. I don't know that best way to achieve this just yet.
Perhaps something in the emscripten tree? or perhaps an external repo
(like llvm's test suite: https://llvm.org/docs/TestSuiteGuide.html)?
Perhaps driven by embuilder? Perhaps not?
> Cheers!
> Sylvain
> --
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
Gabriel Cuvillier
"Web Ports" system ?
https://chromium.googlesource.com/webports
For example, the list of ports for Pepper 47 version:
https://chromium.googlesource.com/webports/+/pepper_47/docs/port_list.md
This is a 3rd party repository (that is, not in NaCL main tree) for
projects buildable on the NaCl platform. It was a very handy by the time.
I understand the reluctance to have every possible ports in the
Emscripten tree (apart maybe some very common stuff, such as SDL2, zlib
or Freetype), though having a "ports ecosystem" is nevertheless
something needed
Alon Zakai
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/421d3f5d-0b5b-dcc2-4e29-03e4fe9c7dd4%40gmail.com.
Sam Clegg
>
> Yeah, I think we can add repos to emscripten-ports for things like that. We don't need to add support in the emscripten repo itself for them, which does make it more like nacl-ports. So there would be two tiers in emscripten-ports support; they'd all be in emscripten-ports, but only some would have code in emscripten itself.
>
> Concretely here, we could add ffmpeg to emscripten-ports but not to emscripten, and add a test in emscripten, and then we'd get regression tests to avoid issues there, which given the history could be useful. The only objection I might have to that is if the test is very slow. Otherwise, if someone's interested to do it, it sounds good to me.
>
ports should not need to be forked to exist in this system. It
should be enough to commit a simple build recipe (e.g. emconfigure &&
emake) along with an upstream URL. The emscripten-ports
organization is mostly used/useful for hosting forked repos. I think
I'm suggesting something more like the "emscripten/tools/ports" tree,
where each project is represented by some metadata and they all live
in the same tree.
For now I propose the following:
1. Add ffmpeg to "emscripten/tools/ports". Use emmake and emconfgiure
here to mimic the build process used by upstream/Beuc.
2. Add an optional flag to mark ports as "contrib" or "second tier".
This would signal that normal CI doesn't build them, and they don't
get built by "embuilder build ALL" or get listed by default by
"embuilder list".
Perhaps we can use a subdirectory called "contrib" under
"emscripten/tools/ports". Any ports in that tree would be less
frequently tested and have a lower bar for entry. Then we can have
people add whatever ports they want there.
Alon Zakai
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28ik-mb7pop9PyU_O-P3SZ6rK%3DCB2kkCSO7ZgivoiLM8g%40mail.gmail.com.
Beuc
In case you'd like to inspect some build recipes for inspiration :)
https://github.com/renpy/renpyweb/blob/master/scripts/ffmpeg.sh (broke in the past)
https://github.com/renpy/renpyweb/blob/master/scripts/libjpeg-turbo.sh (broke in the past)
https://github.com/renpy/renpyweb/blob/master/scripts/fribidi.sh (simple autotools)
https://github.com/renpy/renpyweb/blob/master/scripts/libpng.sh (less simple autotools)
https://github.com/renpy/renpyweb/blob/master/scripts/libzip.sh (cmake)
https://github.com/renpy/renpyweb/blob/master/scripts/SDL2_image.sh (cross-compile with webp support)
https://github.com/renpy/renpyweb/tree/master/scripts (for more)
All are currently built static-only, remove --disable-shared for testing dynamic builds.
Tarballs URLs are in https://github.com/renpy/renpyweb/blob/master/Makefile
- Beuc