Backends for Emscripten SDK

[Piotr Paczkowski]

Hi,

I'm a maintainer of https://hub.docker.com/r/trzeci/emscripten/ and I need some hand with last changes for Emscripten backend

My current observation is that 1.38.31 was the last SDK version which was possible to compile fastcomp from sources, then every other version includes this 'frozen' version. Than is if I install 1.38.40 (non-upstream) I'm getting binaries of clang that I got with 1.38.31. Is that correct?

In case of the upstream version I'm also getting aside fastcomp version:
```
./upstream/fastcomp/bin/clang --version
clang version 6.0.1 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp--clang 98df4be387dde3e3918fa5bbb5fc43e1a0e1daac) (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp 1b4148f39a69c7fc62edadd85e4122b68694dfb7) (emscripten 1.38.31 : 1.38.31)
```
Why is that needed still?

Last question about the upstream clang version which is ATM: clang version 10.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project ea134f221f2a5c075b7539876a444b4a07362912).

- is there something special with served binaries (like special compilation flags, or just a regular clang 10 release)? Is it needed to be on clang 10.0.0?

What I notice is that regardless if I install 1.38.34-upstream or 1.38.41-upstream - I'm getting exactly the same binaries of clang

Any help with solving/confirming issues from above will be helpful to get back on track with docker images.

Cheers!

[Alon Zakai]

On Mon, Aug 12, 2019 at 4:10 PM Piotr Paczkowski <pp....@gmail.com> wrote:

Hi,

I'm a maintainer of https://hub.docker.com/r/trzeci/emscripten/ and I need some hand with last changes for Emscripten backend

Great, getting wasm backend support there is very important I think!

 

My current observation is that 1.38.31 was the last SDK version which was possible to compile fastcomp from sources, then every other version includes this 'frozen' version. Than is if I install 1.38.40 (non-upstream) I'm getting binaries of clang that I got with 1.38.31. Is that correct?

Yes, fastcomp hasn't changed since 1.38.31 - it's the same code since then (we would fix any urgent things if necessary, but there haven't been any, and otherwise our focus is on the new backend).

 

In case of the upstream version I'm also getting aside fastcomp version:
```
./upstream/fastcomp/bin/clang --version
clang version 6.0.1 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp--clang 98df4be387dde3e3918fa5bbb5fc43e1a0e1daac) (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp 1b4148f39a69c7fc62edadd85e4122b68694dfb7) (emscripten 1.38.31 : 1.38.31)
```
Why is that needed still?

I'm not sure what you mean by "aside" in that sentence? And not sure I understand the question, sorry.

The directory structure is that fastcomp binaries are in a subdirectory, so ./upstream/fastcomp/bin/clang is fastcomp, while ./upstream/bin/clang (without /fastcomp/) will be upstream. So that --version output shows fastcomp (which uses old 6.0.1 clang, unlike upstream which is 10.0.0).

Last question about the upstream clang version which is ATM: clang version 10.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project ea134f221f2a5c075b7539876a444b4a07362912).

- is there something special with served binaries (like special compilation flags, or just a regular clang 10 release)? Is it needed to be on clang 10.0.0?

A special revision of clang/llvm is needed. For example a change to wasm-ld in LLVM will require a corresponding change in emscripten's code that drives wasm-ld. See the DEPS notes and links here:

https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten

Basically, that emscripten-releases repo has all the info for which revision should work with which other revision. (See also notes on which llvm tools are needed, like wasm-ld etc.)

Btw, does the Docker image use the emsdk? In that case these details wouldn't be necessary. It will also generate a .emscripten file looking at the right bin/ directory etc. I wonder if the Docker image could just do that?

What I notice is that regardless if I install 1.38.34-upstream or 1.38.41-upstream - I'm getting exactly the same binaries of clang

The fastcomp binaries will be the same, but the binaries in the bin/ (without fastcomp/) should not be.

If that doesn't answer your question, what emsdk (or other) commands specifically are you doing?

- Alon

Any help with solving/confirming issues from above will be helpful to get back on track with docker images.

Cheers!

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e8741ec1-f53e-4a39-afc1-5461a8cfee29%40googlegroups.com.

[Piotr Paczkowski]

Thank you Alon for the invaluable reply. I've managed to recover building of fastcomp with right components used.

I was investigating usage of waterfall build system, like a lot that it stores full look-up table of used components. Slightly confusing why it's hold under chromium.googlesource.com.

Eventually I had to take a decision to don't use waterfall, and go with the old `emsdk` adding some illusion that actually I can compile all tools with right versions for SDK 1.38.34+ using `emsdk install` command, mostly for the sake of users waiting to use the newest emscripten with docker.

This was primarily a step to unlock other people.

My next step is to do a quick investigation if llvm upstream can be added in a similar way. 

Further I will investigate switching to waterfall - both fastcomp + upstream. 

The ultimate step will be to have a clean solution based on waterfall, that can be used for your pipeline, so that an official image can be provided. 

Thanks!

On Tuesday, August 13, 2019 at 2:38:57 AM UTC+2, Alon Zakai wrote:

On Mon, Aug 12, 2019 at 4:10 PM Piotr Paczkowski <pp....@gmail.com> wrote:

Hi,

I'm a maintainer of https://hub.docker.com/r/trzeci/emscripten/ and I need some hand with last changes for Emscripten backend

Great, getting wasm backend support there is very important I think!

 

My current observation is that 1.38.31 was the last SDK version which was possible to compile fastcomp from sources, then every other version includes this 'frozen' version. Than is if I install 1.38.40 (non-upstream) I'm getting binaries of clang that I got with 1.38.31. Is that correct?

Yes, fastcomp hasn't changed since 1.38.31 - it's the same code since then (we would fix any urgent things if necessary, but there haven't been any, and otherwise our focus is on the new backend). 

 

In case of the upstream version I'm also getting aside fastcomp version:
```
./upstream/fastcomp/bin/clang --version
clang version 6.0.1 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp--clang 98df4be387dde3e3918fa5bbb5fc43e1a0e1daac) (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-emscripten--core-emscripten--fastcomp 1b4148f39a69c7fc62edadd85e4122b68694dfb7) (emscripten 1.38.31 : 1.38.31)
```
Why is that needed still?

I'm not sure what you mean by "aside" in that sentence? And not sure I understand the question, sorry.

 

When an `upstream` version is installed, then together with clang-10 I have clang-6 in the `emsdk` folder. 

./emsdk install 1.38.43-upstream

Installing SDK 'sdk-releases-upstream-737d4a07be76c15124adf3c6ef2c218123f7a67f-64bit'..

Installing tool 'releases-upstream-737d4a07be76c15124adf3c6ef2c218123f7a67f-64bit'..

Downloading: /home/trzeci/Projects/emscripten-docker/emsdk/zips/737d4a07be76c15124adf3c6ef2c218123f7a67f-wasm-binaries.tbz2 from https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/737d4a07be76c15124adf3c6ef2c218123f7a67f/wasm-binaries.tbz2, 160873353 Bytes

Unpacking '/home/trzeci/Projects/emscripten-docker/emsdk/zips/737d4a07be76c15124adf3c6ef2c218123f7a67f-wasm-binaries.tbz2' to '/home/trzeci/Projects/emscripten-docker/emsdk/upstream'

Done installing tool 'releases-upstream-737d4a07be76c15124adf3c6ef2c218123f7a67f-64bit'.

Installing tool 'node-8.9.1-64bit'..

Downloading: /home/trzeci/Projects/emscripten-docker/emsdk/zips/node-v8.9.1-linux-x64.tar.xz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v8.9.1-linux-x64.tar.xz, 11387108 Bytes

Unpacking '/home/trzeci/Projects/emscripten-docker/emsdk/zips/node-v8.9.1-linux-x64.tar.xz' to '/home/trzeci/Projects/emscripten-docker/emsdk/node/8.9.1_64bit'

Done installing tool 'node-8.9.1-64bit'.

Done installing SDK 'sdk-releases-upstream-737d4a07be76c15124adf3c6ef2c218123f7a67f-64bit'.

➜ find . -name clang ! -type d -exec ls -la {} \;

lrwxrwxrwx. 1 trzeci trzeci 9 Aug 30 17:15 ./upstream/fastcomp/bin/clang -> clang-6.0

lrwxrwxrwx. 1 trzeci trzeci 8 Aug 30 17:15 ./upstream/bin/clang -> clang-10

Just tried to investigate which clang is actually used 

// tested with 1.38.43

strace -f -e execve emcc ./main.cpp -o main.js |& grep clang            // clang-10

The same result whether system env EMCC_WASM_BACKEND=1 or =0

Didn't investigate further why fastcomp clang is there. 

 

The directory structure is that fastcomp binaries are in a subdirectory, so ./upstream/fastcomp/bin/clang is fastcomp, while ./upstream/bin/clang (without /fastcomp/) will be upstream. So that --version output shows fastcomp (which uses old 6.0.1 clang, unlike upstream which is 10.0.0).

Last question about the upstream clang version which is ATM: clang version 10.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project ea134f221f2a5c075b7539876a444b4a07362912).

- is there something special with served binaries (like special compilation flags, or just a regular clang 10 release)? Is it needed to be on clang 10.0.0?

A special revision of clang/llvm is needed. For example a change to wasm-ld in LLVM will require a corresponding change in emscripten's code that drives wasm-ld. See the DEPS notes and links here:

https://github.com/emscripten-core/emscripten/blob/incoming/docs/process.md#packaging-emscripten

Basically, that emscripten-releases repo has all the info for which revision should work with which other revision. (See also notes on which llvm tools are needed, like wasm-ld etc.)

Btw, does the Docker image use the emsdk? In that case these details wouldn't be necessary. It will also generate a .emscripten file looking at the right bin/ directory etc. I wonder if the Docker image could just do that?

What I notice is that regardless if I install 1.38.34-upstream or 1.38.41-upstream - I'm getting exactly the same binaries of clang

The fastcomp binaries will be the same, but the binaries in the bin/ (without fastcomp/) should not be.

If that doesn't answer your question, what emsdk (or other) commands specifically are you doing?

- Alon

Any help with solving/confirming issues from above will be helpful to get back on track with docker images.

Cheers!

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.