Using python literals for EM_CONFIG/--em-config

[Sam Clegg]

emscripten has a feature where you can avoid using a config file

completely and instead pass a python literal in the EM_CONFIG

environment variable or `--em-config` command line.

e.g.  `emcc --em-config="<python code>`

As part of routine cleanup I'm hoping to be able to remove this

feature and allow the code to depend on the existence of  a config

file on disk.

I'm not imagining there are any users of this feature, and we can

make it into warning a few releases to confirm this.

Does anyone know of any users of this feature?

cheers,

sam

[Floh]

Good riddance ;)

That reminds me that the .emscripten file is also a python script which 

must be evaluated with a python interpreter to make any sense of the content

For instance:

import os

emsdk_path = os.path.dirname(os.environ.get('EM_CONFIG')).replace('\\', '/')

...

Are there any plans on changing that to a plain text file? Might simplify things 

for build tools which try to figure out where the emscripten SDK is actually located.

I also remember that there was some obscure voodoo happening with the

"emcc --em-config=..." option, if I remember right, the passed string was assigned

to an environment variable EM_CONFIG (or even python variable?) in order

to make followup tools work, and somehow this might also have been related

to the .emscripten file...

At least that's my hazy memory from long ago when I tried to make sense of 

how "embedded SDKs" work, and what's up with .emscripten being a

python file. Any simplifications in that area are definitely welcome ;)

Cheers,

-Floh.

[Floh]

PS: I'm depending on the --em-config option (because I'm installing the emsdk with --embedded),

but I'm passing in a regular string, not some sort of python expression).

Are you going to remove the entire --em-config option, or just that it is evaluated as a python string?

I'm fine with both, as long as locally installed SDKs installed with '--embedded' continue

to work :)

[キャロウ マーク]

On May 20, 2020, at 12:28, Floh <flo...@gmail.com> wrote:

PS: I'm depending on the --em-config option (because I'm installing the emsdk with --embedded),

but I'm passing in a regular string, not some sort of python expression).

Did you not see this in another thread?

We have made a number of changes over the last few weeks that affect the emscripten cache.  The biggest one being that the emscripten cache now lives inside the emscripten directory by default.  This means that it doesn't live in your $HOME and is not shared between emscripten installations.   This also means that `--embedded` flag to emsdk no longer affects the cache since the cache is always embedded.

Regards

    -Mark

[Sam Clegg]

Passing a filename as --em-config or in EM_CONFIG will continue to work yes.  Thats the plan.

By the way if you use `emsdk --embedded` these days it should just work to call emcc without `--em-config`.   enscripten now knows to look for it config relative to itself (firstly its own directory and then it looks two levels up for the emsdk embedded config).

Regarding parsing `~/.emscripten` in order to find emscripten, I really want to stop people from trying to do this.   I'd much rather the PATH was way one finds emscripten.   e.g. `which emcc`.   Failing that an environment variable.   But other tools assuming the location and format of the `.emscripten` config file is not a good idea IMHO.  I know some tools do this already but hopefully we can teach them other ways.

cheers,

sam

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e48aea72-28ad-4121-bc8c-d3414dccf1b1%40googlegroups.com.

[Steven Johnson]

Halide uses the EM_CONFIG variable in its build script.

The reason for this is that we need to ensure that LLVM_ROOT points to the specific version of LLVM being used by the rest of Halide. (Having these out of sync can cause amusingly bad things to happen.) We slurp in the default ~/.emscripten config file and make an altered version (with LLVM_ROOT pointing to the LLVM we are using), so that we can always build with the 'correct' version (ie the one that the rest of Halide is using).

Requiring that ~/.emscripten be edited is problematic as Halide supports multiple versions of LLVM at any one time (currently, v9, v10, and v11/trunk), so this file would need to edited in lockstep with other things in Halide to keep things in step.

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28seE50BNe4BKJ4cFzoQNNZPE6xbFFwDvXQSdWfiF2WAA%40mail.gmail.com.

[Sam Clegg]

On Wed, May 20, 2020 at 2:12 PM 'Steven Johnson' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

Halide uses the EM_CONFIG variable in its build script.

The reason for this is that we need to ensure that LLVM_ROOT points to the specific version of LLVM being used by the rest of Halide. (Having these out of sync can cause amusingly bad things to happen.) We slurp in the default ~/.emscripten config file and make an altered version (with LLVM_ROOT pointing to the LLVM we are using), so that we can always build with the 'correct' version (ie the one that the rest of Halide is using).

Requiring that ~/.emscripten be edited is problematic as Halide supports multiple versions of LLVM at any one time (currently, v9, v10, and v11/trunk), so this file would need to edited in lockstep with other things in Halide to keep things in step.

Thanks for the heads up steven!

We had another report of someone using it, so I reverted the warning for now.

I'm hoping we can move both of you in different direction over time.   For example, did you know that you can set the LLVM_ROOT using the `EM_LLVM_ROOT` environment variable?  This seems more targeted and easier for you.  Any variable that you can set from the config file can also be set via the environment in the form of `EM_<NAME>`.

cheers,

sam

 

On Tue, May 19, 2020 at 11:46 AM 'Sam Clegg' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

emscripten has a feature where you can avoid using a config file

completely and instead pass a python literal in the EM_CONFIG

environment variable or `--em-config` command line.

e.g.  `emcc --em-config="<python code>`

As part of routine cleanup I'm hoping to be able to remove this

feature and allow the code to depend on the existence of  a config

file on disk.

I'm not imagining there are any users of this feature, and we can

make it into warning a few releases to confirm this.

Does anyone know of any users of this feature?

cheers,

sam

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28seE50BNe4BKJ4cFzoQNNZPE6xbFFwDvXQSdWfiF2WAA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAM%3DdnveGTKtEqGWue_qyUMjdyLdGXZbXEm8m4fhPjreRgd4tGQ%40mail.gmail.com.

[Steven Johnson]

I did not know that. That sounds like a much better approach anyway :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va29xtsd5emac0nfmvUys84xaBdx-xMes515U1sATFPJe1w%40mail.gmail.com.

[Sam Clegg]

On Wed, May 20, 2020 at 6:55 PM 'Steven Johnson' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

I did not know that. That sounds like a much better approach anyway :-)

I'll hold off changing any of this until you can make your change downstream.  Can you let me know when have converted?

cheers,

sam 

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAM%3Ddnvf79x%3DXt_CLym76VCoTeJBmpu0pw9%2BWWR0CH6ixTcjptg%40mail.gmail.com.

[Floh]

> I really want to stop people from trying to do this.   I'd much rather the PATH was way one finds emscripten.   e.g. `which emcc`.   Failing that an environment variable.

What about having several emsdk installations side by side though, without any of them being the "default version".

But if I understand your other posts right, that won't be an issue in the future (e.g. if all the tools in the SDK can figure out the location of "their" SDK without external hints like the --em-config or --cache command line args). Ideally this should work without having the PATH or setting "global" environment variables.

[Floh]

Ok, great news, I also removed the --em-config arg now (in addition to the --cache arg) and it works perfectly, this simplifies my cmake toolchain setup a lot. I'm passing in the absolute path to the directory where the emcc etc... tools are located, and in the cmake toolchain file set the compiler variables to absolute paths (so "/abs/path/to/emscripten/emcc").

No environment variables or additions to the PATH variables needed, only the absolute paths to the toolchain tools. It's perfect now :)

-Floh.

[Steven Johnson]

It will be next week at the earliest -- I am currently elbow-deep in work on the Halide build/test setup (and, as it turns out, the emcc testing hasn't been enabled for our public repo recently, so it might have broken for other reasons).

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2_CFGdXYRhbB2Vx3Cim8qo0Z1NuErepc-GapsAmDWNBLQ%40mail.gmail.com.

[Sam Clegg]

On Thu, May 21, 2020 at 9:41 AM Floh <flo...@gmail.com> wrote:

> I really want to stop people from trying to do this.   I'd much rather the PATH was way one finds emscripten.   e.g. `which emcc`.   Failing that an environment variable.

What about having several emsdk installations side by side though, without any of them being the "default version".

If you mean literally multiple emsdk checkouts, then yes, with --embedded allows them to work at the same time with neither one being the default.

If you mean one checkout of emsdk with several SDKs inside it?   Hasn't this setup always required a default version?  Haven't you always had to switch between them using `emsdk activate`  (I believe that has always been the case).   I do have future plans to make even that unnecessary but that would require embedding config even deeper, inside the emscripten directory itself: https://github.com/emscripten-core/emsdk/pull/367.   

 

But if I understand your other posts right, that won't be an issue in the future (e.g. if all the tools in the SDK can figure out the location of "their" SDK without external hints like the --em-config or --cache command line args). Ideally this should work without having the PATH or setting "global" environment variables.

Yes that is the plan.  I would like for the execution of /path/to/emcc to more-or-less always just work based on its own embedded configuration, with no single emscripten being the default one.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/706fb54f-f3bb-4a3e-8cc0-d3c28d438a08%40googlegroups.com.

[Floh]

> If you mean literally multiple emsdk checkouts, then yes, with --embedded allows them to work at the same time with neither one being the default.

This is the important use case for me, not the other one. I have a setup helper script which installs a "local" emsdk in a subdirectory with --embedded, and a tailored cmake toolchain file which uses this local emsdk. This is basically to make sure that this local emsdk version isn't interfering with the global SDK the user might have installed, or other local emsdk's in other locations.

It already works as expected, and now that this works without --cache and --em-config it's easier then ever. I'm very happy with your changes :)

[Sam Clegg]

And soon `--embedded` will be the default so this should be even easier:  https://github.com/emscripten-core/emsdk/pull/472

 

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/51289c8c-f3ac-43ce-a68f-3ddfc1479c41%40googlegroups.com.

[Sam Clegg]

On Thu, May 21, 2020 at 11:10 AM 'Steven Johnson' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

It will be next week at the earliest -- I am currently elbow-deep in work on the Halide build/test setup (and, as it turns out, the emcc testing hasn't been enabled for our public repo recently, so it might have broken for other reasons).

Hi Steven,

I'm hoping to re-land this deprecation this week:  https://github.com/emscripten-core/emscripten/pull/11684

Have you managed to transition away from using this feature yet?

cheers,

sam

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAM%3DdnvcWFdmoiG-1-z2yhPqgEHdSXs5aRHJpc%2BTtQXL26Z0pGA%40mail.gmail.com.

[Sam Clegg]

FYI, I'm hoping to remove this feature this week: https://github.com/emscripten-core/emscripten/pull/13855

IIUC nobody is relying on it anymore.

[Floh]

I guess this was released in 2.0.17 yesterday (according to the release notes at least)? Happy to report that nothing broke on my side ;)

(I'm extracting the EMSCRIPTEN_ROOT value for use in a cmake toolchain file by evaluating the .emscripten config file, and this still seems to work, I'm still setting an EM_CONFIG environment variable before running the .emscripten file through python, but I guess this isn't needed anymore now, but it doesn't hurt either).

Cheers!

[Sam Clegg]

On Tue, Apr 13, 2021 at 3:46 AM Floh <flo...@gmail.com> wrote:

I guess this was released in 2.0.17 yesterday (according to the release notes at least)? Happy to report that nothing broke on my side ;)

(I'm extracting the EMSCRIPTEN_ROOT value for use in a cmake toolchain file by evaluating the .emscripten config file, and this still seems to work, I'm still setting an EM_CONFIG environment variable before running the .emscripten file through python, but I guess this isn't needed anymore now, but it doesn't hurt either).

Its really a separate issue but I'd love to move folks away from parsing `.emscripten` using non-emscripten tools.   Do you think we can find a way to move you away from doing that?

How about:

a) Use `which emcc` to find where emscripten is

b) Have emsdk set `EMSCRIPTEN_ROOT` directly in the environment rather than in the config file. 

The reason I don't like (b) much is that we have to deal with the case where somebody runs `path/foo/emcc` on the command line with `EMSCRIPTEN_ROOT=path/bar` in the environment.   We already have this issue with `EMSCRIPTEN_ROOT` in the config file, and I'd love to remove it one day.

Regardless, unless I'm misunderstanding this is unrelated to this thread which is about putting python literals in the EM_CONFIG environment.. You aren't relying on that are you?

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/185eae5b-2ee6-462d-8719-1210204d4f5cn%40googlegroups.com.

[Floh]

Yeah, I never used python literals in EM_CONFIG, misunderstanding on my part :)

About the other thing:

> a) Use `which emcc` to find where emscripten is

In my case, emcc is not in the path because this is a local installation (via "emsdk activate --embedded") without any traces in the environment (I'm setting EM_CONFIG only temporary in the "python environment" before evaluating .emscripten as a workaround.

> b) Have emsdk set `EMSCRIPTEN_ROOT` directly in the environment rather than in the config file. 

...this also isn't an option for the same reason as above, there is no "global emscripten installation" at all, and there may be multiple installations of the emscripten SDK with different versions side by side.

Cmake takes are of making all paths (compiler tools, header search paths etc...) absolute starting at EMSCRIPTEN_ROOT (which cmake doesn't read from an environement variable but gets as command line argument, like cmake -DEMSCRIPTEN_ROOT=xxx.

However as long as EMSCRIPTEN_ROOT is always is at "/upstream/emscripten" under the SDK's installation directory, I think I don't need to "parse" .emscripten any longer (IFIR in the past this wasn't true, it was something like "/[version]/emscripten", and this was why I had to peek into .emscripten in the first place.

Some other alternatives which would work for me:

- turn .emscripten into a plain text file with simple "key = value" lines which is generated at SDK installation time (no more python magic)

- ...or add a subcommand to the emsdk tool which prints the emscripten root directory to stdout (which I could then call in my cmake wrapper scripts)

I basically need a way to get the absolute directory where emcc is located if all I know is the absolute directory where the emsdk tool is located.

Cheers!

-Andre.

[Floh]

Small PS to my last post:

In my 'cmake wrapper' (which also takes care of managing a lokal "per-workspace" emscripten SDK installation) I have added the following subcommands on top of emsdk (other command simply wrap existing emsdk subcommands)

# show path of the local .emscripten file:

> fips emsdk show-em-config

E:/projects/fips-sdks/emsdk/.emscripten

# show the emscripten root directory (extracted from .emscripten)

> fips emsdk show-emscripten-root

E:/projects/fips-sdks/emsdk/upstream/emscripten

# and finally, show the entire evaluated .emscripten file (mostly useful for build system debugging, the only important part is EMSCRIPTEN_ROOT

> fips emsdk show-config

NODE_JS: E:/projects/fips-sdks/emsdk/node/12.18.1_64bit/bin/node.exe

PYTHON: E:/projects/fips-sdks/emsdk/python/3.7.4-pywin32_64bit/python.exe   

JAVA: E:/projects/fips-sdks/emsdk/java/8.152_64bit/bin/java.exe

LLVM_ROOT: E:/projects/fips-sdks/emsdk/upstream/bin

BINARYEN_ROOT: E:/projects/fips-sdks/emsdk/upstream

EMSCRIPTEN_ROOT: E:/projects/fips-sdks/emsdk/upstream/emscripten

TEMP_DIR: E:/projects/fips-sdks/emsdk/tmp

COMPILER_ENGINE: E:/projects/fips-sdks/emsdk/node/12.18.1_64bit/bin/node.exe

JS_ENGINES: ['E:/projects/fips-sdks/emsdk/node/12.18.1_64bit/bin/node.exe']

...the users can have as many such workspaces (in this case "E:/projects") as they want side by side, with different SDK versions (I'm managing the Android SDK/NDK in a similar way for instance)

[Sam Clegg]

On Tue, Apr 13, 2021 at 1:27 PM Floh <flo...@gmail.com> wrote:

Yeah, I never used python literals in EM_CONFIG, misunderstanding on my part :)

About the other thing:

> a) Use `which emcc` to find where emscripten is

In my case, emcc is not in the path because this is a local installation (via "emsdk activate --embedded") without any traces in the environment (I'm setting EM_CONFIG only temporary in the "python environment" before evaluating .emscripten as a workaround.

Is there some reason you don't want to add the emscripten directory to you PATH?  Is that somehow worse than setting EM_CONFIG and then parsing it?

> b) Have emsdk set `EMSCRIPTEN_ROOT` directly in the environment rather than in the config file. 

...this also isn't an option for the same reason as above, there is no "global emscripten installation" at all, and there may be multiple installations of the emscripten SDK with different versions side by side.

I'm not sure I understand.  You say you are setting `EM_CONFIG` in your environment according to your needs, so that there is not one global `EM_CONFIG`.  Is there some reason you couldn't alternatively set `EMSCRIPTEN_ROOT`?

I don't see how either of these imply some "global emscripten installation".  In both cases the current environment determins the installation to use, no?.   In other words, would it be possible (in your setup) to point to emscripten directly via `EMSCRIPTEN_ROOT` rather than pointing at a config file via `EM_CONFIG` (and then parsing the config file to find EMSCRIPTEN_ROOT)?  (it would then be up to the emcc executable to find its own config file).

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/2653d96f-9437-449b-a0dd-7938d0fdd7e9n%40googlegroups.com.

[Floh]

> Is there some reason you don't want to add the emscripten directory to you PATH?  Is that somehow worse than setting EM_CONFIG and then parsing it?

Because there isn't a single "emscripten directory", but one emscripten directory per "workspace directory". I might cd to a different workspace with its own embedded emscripten SDK installation at any time in the same terminal session. With my current setup, the correct emscripten SDK is automatically selected depending on the directory the user is currently in (and then when running cmake to generate the build scripts, cmake will "bake" the absolute paths into the generated build scripts based on a cmake option which points to EMSCRIPTEN_ROOT (which however doesn't exist as environment variable)).

> You say you are setting `EM_CONFIG` in your environment [...]  Is there some reason you couldn't alternatively set `EMSCRIPTEN_ROOT`?

EM_CONFIG is not set persistently in the user environment, it's only set temporary inside a python script (as "os.environ['EM_CONFIG'] = ...") before the .emscripten file is evaluated, and only because of this one line in the .emscripten file: 

emsdk_path = os.path.dirname(os.environ.get('EM_CONFIG')).replace('\\', '/')

The EM_CONFIG environment variable doesn't survive execution of this python script however.

> to point to emscripten directly via `EMSCRIPTEN_ROOT` ...

Isn't this a chicken-egg problem? The EMSCRIPTEN_ROOT variable isn't set anywhere on my machine, except after running the platform-specific emsdk_env.xxx scripts (which I don't usually do, except when I need emcc in the path for some reason). I'd need to run this script (or 'emsdk construct_env') of a specific emscripten SDK installation from inside python. But running emsdk from within Python which sets environment variables doesn't pass this new environment up to the Python process where the environment variables would be needed for invoking cmake (AFAIK environment variables can only be inherited from parent to child processes, but not the other way around).

All in all, the way the current Emscripten SDK works when activated as "--embedded" is pretty much perfect because everything already works without anything in the PATH or in environment variables. The only wonky part is how to get from the SDK installation directory (where 'emsdk' is located - which is under my control) to the 'compiler toolchain directory' which is needed by cmake (e.g. "[emsdk_dir]/upstream/emscripten"). I consider the current way to evaluate .emscripten inside python a hack, but it's a reasonably simple hack that works ;)

Any alternative should be simpler then that though ("emsdk construct_env" does a lot more things then just setting up 'EMSCRIPTEN_ROOT', and as I described above, doesn't really work for my use case because the modified environment is "lost" in a child process).

Phew :)

-Floh.

[Sam Clegg]

On Wed, Apr 14, 2021 at 2:44 AM Floh <flo...@gmail.com> wrote:

> Is there some reason you don't want to add the emscripten directory to you PATH?  Is that somehow worse than setting EM_CONFIG and then parsing it?

Because there isn't a single "emscripten directory", but one emscripten directory per "workspace directory". I might cd to a different workspace with its own embedded emscripten SDK installation at any time in the same terminal session. With my current setup, the correct emscripten SDK is automatically selected depending on the directory the user is currently in (and then when running cmake to generate the build scripts, cmake will "bake" the absolute paths into the generated build scripts based on a cmake option which points to EMSCRIPTEN_ROOT (which however doesn't exist as environment variable)).

> You say you are setting `EM_CONFIG` in your environment [...]  Is there some reason you couldn't alternatively set `EMSCRIPTEN_ROOT`?

EM_CONFIG is not set persistently in the user environment, it's only set temporary inside a python script (as "os.environ['EM_CONFIG'] = ...") before the .emscripten file is evaluated, and only because of this one line in the .emscripten file: 

emsdk_path = os.path.dirname(os.environ.get('EM_CONFIG')).replace('\\', '/')

The EM_CONFIG environment variable doesn't survive execution of this python script however.

> to point to emscripten directly via `EMSCRIPTEN_ROOT` ...

Isn't this a chicken-egg problem? The EMSCRIPTEN_ROOT variable isn't set anywhere on my machine, except after running the platform-specific emsdk_env.xxx scripts (which I don't usually do, except when I need emcc in the path for some reason). I'd need to run this script (or 'emsdk construct_env') of a specific emscripten SDK installation from inside python. But running emsdk from within Python which sets environment variables doesn't pass this new environment up to the Python process where the environment variables would be needed for invoking cmake (AFAIK environment variables can only be inherited from parent to child processes, but not the other way around).

All in all, the way the current Emscripten SDK works when activated as "--embedded" is pretty much perfect because everything already works without anything in the PATH or in environment variables. The only wonky part is how to get from the SDK installation directory (where 'emsdk' is located - which is under my control) to the 'compiler toolchain directory' which is needed by cmake (e.g. "[emsdk_dir]/upstream/emscripten"). I consider the current way to evaluate .emscripten inside python a hack, but it's a reasonably simple hack that works ;)

Any alternative should be simpler then that though ("emsdk construct_env" does a lot more things then just setting up 'EMSCRIPTEN_ROOT', and as I described above, doesn't really work for my use case because the modified environment is "lost" in a child process).

Thanks for all the details.  I feel confident we can come up with a nicer version of your current hack if we work together.   I will open an emsdk issue for this and maybe reach out to you on discord (are you there?) to chat about the details.

cheers,

sam

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/6be4ea24-59cd-4a96-9c75-644a1c18e395n%40googlegroups.com.

[Floh]

I'm not on Discord, just mention me in the ticket (floooh) and I'll be there ;)