Error: Environment size exceeded TOTAL_ENV_SIZE!

[mertes...@gmail.com]

Hello

I made progress in calling a C function from JavaScript.
Many thanks to Floh. But now I have a different problem:
Setting the environment with setenv() triggers the error:

exception thrown: Error: Environment size exceeded TOTAL_ENV_SIZE!,Error: Environment size exceeded TOTAL_ENV_SIZE!
    at ___buildEnvironment (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1791:15)
    at _setenv (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1944:7)
    at wasm-function[21]:89
    at Module._setEnvVar (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:2138:38)
    at ccall (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:636:18)
    at /run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:644:12
    at /run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1708:156
    at Array.forEach (<anonymous>)
    at Array.ASM_CONSTS (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1708:132)
    at _emscripten_asm_const_i (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1711:26)

This is a problem.
When I execute a program with node.js I REALLY need
the environment. Therefore I use code to copy the system
environment to the emscripten environment. This is just
done when running under node.js.

My test program is:

-------------------- Begin of tst265.c --------------------
#include "stdio.h"
#include "string.h"
#include "stdlib.h"
#include "emscripten.h"


EMSCRIPTEN_KEEPALIVE void setEnvVar (char *key, char *value)

  { /* setEnvVar */
    printf("setEnvVar(\"%s\", \"%s\")\n", key, value);
    setenv(key, value, 1);
  } /* setEnvVar */


int main (int argc, char *argv[])

  { /* main */
    printf("begin main\n");
    EM_ASM(
      let setEnvVar = Module.cwrap('setEnvVar', 'number', ['string', 'string']);
      Object.keys(process.env).forEach(function(key) {
        setEnvVar(key, process.env[key]);
      });
    );
    printf("end main\n");
  } /* main */
-------------------- End of tst265.c --------------------

I compile the test program with:

  emcc tst265.c -s EXTRA_EXPORTED_RUNTIME_METHODS=['ccall','cwrap'] -o tst265.js

Then I execute the program with:

  node tst265.js

The program writes several log lines, but then I get the error:

-------------------- Begin output of tst265 --------------------
begin main
setEnvVar("SHELL", "/bin/bash")

... more setEnvVar lines follow ...

setEnvVar("GPG_TTY", "/dev/pts/6")
exception thrown: Error: Environment size exceeded TOTAL_ENV_SIZE!,Error: Environment size exceeded TOTAL_ENV_SIZE!
    at ___buildEnvironment (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1791:15)
    at _setenv (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1944:7)
    at wasm-function[21]:89
    at Module._setEnvVar (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:2138:38)
    at ccall (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:636:18)
    at /run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:644:12
    at /run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1708:156
    at Array.forEach (<anonymous>)
    at Array.ASM_CONSTS (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1708:132)
    at _emscripten_asm_const_i (/run/media/tm/disk2_460GiB/home/tm/seed7_5/src/tst265.js:1711:26)
-------------------- End output of tst265 --------------------

I looked at the functions _setenv() and ___buildEnvironment().
The whole thing looks strange. The functions _getenv() and _setenv()
use a map to store the environment variables. So far this is okay.
After _setenv() has finished it calls ___buildEnvironment() to set up
the 'environ' variable. The function ___buildEnvironment() does the
time consuming work. The whole map of environment values is processed
and all is written into a buffer of fixed size (actually there are two
buffers). When the fixed size is not enough the error is triggered.

Since the function ___buildEnvironment() is already heavvy I propose:
The function ___buildEnvironment() could determine the size necessary
for the 'environ' buffer and allocate a buffer with the correct size.
Actually ___buildEnvironment() uses two buffers, but this can be done
for both buffers. Afterwards the buffer(s) could be filled without the
danger to get an error. Okay, that would be even more time consuming,
than just processing all environment values from the map. But it is
only done, when the environment is changed with setenv(). Additionally
it would work without "Environment size exceeded" error. Since I am
not a JavaScript/Emscripten expert I did not try to implement my
proposed solution, sorry.

Another question is: The functions getenv() and setenv() can be easily
implemented in C. This could be compiled easily to WebAssembly. Why is
a solution with JavaScript maps used, when a pure C implementation
would work without fixed buffer size limitation?

Just for fun I wrote getenv() and setenv() myself in C.
Feel free to use it:

-------------------- Begin of code snippet --------------------
char **environ = NULL;


char *getenv (const char *name)

  {
    size_t len;
    char **p, *c;

  /* getenv */
    if (name == NULL || environ == NULL || strchr(name, '=') != NULL) {
      return NULL;
    } else {
      len = strlen(name);
      for (p = environ; (c = *p) != NULL; ++p) {
        if (strncmp(c, name, len) == 0 && c[len] == '=') {
          return &c[len + 1];
        } /* if */
      } /* for */
      return NULL;
    } /* if */
  } /* getenv */


int setenv (const char *name, const char *value, int overwrite)

  {
    size_t len;
    char **p, *c;
    size_t nameCount = 0;

  /* setenv */
    if (name == NULL || name[0] == '\0' || strchr(name, '=') != NULL) {
      errno = EINVAL;
      return -1;
    } else {
      len = strlen(name);
      if (environ != NULL) {
        for (p = environ; (c = *p) != NULL; ++p) {
          nameCount ++;
          if (strncmp(c, name, len) == 0 && c[len] == '=') {
            if (overwrite) {
              if ((*p = realloc(*p, len + strlen(value) + 2)) == NULL) {
        errno = ENOMEM;
        return -1;
          } else {
        strcpy(&c[len + 1], value);
          } /* if */
        } /* if **/
        return 0;
          } /* if */
        } /* for */
      } /* if */
      if ((environ = realloc(environ, nameCount + 2)) == NULL ||
          (c = malloc(len + strlen(value) + 2)) == NULL) {
        errno = ENOMEM;
        return -1;
      } else {
        memcpy(c, name, len);
        c[len] = '=';
    strcpy(&c[len + 1], value);
        environ[nameCount] = c;
        environ[nameCount + 1] = NULL;
        return 0;
      } /* if */
    } /* if */
  } /* setenv */
-------------------- Begin of code snippet --------------------

I had not the time to make tests with getenv() and setenv(), but
"It compiles so it must be okay". (Copyright Thomas Mertes) :-)
Feel free to use it for emscripten.

As I already said: When my program(s) are executed with node.js I
REALLY need the environment. Hopefully somebody fixes this
"Environment size exceeded" error.


Regards,
Thomas Mertes

--
Seed7 Homepage:  http://seed7.sourceforge.net
Seed7 - The extensible programming language: User defined statements
and operators, abstract data types, templates without special
syntax, OO with interfaces and multiple dispatch, statically typed,
interpreted or compiled, portable, runs under linux/unix/windows.

[mertes...@gmail.com]

At Friday, 26. April 2019 10:32:34 UTC+2 mertes...@gmail.com wrote:

    Hello

    I made progress in calling a C function from JavaScript.

   ...

    I had not the time to make tests with getenv() and setenv(), but
    "It compiles so it must be okay". (Copyright Thomas Mertes) :-)
    Feel free to use it for emscripten.

Famous last words ...
Now I had time to do some tests with my C implementations of
getenv() and setenv(). And I have to admit: There was actually
a bug in setenv(). A multiplication with sizeof(char *) was
missing in the realloc...

Below are the newest versions of getenv() and setenv().
Feel free to use them for Emscripten.

      if ((environ = realloc(environ7, sizeof(char *) * (nameCount + 2))) == NULL ||

          (c = malloc(len + strlen(value) + 2)) == NULL) {
        errno = ENOMEM;
        return -1;
      } else {
        memcpy(c, name, len);
        c[len] = '=';
        strcpy(&c[len + 1], value);
        environ[nameCount] = c;
        environ[nameCount + 1] = NULL;
        return 0;
      } /* if */
    } /* if */
  } /* setenv */

-------------------- End of code snippet --------------------

I would really prefer, when the runtime library of emcc
would have a working getenv() and setenv() without

  Error: Environment size exceeded TOTAL_ENV_SIZE!

This is not a way a C run-time function should work.
Artificial limitations like: "The size of the environment
must be less than 1024", should be a thing of the past.

Sorry for the rant. Emscripten is great and I succeed in
compiling the Seed7 interpreter with it. The test programs
for the Seed7 interpreter succeed. The latest release of
Seed7 contains the changes necessary to compile Seed7
with emcc. You can download seed7_05_20190407.tgz from

  https://sourceforge.net/projects/seed7/files/

The file seed7/src/read_me.txt contains detailed
information how to compile Seed7 with emcc.

Many thanks to the developers of Emscripten.
Keep on with the good work.

[Alon Zakai]

Glad you figured this out ! Can you please open a PR with your improvements?

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mertes...@gmail.com]

On Mon, Apr 29, 2019 at 18:23:48 UTC+2 Alon Zakai wrote:

    Glad you figured this out ! Can you please open a PR with your improvements?

Sorry to say, but my proposals are not ready for a pull request.
I proposed two possibilities to solve this issue:

  1. Extend the existing implementation.
  2. Switch to a C implementation.

In both cases there are several questions open.

 - Are the maps used by the JavaScript implementation of
   getenv() and setenv() used elsewhere from JavaScript?
 - The current JavaScript implementation puts the whole
   environment into one block of data. Is this necessary?
 - My C implementation of getenv() and setenv() does not
   create the environment in one block of data. Is this
   acceptable?

Somebody with more knowledge about Emscripten is probably
able to use my proposals to write a solution.

At the end the function setenv() should NEVER trigger:

  Error: Environment size exceeded TOTAL_ENV_SIZE!

This is just not the way a C library function should work.

[Sam Clegg]

From: <mertes...@gmail.com>
Date: Fri, May 3, 2019 at 1:41 AM
To: emscripten-discuss

> On Mon, Apr 29, 2019 at 18:23:48 UTC+2 Alon Zakai wrote:
>
> Glad you figured this out ! Can you please open a PR with your improvements?
>
> Sorry to say, but my proposals are not ready for a pull request.
> I proposed two possibilities to solve this issue:
>
> 1. Extend the existing implementation.
> 2. Switch to a C implementation.
>
> In both cases there are several questions open.
>
> - Are the maps used by the JavaScript implementation of
> getenv() and setenv() used elsewhere from JavaScript?
> - The current JavaScript implementation puts the whole
> environment into one block of data. Is this necessary?
> - My C implementation of getenv() and setenv() does not
> create the environment in one block of data. Is this
> acceptable?
>
> Somebody with more knowledge about Emscripten is probably
> able to use my proposals to write a solution.
>
> At the end the function setenv() should NEVER trigger:
>
> Error: Environment size exceeded TOTAL_ENV_SIZE!
>
> This is just not the way a C library function should work.
>

Technically according to the manpage setenv is allowed to fail with
out of memory:

ERRORS
EINVAL name is NULL, points to a string of length 0, or
contains an '=' character.

ENOMEM Insufficient memory to add a new variable to the environment.

However I imagine in this case you are seeing a bug in the
implementation rather than actual memory exhaustion so we probably do
need to fix emscripten.

> Regards,
> Thomas Mertes
>
> --
> Seed7 Homepage: http://seed7.sourceforge.net
> Seed7 - The extensible programming language: User defined statements
> and operators, abstract data types, templates without special
> syntax, OO with interfaces and multiple dispatch, statically typed,
> interpreted or compiled, portable, runs under linux/unix/windows.
>

[Thomas Mertes]

As you said, I am not seeing this as memory exhaustion.

Fixing emscripten in this regard would be really really helpful.

Emscripten works great and this problem is one of the things that

hinders my efforts to compile and run my project (Seed7) with emscripten.

Many thanks in advance for the hero, who fixes setenv().