System.get_env/0 Performance Improvement

[Trevor Brown]

Background

I was recently working on a project that invoked `System.get_env/0` on every request it handled. Someone changed the infrastructure the application was deployed on and greatly increased the number of environment variables (3x more) in the environment the Elixir application ran in, and the performance of the application degraded by about that same amount (3x slower than before). When I profiled the application I quickly found that most of the request handling was spent waiting for the `System.get_env/0` function to return. I changed the code so `System.get_env/0` was only invoked on startup and the performance issue was resolved.

Problem

When I was working on the performance issue in the app the flame graphs revealed something interesting - Erlang was actually formatting the environment variables from a proplist to a list of strings in certain format, and then Elixir was undoing the formatting by parsing the strings back to a map! Because of this the performance of the `System.get_env/0` function call is tied to the number of variables in the environment.

• The Erlang code that formats the variables: https://github.com/erlang/otp/blob/master/lib/kernel/src/os.erl#L118
• The Elixir code that parses them and puts them in a dictionary: https://github.com/elixir-lang/elixir/blob/v1.10.3/lib/elixir/lib/system.ex#L438

Possible Solutions

1. Update the `System.get_env/0` function to use `os:list_env_vars/0` function

This function already exists returns a proplists in the format `{VariableName, VariableValue}`. All we need to do is replace the existing code with

def get_env do
   Enum.into(:os.list_env_vars(), %{})
end

This would solve the performance issue and simplify the Elixir code! The downside is that `os:list_env_vars/0` is not a documented function, which I assume means it's not recommended for other applications to rely on it.

2. Wait for Erlang to change

Because I assumed `os:list_env_vars/0` is not a function that Elixir should be relying on (although nothing prevents it from doing so), I have created an enhancement ticket on the Erlang issue tracker to document `os:list_env_vars/0` or implement another function that returns the environment variables in a proplist with `{VariableName, VariableValue}` tuples - https://bugs.erlang.org/browse/ERL-1332. Once Erlang implements a documented function that does this we could then begin updating Elixir.

Thoughts? I doubt this is something that is run into very often but it could be easily improved.

[José Valim]

We need to go route 2, as 1 can change at any time (including if 2 is implemented). So thank you for opening an issue on Erlang's issues tracker, I am now watching it and I can act based on its status. :)

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0aecb090-2cbb-46af-8ff4-e34337b123d4o%40googlegroups.com.

[Alexei Sholik]

The use of String.split instead of :binary.split seems excessive here. Could be a small but quick win.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KHzHHTw_mU6vCUoK3yPYT5cQQBWicMkQ5cGTpZTmT26Q%40mail.gmail.com.

--

Best regards
Alexei Sholik

[Trevor Brown]

:binary.split only works on binaries and os:getenv/0 returns a list of lists.

On Tuesday, August 18, 2020 at 4:04:00 AM UTC-4, alco wrote:

The use of String.split instead of :binary.split seems excessive here. Could be a small but quick win.

On Tue, Aug 18, 2020 at 9:03 AM José Valim <jose...@dashbit.co> wrote:

We need to go route 2, as 1 can change at any time (including if 2 is implemented). So thank you for opening an issue on Erlang's issues tracker, I am now watching it and I can act based on its status. :)

On Mon, Aug 17, 2020 at 11:45 PM Trevor Brown <ad...@stratus3d.com> wrote:

Background

I was recently working on a project that invoked `System.get_env/0` on every request it handled. Someone changed the infrastructure the application was deployed on and greatly increased the number of environment variables (3x more) in the environment the Elixir application ran in, and the performance of the application degraded by about that same amount (3x slower than before). When I profiled the application I quickly found that most of the request handling was spent waiting for the `System.get_env/0` function to return. I changed the code so `System.get_env/0` was only invoked on startup and the performance issue was resolved.

Problem

When I was working on the performance issue in the app the flame graphs revealed something interesting - Erlang was actually formatting the environment variables from a proplist to a list of strings in certain format, and then Elixir was undoing the formatting by parsing the strings back to a map! Because of this the performance of the `System.get_env/0` function call is tied to the number of variables in the environment.

• The Erlang code that formats the variables: https://github.com/erlang/otp/blob/master/lib/kernel/src/os.erl#L118
• The Elixir code that parses them and puts them in a dictionary: https://github.com/elixir-lang/elixir/blob/v1.10.3/lib/elixir/lib/system.ex#L438

Possible Solutions

1. Update the `System.get_env/0` function to use `os:list_env_vars/0` function

This function already exists returns a proplists in the format `{VariableName, VariableValue}`. All we need to do is replace the existing code with

def get_env do
   Enum.into(:os.list_env_vars(), %{})
end

This would solve the performance issue and simplify the Elixir code! The downside is that `os:list_env_vars/0` is not a documented function, which I assume means it's not recommended for other applications to rely on it.

2. Wait for Erlang to change

Because I assumed `os:list_env_vars/0` is not a function that Elixir should be relying on (although nothing prevents it from doing so), I have created an enhancement ticket on the Erlang issue tracker to document `os:list_env_vars/0` or implement another function that returns the environment variables in a proplist with `{VariableName, VariableValue}` tuples - https://bugs.erlang.org/browse/ERL-1332. Once Erlang implements a documented function that does this we could then begin updating Elixir.

Thoughts? I doubt this is something that is run into very often but it could be easily improved.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0aecb090-2cbb-46af-8ff4-e34337b123d4o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KHzHHTw_mU6vCUoK3yPYT5cQQBWicMkQ5cGTpZTmT26Q%40mail.gmail.com.

[Alexei Sholik]

The same is true for String.split, that's not the point.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c3b36f04-4062-42d9-ba7d-7aa5fda8134ao%40googlegroups.com.