Re: [elixir-core:9161] Running phoenix locally in Docker on Mac

[José Valim]

There is no way to patch out Mix.Utils to run your own code. The proper fix would be to figure out why the path traversal is slow on Elixir (it is implemented on top of filelib:wildcard) so we can fix it rather permanently. How long does "Path.wildcard("lib/**/*.ex")" take? Can you implement your own path traversal in Elixir? Is it faster than Path.wildcard?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Mon, Oct 14, 2019 at 8:08 PM Michael St Clair <micha...@gmail.com> wrote:

I've been doing some digging around to see how to make phoenix run faster in Docker. Without this plug the app runs just as fast as native `plug(Phoenix.CodeReloader`. I think I have traced it down to this https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237. Running just that function explains most if not all of the extra load time on my endpoints. However this code `"ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") |> IO.inspect` appears to do the same thing and much faster. I'm just running into the problem of how to override that Mix.Utils file to test it here https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36. Any suggestions?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com.

[Michael St Clair]

What is the best way to experiment with that? As I am showing that using the ls command is much faster (times below)

On my Mac:

iex(3)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.095601

iex(4)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.071896

fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.034176

iex(6)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.028481

On our production instance running docker (elixir:1.9.1-alpine):

iex(zipbooks@phoenix-360h)1> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.448345

iex(zipbooks@phoenix-360h)2> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.35429

iex(zipbooks@phoenix-360h)7> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.00361

iex(zipbooks@phoenix-360h)8> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.003731

Local docker container with mounted directories(elixir:1.9.1-alpine):

iex(1)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)   

0.881692

iex(2)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.897629

fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.031227

iex(4)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.031241

On Monday, October 14, 2019 at 12:14:57 PM UTC-6, José Valim wrote:

There is no way to patch out Mix.Utils to run your own code. The proper fix would be to figure out why the path traversal is slow on Elixir (it is implemented on top of filelib:wildcard) so we can fix it rather permanently. How long does "Path.wildcard("lib/**/*.ex")" take? Can you implement your own path traversal in Elixir? Is it faster than Path.wildcard?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Mon, Oct 14, 2019 at 8:08 PM Michael St Clair <micha...@gmail.com> wrote:

I've been doing some digging around to see how to make phoenix run faster in Docker. Without this plug the app runs just as fast as native `plug(Phoenix.CodeReloader`. I think I have traced it down to this https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237. Running just that function explains most if not all of the extra load time on my endpoints. However this code `"ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") |> IO.inspect` appears to do the same thing and much faster. I'm just running into the problem of how to override that Mix.Utils file to test it here https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36. Any suggestions?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

[José Valim]

Right. Now try running File.ls in some directories and see if that is enough to reproduce the issue. Then I would implement a C script that calls ls as a NIF, see if it is fast or not. And so on.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com.

[Michael St Clair]

File.ls doesn’t appear to work with wildcards, I get this error `{:error, :enoent}`. But File.ls(“lib”) is also fast. The C script and NIF are beyond my knowledge.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K8G5py3P0eGxQ83C%2Bk0Ajwyjc4BGFDUf3dPt%3DS2rDGUQ%40mail.gmail.com.

[Michael St Clair]

File.ls doesn’t appear to work with wildcards, I get this error `{:error, :enoent}`. But File.ls(“lib”) is also fast. The C script and NIF are beyond my knowledge.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com.

[José Valim]

Trying doing File.ls a couple hundred times in the same measurement. Try to get a folder with many files. Also take a look at doing the same for File.stat.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/13ce15ce-7bad-468a-84c5-10be4044935c%40googlegroups.com.

[Michael St Clair]

lib/utils is our directory with the most files (100+). I assume this is what you were talking about doing.

This is on the local docker container

iex(6)> fn -> Enum.each(0..500, fn _ -> File.ls("lib/utils") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

1.502188

iex(7)> fn -> Enum.each(0..500, fn _ -> File.stat("lib/utils") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.268047

[José Valim]

And how long does it take to call File.ls 500 times using os:cmd?

[Michael St Clair]

I can check later this afternoon. But doesn't the improvement come with only having to call :os.cmd once with `ls lib/**/*.ex`?

[José Valim]

Well, I am not trying to solve the problem yet, I am trying to find the root cause of the problem. And to find the root cause, we need to compare our `ls` with something. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Mon, Oct 14, 2019 at 9:38 PM Michael St Clair <micha...@gmail.com> wrote:

I can check later this afternoon. But doesn't the improvement come with only having to call :os.cmd once with `ls lib/**/*.ex`?

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c011f660-ec18-457e-bf45-37f6432bf237%40googlegroups.com.

[Michael St Clair]

On my Mac :os.cmd is way slower (I can test on docker later this afternoon if needed, but I assume we would see something similar).

iex(8)> fn -> Enum.each(0..500, fn _ -> :os.cmd('ls lib/utils') |> to_string() |> String.split("\n") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

5.006744

iex(9)> fn -> Enum.each(0..500, fn _ -> File.ls("lib/utils") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

0.08175

[Michael St Clair]

I finally tested this in Docker the results were strange, File.ls is still faster but not by as much.

iex(20)> fn -> Enum.each(0..500, fn _ -> File.ls("lib/utils") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

1.05333

iex(21)> fn -> Enum.each(0..500, fn _ -> :os.cmd('ls lib/utils') |> to_string() |> String.split("\n") end) end |> :timer.tc |> elem(0) |> Kernel./(1_000_000)

2.819828

[Michael St Clair]

Running File.ls many times and seeing how fast it was gave me an idea. So I wrote this code and benchmarked it.

 

defmodule Benchmark do

  def run(function) do

    function |> :timer.tc() |> elem(0) |> Kernel./(1_000_000)

  end

end

defmodule MyFile do

  def list(directory, filter) do

    Enum.reduce([directory], [], fn item, acc ->

      if String.ends_with?(item, filter) do

        [item] ++ acc

      else

        item

        |> File.ls()

        |> case do

          {:ok, files} -> Enum.reduce(files, [], &(item |> Path.join(&1) |> list(filter) |> Kernel.++(&2))) ++ acc

          _ -> acc

        end

      end

    end)

  end

end

Docker Results

iex(45)> Benchmark.run(fn -> Path.wildcard("lib/**/*.ex") end)

0.877545

iex(46)> Benchmark.run(fn -> Enum.each(0..10, fn _ -> Path.wildcard("lib/**/*.ex") end) end)

9.808442

iex(47)> Benchmark.run(fn -> MyFile.list("lib", ".ex") end)

0.380802

iex(48)> Benchmark.run(fn -> Enum.each(0..10, fn _ -> MyFile.list("lib", ".ex") end) end)

4.368466

Mac Results

iex(29)> Benchmark.run(fn -> Path.wildcard("lib/**/*.ex") end)

0.053149

iex(30)> Benchmark.run(fn -> Enum.each(0..10, fn _ -> Path.wildcard("lib/**/*.ex") end) end)

0.641741

iex(31)> Benchmark.run(fn -> MyFile.list("lib", ".ex") end)

0.019919

iex(32)> Benchmark.run(fn -> Enum.each(0..10, fn _ -> MyFile.list("lib", ".ex") end) end)

0.248769