Protocol Consolidation bug in Umbrella apps

[Ben Wilson]

Hey folks, protocol consolidation does not seem to work properly in umbrella apps.

See this in nicer formatting and with code here: https://github.com/benwilson512/protocol-issue

If you run a function that uses a protocol heavily, it will be substantially slower when run from the umbrella root vs within the sub app directory, even when both are in production.

## Test Function

```elixir

def test do

  0..1_000_000

  |> Enum.each(fn(i) ->

    %{x: [1,2,3]} |> Poison.encode!

  end)

end

```

## From the umbrella dir

```elixir

$ MIX_ENV=prod iex -S mix

Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.0.4) - press Ctrl+C to exit (type h() ENTER for help)

iex(1)> SubApp.benchmark

{11782348, :ok}

```

## From the sub project dir

```elixir

$ cd apps/sub_app/

ben:sub_app ben$ MIX_ENV=prod iex -S mix

Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.0.4) - press Ctrl+C to exit (type h() ENTER for help)

iex(1)> SubApp.benchmark

{1545892, :ok}

```

That's 7.6x slower from the umbrella dir.

Am I missing something?

Thanks!

- Ben

[José Valim]

It is a known bug and it has been fixed in master. Unfortunately it required bigger changes and we cannot backport it to v1.0. Can you verify it works if you call mix compile.protocols explicitly?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e1e47873-f686-40a4-ad0f-cf16c49af373%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Wilson]

Hm yes that does work in my example app here. Weirdly enough I have a situation with a production app where Enumerable |> Protocol.consolidated? returns true, but Poison.Encoder returns false. When I run mix compile.protocols during compilation I see Poison.Encoder listed as compiled, and its beamfile exists in _build/prod/consolidated. Nonetheless when logging http requests in the app or when sending  other_node |> :rpc.call(Protocol, :consolidated?, [Poison.Encoder]) it fails to be consolidated.

Is this possibly related?

[Ben Wilson]

After some additional research I've been able to isolate and reproduce the problem in my example app. See: https://github.com/benwilson512/protocol-issue#issue-with-tasks

Basically, when running a task in one of the sub apps from the root directory there's a seemingly random set of protocols that fail to show up as consolidated.

$ elixir -pa  _build/dev/consolidated -S mix speed_test.start

Elixir.Poison.Decoder |> Protocol.consolidated? #=> false

Elixir.Poison.Encoder |> Protocol.consolidated? #=> false

Elixir.Access |> Protocol.consolidated? #=> false

Elixir.Collectable |> Protocol.consolidated? #=> true

Elixir.Enumerable |> Protocol.consolidated? #=> true

Elixir.Inspect |> Protocol.consolidated? #=> true

Elixir.List.Chars |> Protocol.consolidated? #=> false

Elixir.Range.Iterator |> Protocol.consolidated? #=> true

Elixir.String.Chars |> Protocol.consolidated? #=> true

[Ben Wilson]

Apologies for the continued emailing. As I've continued to look into this later issue it appears to be a problem with explicitly including the consolidated protocols and then running a task in v1.0.4 whether you're in an umbrella app or not. I copied the task created above into a different v1.0.4 app and while it printed true for them all without explicitly including them, adding -pa _build/prod/consolidated introduces the inconsistent output again.

I should add that removing build_embedded does not help in either the case of the umbrella or regular app. I've downgraded to v1.0.2 (the last version we were using) for the time being.

[José Valim]

Ben, if it is not too late, please try this:

1. Set build_embedded to true for all projects (including the umbrella parent)

2. If running the umbrella project, do as before: call compile.protocols and pass the -pa flag

3. If running the children, no need for explicit compilation nor the -ps flag 

That's all on 1.0.4.

Please let me know how it goes!

--

[Ben Wilson]

Thanks for your continued help Jose.

Unfortunately those steps are precisely what I'm doing, the only caveat is that I'm running a task that is defined within the sub project from the umbrella root. You can try it for yourself by cloning that repository and typing:

Root mix.exs: https://github.com/benwilson512/protocol-issue/blob/master/mix.exs

Sub app mix.exs: https://github.com/benwilson512/protocol-issue/blob/master/apps/sub_app/mix.exs

$ MIX_ENV=prod mix do compile.protocols

Consolidated Poison.Decoder

Consolidated Poison.Encoder

Consolidated Access

Consolidated Collectable

Consolidated Enumerable

Consolidated Inspect

Consolidated List.Chars

Consolidated Range.Iterator

Consolidated String.Chars

Consolidated protocols written to _build/prod/consolidated

$ MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix speed_test.start

Elixir.Poison.Decoder |> Protocol.consolidated? #=> false

Elixir.Poison.Encoder |> Protocol.consolidated? #=> false

Elixir.Access |> Protocol.consolidated? #=> false

Elixir.Collectable |> Protocol.consolidated? #=> true

Elixir.Enumerable |> Protocol.consolidated? #=> true

Elixir.Inspect |> Protocol.consolidated? #=> true

Elixir.List.Chars |> Protocol.consolidated? #=> false

Elixir.Range.Iterator |> Protocol.consolidated? #=> true

Elixir.String.Chars |> Protocol.consolidated? #=> true

If I don't run a task at all, but instead just do -S mix then everything is fine, except for the part where I'm no longer running my script. Running a task from a sub project in an umbrella app ought not to cause consolidation issues.

[Ben Wilson]

I should add that I just tried this most recent thing on elixir master and it had the same result. Either I'm doing something particularly wrong or this problem is still around.

[José Valim]

It has worked on master and on v1.0.4 if I manually compile.protocols in the umbrella.

Your mix task will never work though because you never start the app (nor load its paths), so we never get to load the proper implementation. Running Mix.Task.run "app.start" before hand should solve it!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Ben Wilson]

Ah. That did it. Thanks so much for your help, apologies on the confusion on my part.