Elixir Heroku buildpack leaking memory

402 views
Skip to first unread message

Victor Nascimento

unread,
May 5, 2015, 3:29:15 PM5/5/15
to elixir-l...@googlegroups.com
Hey people, wassup?

I am deploying @ Heroku using elixir buildpack and am experiencing a huge memory leak that does not happen locally. I am not fluent with Heroku tools so I'd appreciate any help.

Right after I deploy to Heroku I gather this from the logs (without performing any actions on the server):

2015-05-05T18:14:13.093262+00:00 app[web.1]: 18:14:13.093 [info]  Starting Cowboy on port 52282
2015-05-05T18:14:13.570734+00:00 heroku[web.1]: State changed from starting to up
2015-05-05T18:14:38.165998+00:00 heroku[web.1]: Process running mem=805M(157.4%)
2015-05-05T18:14:38.166200+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-05-05T18:15:00.356046+00:00 heroku[web.1]: Process running mem=1052M(205.5%)
2015-05-05T18:15:00.356046+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-05-05T18:15:22.368075+00:00 heroku[web.1]: Process running mem=1536M(300.2%)
2015-05-05T18:15:22.368209+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-05-05T18:15:44.389063+00:00 heroku[web.1]: Process running mem=1973M(385.5%)
2015-05-05T18:15:44.389287+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-05-05T18:16:06.473762+00:00 heroku[web.1]: Process running mem=2375M(464.0%)
2015-05-05T18:16:06.473762+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-05-05T18:16:28.557342+00:00 heroku[web.1]: Process running mem=2625M(512.8%)
2015-05-05T18:16:28.557342+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2015-05-05T18:16:28.557943+00:00 heroku[web.1]: Stopping process with SIGKILL
2015-05-05T18:16:30.090269+00:00 heroku[web.1]: State changed from up to crashed
2015-05-05T18:16:30.037819+00:00 heroku[web.1]: Process exited with status 137

Locally the process is steady at around 35MB of memory consumption. The app is a simple REST API using Plug and Ecto mainly. It also uses Poison, Comeonin and Joken for JWTs.

My Procfile follows the documentation:

web: yes | elixir -pa _build/prod/consolidated -S mix run --no-halt

And a standard buildpack config:

erlang_version=17.5
elixir_version=1.0.4
always_rebuild=false
config_vars_to_export=(DATABASE_URL)

Has anyone else experienced this? I can share the project as it has nothing special in it.

Thanks in advance!

José Valim

unread,
May 5, 2015, 3:40:28 PM5/5/15
to elixir-l...@googlegroups.com
Shouldn't you set MIX_ENV=prod in your Procfile or is that done automatically?

Also, do you see the issue if you run the app on your machine but in production environment?



José Valim
Skype: jv.ptec
Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/180ad6f6-d7f5-44e2-9b56-14b59eda8ad4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Victor Nascimento

unread,
May 5, 2015, 3:57:56 PM5/5/15
to elixir-l...@googlegroups.com
I set MIX_ENV using heroku variables through the cli client. I can see the value is set through the dashboard.

I have run with MIX_ENV=prod locally and it keeps at 35MB memory without a problem.

As I said, it is a simple app which does nothing after it is started. Here is my base module:

defmodule Baseline.App do
    use Application
    require Logger

    def start(_type, args) do
    import Supervisor.Spec

    tree = [
            worker(Baseline.Repo, [])
        ]
    sup_opts = [name: Baseline.Sup, strategy: :one_for_one]
        http_opts  = [port: 4004, compress: true]
       
        if port = System.get_env("PORT") do
            http_opts = Keyword.put(http_opts, :port, String.to_integer(port))
        end

    Logger.info "Starting Cowboy on port #{http_opts[:port]}"
        Plug.Adapters.Cowboy.http Baseline.ApiRouter, [], http_opts
        Supervisor.start_link(tree, sup_opts)
  end
end

defmodule Baseline.Repo do
    use Ecto.Repo, otp_app: :baseline_server
end


--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4LSV5FjfhZiHD22HPh_hGrMerARqCv%3DQ38Dw2F4oeQBmg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Victor Oliveira Nascimento
Concrete Solutions
Senior Analyst/Developer
victor.n...@concretesolutions.com.br
http://www.concretesolutions.com.br
Mobile: +55 11 9 8923-8061



Rio de Janeiro                                     São Paulo
Rua São José 90, sala 2121                 Rua Flórida 1670, Sala 22
Cep: 20010-020                                   Cep: 04565-001
Centro - Rio de Janeiro - Brasil             Brooklin - São Paulo – Brasil
Phone : +55 21 2240-2030                   Phone: +55 11 4119 0449

José Valim

unread,
May 5, 2015, 4:43:46 PM5/5/15
to elixir-l...@googlegroups.com
This is very strange. Just to clarify, Hex runs on Heroku with Plug and Ecto, using the same build pack, and we haven't seen memory leaks.

Some suggestions:

* Read Erlang in Anger if you haven't yet (it is short): http://www.erlang-in-anger.com

* Running on Heroku is going to limit what you can debug your system. One idea is to create a process that logs data from recon (the book talks about it) every minute so at least you can see which part of your system is leaking. For example, this will detect all processes that take more than 1MB in memory and log them:

require Logger
for pid <- Process.list,
    Process.info(pid, :memory) > {:memory, 1000000} do
  Logger.warn inspect({pid, Process.info(pid)})
end

I hope this gives some direction to identify the bug. Since memory is growing quick, I would bet on some loop that is never resolving and increasing the memory until it crashes. 


José Valim
Skype: jv.ptec
Founder and Lead Developer

Victor Nascimento

unread,
May 5, 2015, 4:53:29 PM5/5/15
to elixir-l...@googlegroups.com
Yeah I've seen hex_web repo :D Great source of inspiration!

I have also read that book :D Mandatory read to anyone in the beam world!

I will try to dig it further. I will debug more thoroughly. Perhaps the nifs in comeonin lib?

Thanks for your attention!

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4Jk5AqKg3D4Z97mTL32hRYa87M8O5J%3D9XPefFWAGZdL6Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Victor Nascimento

unread,
May 6, 2015, 11:12:12 AM5/6/15
to elixir-l...@googlegroups.com
Okay, tried a much simpler app:

It has the same application module, an Ecto Repo and one Plug Router with one route that replies with "Ok!".

Still the same effect. Memory goes off the roof. I am beginning to think it has nothing to do with the app. Just in case here is the link to the app if anyone wants to take a look (it has nothing... really).

https://github.com/cs-victor-nascimento/test-elixir

Cheers,

On Tuesday, May 5, 2015 at 5:53:29 PM UTC-3, Victor Nascimento wrote:
Yeah I've seen hex_web repo :D Great source of inspiration!

I have also read that book :D Mandatory read to anyone in the beam world!

I will try to dig it further. I will debug more thoroughly. Perhaps the nifs in comeonin lib?

Thanks for your attention!
On Tue, May 5, 2015 at 5:43 PM, José Valim <jose....@plataformatec.com.br> wrote:
This is very strange. Just to clarify, Hex runs on Heroku with Plug and Ecto, using the same build pack, and we haven't seen memory leaks.

Some suggestions:

* Read Erlang in Anger if you haven't yet (it is short): http://www.erlang-in-anger.com

* Running on Heroku is going to limit what you can debug your system. One idea is to create a process that logs data from recon (the book talks about it) every minute so at least you can see which part of your system is leaking. For example, this will detect all processes that take more than 1MB in memory and log them:

require Logger
for pid <- Process.list,
    Process.info(pid, :memory) > {:memory, 1000000} do
  Logger.warn inspect({pid, Process.info(pid)})
end

I hope this gives some direction to identify the bug. Since memory is growing quick, I would bet on some loop that is never resolving and increasing the memory until it crashes. 


José Valim
Skype: jv.ptec
Founder and Lead Developer

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-talk+unsubscribe@googlegroups.com.



--
Victor Oliveira Nascimento
Concrete Solutions
Senior Analyst/Developer
victor.nascimento@concretesolutions.com.br

http://www.concretesolutions.com.br
Mobile: +55 11 9 8923-8061



Rio de Janeiro                                     São Paulo
Rua São José 90, sala 2121                 Rua Flórida 1670, Sala 22
Cep: 20010-020                                   Cep: 04565-001
Centro - Rio de Janeiro - Brasil             Brooklin - São Paulo – Brasil
Phone : +55 21 2240-2030                   Phone: +55 11 4119 0449

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-talk+unsubscribe@googlegroups.com.



--
Victor Oliveira Nascimento
Concrete Solutions
Senior Analyst/Developer
victor.nascimento@concretesolutions.com.br

José Valim

unread,
May 6, 2015, 11:26:10 AM5/6/15
to elixir-l...@googlegroups.com
Ok. So the issue is related to "yes |".

If I run it locally using "yes |", I can see the memory leak, if I don't use "yes |", it works just fine. I will try to understand why it is happening. I suspect that the application starts to log, then yes is written to stdin, and then it becomes buffered eternally.

Also, you don't need elixir -pa ... If you are using elixir 1.0.4.



José Valim
Skype: jv.ptec
Founder and Lead Developer

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.



--
Victor Oliveira Nascimento
Concrete Solutions
Senior Analyst/Developer
victor.n...@concretesolutions.com.br

http://www.concretesolutions.com.br
Mobile: +55 11 9 8923-8061



Rio de Janeiro                                     São Paulo
Rua São José 90, sala 2121                 Rua Flórida 1670, Sala 22
Cep: 20010-020                                   Cep: 04565-001
Centro - Rio de Janeiro - Brasil             Brooklin - São Paulo – Brasil
Phone : +55 21 2240-2030                   Phone: +55 11 4119 0449

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.



--
Victor Oliveira Nascimento
Concrete Solutions
Senior Analyst/Developer
victor.n...@concretesolutions.com.br

http://www.concretesolutions.com.br
Mobile: +55 11 9 8923-8061



Rio de Janeiro                                     São Paulo
Rua São José 90, sala 2121                 Rua Flórida 1670, Sala 22
Cep: 20010-020                                   Cep: 04565-001
Centro - Rio de Janeiro - Brasil             Brooklin - São Paulo – Brasil
Phone : +55 21 2240-2030                   Phone: +55 11 4119 0449

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/58f07124-ef31-492b-a27c-cc8f18876a84%40googlegroups.com.

Ross Jones

unread,
May 6, 2015, 11:56:20 AM5/6/15
to elixir-l...@googlegroups.com


José Valim wrote:
> Ok. So the issue is related to "yes |".
>
> If I run it locally using "yes |", I can see the memory leak, if I don't
> use "yes |", it works just fine. I will try to understand why it is
> happening. I suspect that the application starts to log, then yes is
> written to stdin, and then it becomes buffered eternally.

FWIW I can replicate the leak with just 'yes | elixir --no-halt'.
OSX/17.5/1.0.4.

I think piping in an infinite stream of Ys, there probably isn't much
Elixir can do with it is there, is Elixir waiting for an EOF?

On the same machine doing 'yes | python' has exactly the same behaviour,
except it uses up all my memory faster (I knew it'd be fast at
*something*!!).


Ross.

Victor Nascimento

unread,
May 6, 2015, 12:48:51 PM5/6/15
to elixir-l...@googlegroups.com
Wow... nice catch. I took the Procfile command from: http://learnelixir.com/blog/2014/10/15/deploy-phonenix-application-to-heroku-server/ and did not bother trying other things there.

Looking @ Hex web Procfile makes sense now.

Thanks all for your help! I will send a note to that blog author's just to help.



--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/XXtNWHhDWrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/554A39A0.1090400%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

Ross Jones

unread,
May 6, 2015, 12:58:55 PM5/6/15
to elixir-l...@googlegroups.com

Victor Nascimento wrote:
> Wow... nice catch. I took the Procfile command from:
> http://learnelixir.com/blog/2014/10/15/deploy-phonenix-application-to-heroku-server/
> and did not bother trying other things there.

I think in that particular case it was calling 'yes | mix
compile.protocols && some other stuff' which is fine under the general
rule of "Only use 'yes' if the process it is being piped to will exit
soon".

Ross.
Reply all
Reply to author
Forward
0 new messages