Pheonix app build starts / runs fine on OS X, crashes on Linux

[Brad O'Hearne]

I have a Phoenix app that compiles and builds a production web app cleanly, and runs cleanly on OS X. On Linux, however, while it compiles and builds a production web app cleanly, it crashes. Here is the error I receive when trying to start a release build from the command line: 

$ sudo rel/brad/bin/brad console

Using /home/centos/Apps/Brad/Web/brad/rel/brad/releases/0.0.2/brad.sh

Exec: /home/centos/Apps/Brad/Web/brad/rel/brad/erts-7.2/bin/erlexec -boot /home/centos/Apps/Brad/Web/brad/rel/brad/releases/0.0.2/brad -boot_var ERTS_LIB_DIR /home/centos/Apps/Brad/Web/brad/rel/brad/erts-7.2/../lib -env ERL_LIBS /home/centos/Apps/Brad/Web/brad/rel/brad/lib -config /home/centos/Apps/Brad/Web/brad/rel/brad/running-config/sys.config -pa /home/centos/Apps/Brad/Web/brad/rel/brad/lib/brad-0.0.2/consolidated -args_file /home/centos/Apps/Brad/Web/brad/rel/brad/running-config/vm.args -user Elixir.IEx.CLI -extra --no-halt +iex -- console

Root: /home/centos/Apps/Brad/Web/brad/rel/brad

/home/centos/Apps/Brad/Web/brad/rel/brad

Erlang/OTP 18 [erts-7.2] [source-e6dd627] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

21:27:46.362 [info] Running Brad.Endpoint with Cowboy using http on port 8080

21:27:46.412 [info] Application brad exited: Brad.start(:normal, []) returned an error: shutdown: failed to start child: Brad.Repo

    ** (EXIT) an exception was raised:

        ** (MatchError) no match of right hand side value: {Ecto.Pools.Poolboy, Brad.Repo.Pool, 5000}

            (ecto) lib/ecto/repo/supervisor.ex:106: Ecto.Repo.Supervisor.init/1

            (stdlib) supervisor.erl:272: :supervisor.init/1

            (stdlib) gen_server.erl:328: :gen_server.init_it/6

            (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

{"Kernel pid terminated",application_controller,"{application_start_failure,brad,{{shutdown,{failed_to_start_child,'Elixir.Brad.Repo',{{badmatch,{'Elixir.Ecto.Pools.Poolboy','Elixir.Brad.Repo.Pool',5000}},[{'Elixir.Ecto.Repo.Supervisor',init,1,[{file,\"lib/ecto/repo/supervisor.ex\"},{line,106}]},{supervisor,init,1,[{file,\"supervisor.erl\"},{line,272}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}},{'Elixir.Brad',start,[normal,[]]}}}"}

Crash dump is being written to: erl_crash.dump...done

Kernel pid terminated (application_controller) ({application_start_failure,brad,{{shutdown,{failed_to_start_child,'Elixir.Brad.Repo',{{badmatch,{'Elixir.Ecto.Pools.Poolboy','Elixir.Brad.Repo.Pool',5000

Also note that when run with 'mix phoenix.server' on Linux, it starts and runs fine. So the problem seems to be limited to running a release build. Any ideas? I'm not really sure where to start on this one...

Thanks, 

Brad

[Brad O'Hearne]

Trying to debug this...I have a question about this line in the error output: 

 ** (MatchError) no match of right hand side value: {Ecto.Pools.Poolboy, Clu.Repo.Pool, 5000}

Where is that 5000 coming from? My Ecto.Repo config has a pool_size set to 20...does this number represent something else? 

Brad

[Brad O'Hearne]

Does anyone know what is different about starting a Phoenix app in a PROD environment vs. a DEV environment that could cause the above error? Why would this work just fine in DEV but not in PROD? 

Brad

[pj.beta...@gmail.com]

Hi,

I don't know enough to really answer your question but I've had two different issues while 1) trying to compile and 2) while serving in production in Linux (Ubuntu) apps that were fine in my Mac and I found that for both cases the issue was RAM. Even if I was, theoretically, using just 15% of RAM and swap the truth is that when I upgraded my server those issues were gone...So, how much RAM do you have in Linux?

[Santiago Fernández]

5000 is the value of timeout (in ms) when calling poolboy... Maybe trying to connect to DB, I guess.

Is your database configured ok in production?

$ MIX_ENV=prod mix ecto.create
$ MIX_ENV=prod mix ecto.migrate

--

Santiago

--
You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at https://groups.google.com/group/phoenix-talk.
To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/4b4c486d-a5b1-4475-863c-59bea0a3d344%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Bradley O'Hearne]

PJ, 

Thanks for the reply. Based on the error message (matching issue) I would not have thought of memory. I’ll look into the memory issue. I do have less memory on Linux than on OS X, but the thing which stumps me about it is that 

mix phoenix.server 

on my Linux machine works just fine, but starting the release (PROD) build on Linux crashes. So there’s something obviously different about the startup in DEV vs PROD. I confess, debugging this particular issue just looks like a dense fog to me. I’m not exactly sure what I should be looking for, or what the key points of difference between DEV and PROD would be save their *.exs config files, and those are virtually identical. 

I’ll keep poking around, see what I can find…if anyone has any advice for how to debug this, it would be welcome!

Thanks,

Brad

[pj.beta...@gmail.com]

Brad,

Look, I know what you mean, and also that the error message doesn't seem to be memory related. Nevertheless, you have a process that is closed. I found that in my case processes either were killed or didn't work as expected because of RAM. And with "mix phoenix.server" I had no issue either. Only when I started it with ENV= PROD.

So I agree that there's something different enough to ask for more memory in production. 

But,m again, your problem could be completely different.

[Stephen M]

I wonder if you can post the differences (if any) that exist for the Repo in config.exs. just the config :my_app, MyApp.Repo, .... part. I'm not sure if this will lead to anything but maybe somebody will spot something.

[Brad O'Hearne]

Santiago -- Thanks for the response. To your question, yes the database is configured OK in production, but further, it is MongoDB, so that configuration (from a schema standpoint) isn't needed anyway, as it is created on the fly. But additionally, I can run this: 

MIX_ENV=prod mix phoenix.server

And everything runs just fine. But when I run this, 

rel/brad/bin/brad console

It crashes. So production environment *will* run and serve requests just fine on this box. It just won't start a release. Hmmm...

[Brad O'Hearne]

On Monday, February 29, 2016 at 8:54:01 AM UTC-7, pj.beta...@gmail.com wrote:

And with "mix phoenix.server" I had no issue either. Only when I started it with ENV= PROD.

PJ -- thanks for hanging with me on this. I thought we had a winner with the memory angle, and so I resized the server from the 1GB RAM it had (while that of course is tiny, I have phoenix running on other servers with that RAM just fine), to 16GB RAM, and I still receive the same error when starting the release. 

I ran another test with the PROD environment, and that is specifying it explicitly with mix phoenix.server. The results are interesting. Starting Phoenix with this command: 

MIX_ENV=prod mix phoenix.server

starts cleanly and runs just fine, serving web pages and services, and talking to the database via Ecto without a problem. But when I start the release build of Phoenix with this command: 

rel/brad/bin/brad

This is what crashes with the aforementioned error. So it appears the question is what is different about the startup sequences and configuration with mix phoenix.server in PROD and a release, that could cause a problem for Ecto specifically, or generally, the startup of a child process. Thoughts? 

Thanks everyone for your help thus far. 

[Brad O'Hearne]

On Monday, February 29, 2016 at 11:07:40 AM UTC-7, Stephen M wrote:

I wonder if you can post the differences (if any) that exist for the Repo in config.exs. just the config :my_app, MyApp.Repo, .... part. I'm not sure if this will lead to anything but maybe somebody will spot something.

Stephen -- thanks for the response. At this point, the issue has been tracked down to starting the app using MIX_ENV= prod mix phoenix.server or rel/brad/bin/brad, but both of those are PROD environment, so I don't think there are any differences since both should use prod.exs. 

[Andrei Mihu]

Can you share your mix.exs in a gist? If it's just the packaged exrm release that fails your problem may be the applications list. exrm will only include in the package dependencies you list there, and that causes all sorts of strange startup errors.

[Brad O'Hearne]

On Monday, February 29, 2016 at 1:19:28 PM UTC-7, Andrei Mihu wrote:

Can you share your mix.exs in a gist? If it's just the packaged exrm release that fails your problem may be the applications list. exrm will only include in the package dependencies you list there, and that causes all sorts of strange startup errors.

Sure thing....here you go: 

https://gist.github.com/brado/9d0f54792c9422101502 

[Santiago Fernandez]

Brad,

Is your "rel" directory under version control? if yes: did you try to delete rel directory and run mix compile and mix release again?

Are you using a custom relx.config file?

Is there any reason to run the release with root privileges (with `sudo`)?

--

Santiago

--

You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at https://groups.google.com/group/phoenix-talk.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/4fe57df4-1f21-48c7-a57b-dbf7dbc01656%40googlegroups.com.

[Jeff Weiss]

Brad,

I believe I may know what the issue is: Poolboy is not included in your release generated by exrm. Do you see poolboy listed in rel/brad/lib ?

You'll need dependencies for the release as indicated by this exrm documentation: https://hexdocs.pm/exrm/extra-common-issues.html

--

You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at https://groups.google.com/group/phoenix-talk.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/4fe57df4-1f21-48c7-a57b-dbf7dbc01656%40googlegroups.com.

[Brad O'Hearne]

On Monday, February 29, 2016 at 2:04:28 PM UTC-7, Jeff Weiss wrote:

I believe I may know what the issue is: Poolboy is not included in your release generated by exrm. Do you see poolboy listed in rel/brad/lib ?

You'll need dependencies for the release as indicated by this exrm documentation: https://hexdocs.pm/exrm/extra-common-issues.html

*** PROBLEM SOLVED ***

I cannot claim to know every nuance of why this worked, but the problem appears to be solved. Thank you everyone for your responses and discussion which eventually lead to the discovery of the problem. Thanks especially to Jeff Weiss and your comment above, because while the exact problem was not the poolboy dependency, as it was indeed listed in rel/brad/lib, it drew my attention to inspecting my release dependencies. I compared the dependencies found in rel/brad/lib on OS X to those found on Linux, and I found discrepancies, specifically with the versions of ecto and phoenix_ecto, which on Linux were newer. A footnote on that -- a few months ago I had deployed to Linux with a newer version of ecto and phoenix_ecto (current versions), but discovered that my actual database driver mongo_ecto was not yet compatible with the current versions of these. So I had to downgrade ecto, phoenix_ecto, and phoenix itself. This must be why cleaning dependencies and running with 'mix phoenix.server' works, because it is getting the right dependencies. But with building the release (and I had a script I had created for doing this, which I thought was cleaning everything), while I was executing 'mix release.clean', that doesn't clean dependencies, you have to use the '--implode' option to get rid of it.

MORAL OF THE STORY: run 'mix release.clean --implode' before building a new release. 

Notes: 

• I am not committing the 'rel' directory to source code control, because having to build for multiple platforms, it was a better scenario in my case to tag the source and build on the target platform.
• This does raise an issue in exrm...it is interesting that it doesn't detect / warn or fail the release build if it has dependency versions in its rel/<app>/lib directory which violate the dependency versions specified in mix.exs. More generally, maybe a better question is why the exrm lib directory appears to be shared by release versions, rather than have a lib directory per release version. I probably don't understand exactly how that is being used, but it would seem that dependency versions would commonly vary between app release versions. 

At any rate, I really appreciate all those who contributed to helping me find a solution. 

Thanks, 

Brad