Docker cli succeeds, but nomad service fails

[je...@boltvoice.com]

I'm trying to run a service with nomad, but the service continuously fails to start with

Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"

However, I'm able to take the same image and run it without issue using the docker cli.

I have had this issue with nomad v0.4.1 and v0.5.0 I'm not sure if it's something that I'm doing wrong or something that I should file as a bug. I've attached the output of "docker inspect" for the container that worked "good.json" and the one that didn't "bad.json" as well as the nomad file for the service (with redacted creds) as "call-engine.nomad" Any help in determining what my issue is or if this is a bug would be much appreciated.

[Diptanu Choudhury]

I don't see any major differences in the two json files, except that the one which works doesn't have any ports exposed and Nomad is using syslog for sending log messages. What does the stderr/stdout log files show when the container doesn't work?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/a9f33e63-b29b-413e-b856-596f6fa61fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[je...@boltvoice.com]

Diptanu, thanks for responding. From the syslog

Nov 23 08:39:36 hutch dockerd[1019]: time="2016-11-23T08:39:36.418819889-07:00" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."

Nov 23 08:39:36 hutch kernel: [192207.361095] aufs au_opts_verify:1597:dockerd[1076]: dirperm1 breaks the protection by the permission bits on the lower branch

Nov 23 08:39:36 hutch kernel: [192207.578614] aufs au_opts_verify:1597:dockerd[1076]: dirperm1 breaks the protection by the permission bits on the lower branch

Nov 23 08:39:36 hutch kernel: [192207.690451] aufs au_opts_verify:1597:dockerd[1037]: dirperm1 breaks the protection by the permission bits on the lower branch

Nov 23 08:39:38 hutch dockerd[1019]: time="2016-11-23T08:39:38.383004708-07:00" level=error msg="Handler for POST /containers/5537cd61600bd65a5ca77e16ff4a35113b30dfafa7a9f94cf8620ace9fc17832/stop returned error: Container 5537cd61600bd65a5ca77e16ff4a35113b30dfafa7a9f94cf8620ace9fc17832 is already stopped"

On Wednesday, November 23, 2016 at 6:31:45 AM UTC-7, Diptanu Choudhury wrote:

I don't see any major differences in the two json files, except that the one which works doesn't have any ports exposed and Nomad is using syslog for sending log messages. What does the stderr/stdout log files show when the container doesn't work?

On Tue, Nov 22, 2016 at 6:01 AM, <je...@boltvoice.com> wrote:

I'm trying to run a service with nomad, but the service continuously fails to start with

Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"

However, I'm able to take the same image and run it without issue using the docker cli.

I have had this issue with nomad v0.4.1 and v0.5.0 I'm not sure if it's something that I'm doing wrong or something that I should file as a bug. I've attached the output of "docker inspect" for the container that worked "good.json" and the one that didn't "bad.json" as well as the nomad file for the service (with redacted creds) as "call-engine.nomad" Any help in determining what my issue is or if this is a bug would be much appreciated.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/a9f33e63-b29b-413e-b856-596f6fa61fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Diptanu Choudhury]

I wanted to see the stdout and stderr of the application running in the container. You should be able to find them under alloc/logs. Since the container is exiting with exit code 1, that means the binary is getting started but failing for some other reason.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/2340c934-0ea6-46fd-bff6-0436a6aa9d16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[je...@boltvoice.com]

Sorry for the confusion and showing me how to find the logs. I will know apologize for wasting your time. The logs read

DEBUG [2016-11-23T16:45:20.672Z] main (main.go:38) ▶ Logging setup

DEBUG [2016-11-23T16:45:20.674Z] main (main.go:42) ▶ Resolving RPC interface

DEBUG [2016-11-23T16:45:20.674Z] main (main.go:48) ▶ NOMAD_HOST_PORT_rpc 34455 found

DEBUG [2016-11-23T16:45:20.674Z] main (main.go:52) ▶ Using value 34455

CRITICAL [2016-11-23T16:45:20.674Z] main (main.go:55) ▶ Failed to resolve address 34455: missing port in address 34455

It appears that I miss read the documentation on environments and thought that  NOMAD_HOST_PORT_<label> was the host and port that was assigned rather than the port on the host. I just need to change my code to use  NOMAD_ADDR_<label> instead. Sorry for the bother and thank you for the insight.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/2340c934-0ea6-46fd-bff6-0436a6aa9d16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.