Failed to find docker auth for repo (0.9-rc2)

[Chris Stevens]

I am trying 0.9-rc2 on a test cluster that has been running 0.8.7 without issues for some time.

The client nodes use the docker driver and AWS ECR cred helper successfully on 0.8.7. All allocs failed on 0.9-rc2.

Recent Events:

Time                  Type                   Description

2019-04-04T16:33:52Z  Killing                Sent interrupt

2019-04-04T16:33:52Z  Not Restarting         Error was unrecoverable

2019-04-04T16:33:52Z  Driver Failure         Failed to find docker auth for repo "https://XXXX.dkr.ecr.us-west-2.amazonaws.com/foo/bar": Failed to parse named repo "https://XXXX.dkr.ecr.us-west-2.amazonaws.com/foo/bar": invalid reference format

2019-04-04T16:33:52Z  Downloading Artifacts  Client is downloading artifacts

2019-04-04T16:33:51Z  Task Setup             Building Task Directory

2019-04-04T16:33:51Z  Received               Task received by client

I have not updated the nomad client stanza to use the new plugin syntax yet (still using the client.options settings).

and we are using docker-credential-ecr-login cred helper from AWS.

The docker config.json on the client instance looks like this:

{

    "credHelpers": {

        "XXXX.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"

        },

        "credsStore": ""

}

I am still digging into the issue, but wanted to see if anybody else has encountered something similar.

[net...@hashicorp.com]

Hey Chris, are you able to open an issue on github so we can track it there? Does the "image" value for the task contain "https://"? I know we updated some docker dependencies so it may not be happy with that.

https://github.com/hashicorp/nomad/issues/new

Thanks,

Nick

[Chris Stevens]

Hi Nick.

Yes, the image value in the job file for the task is of the form "https://XXXX.dkr.ecr.us-west-2.amazonaws.com/foo/bar".

I can try deploying a job to the 0.8.7 hosts without the leading "https://" and see how that works and then move it to the 0.9-rc2 host via a drain operation.

[Nick Ethier]

If you could that would be great info to help narrow it down. Regardless, if it worked in 0.8 it should work in 0.9 so we’ll get it fixed before we cut the final release. Thanks!

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/b3d03834-9ac0-4852-a1a6-5393b947fc0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Stevens]

Confirmed. Removing the "https://" from the task image stanza in the job file does appear to allow the task to download the image and startup correctly. Very small sample size right now, but I'll try a few others as I have time.

Does this seem like intended behavior (more restrictive parsing) or a regression of some kind?

[Nick Ethier]

If it worked in 0.8 it should work in 0.9. If I recall I think docker always expects you to drop the scheme. For example you can’t docker pull https://example.com/foo/bar but we’ll maintain backwards compatibility regardless.

On Thu, Apr 4, 2019 at 15:31 Chris Stevens <chris....@traxo.com> wrote:

Confirmed. Removing the "https://" from the task image stanza in the job file does appear to allow the task to download the image and startup correctly. Very small sample size right now, but I'll try a few others as I have time.

Does this seem like intended behavior (more restrictive parsing) or a regression of some kind?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/83382935-2097-451a-b75c-eb96f01a6a8b%40googlegroups.com.

[Chris Stevens]

Sounds good. Filed here: https://github.com/hashicorp/nomad/issues/5517

[Nick Ethier]

Chris,

I wrote a test for this and just pushed a fix for it. Thanks again for testing out the RC and filing this issue!

On Thu, Apr 4, 2019 at 3:45 PM Chris Stevens <chris....@traxo.com> wrote:

Sounds good. Filed here: https://github.com/hashicorp/nomad/issues/5517

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/b4ab9731-72f8-4a53-acc1-cbd50ac8b44a%40googlegroups.com.

[Chris Stevens]

Awesome! Always glad to help out.