[JIRA] [amazon-ecr-plugin] (JENKINS-34958) Getting "Your Authorization Token has expired" when using ECR credentials

[kristoffer@codedivision.com (JIRA)]

Kristoffer Peterhänsel created an issue

Jenkins / JENKINS-34958 Getting "Your Authorization Token has expired" when using ECR credentials Issue Type: Bug Assignee: Nicolas De Loof Components: amazon-ecr-plugin Created: 2016/May/19 2:14 PM Environment: Jenkins 2.4 Docker Build and Publish plugin 1.2.2 (+ PR #41) Priority: Blocker Reporter: Kristoffer Peterhänsel In an attempt to start moving away from our self-hosted Docker Registry. I came across this plugin to make it easier to push to Amazon ECR. And after a (fairly) quick fix of the Docker Build and Publish plugin. Time had come to make that happen. But instead I am getting the this error when it attempt to push. So something is wrong. The push refers to a repository [somerepo.dkr.ecr.eu-west-1.amazonaws.com/imagename] 1b29323a75d2: Preparing 5bf87793f977: Preparing 5ccb950f635d: Preparing 965c3fc60463: Preparing f354df03c5c3: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 9523ecdf69b1: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 6d7b4f405a28: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 099efa904cb9: Preparing 8f83f19c7186: Preparing 1621d30a7846: Preparing e989ce4ed35e: Preparing ae30a2e42fe4: Preparing 461f75075df2: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 6d7b4f405a28: Waiting 099efa904cb9: Waiting 8f83f19c7186: Waiting 1621d30a7846: Waiting e989ce4ed35e: Waiting ae30a2e42fe4: Waiting 461f75075df2: Waiting 5f70bf18a086: Waiting 9523ecdf69b1: Waiting f354df03c5c3: Image push failed f354df03c5c3: Image push failed 461f75075df2: Waiting ae30a2e42fe4: Waiting e989ce4ed35e: Waiting 1621d30a7846: Waiting 8f83f19c7186: Waiting 099efa904cb9: Waiting 6d7b4f405a28: Waiting 9523ecdf69b1: Waiting 5f70bf18a086: Waiting Error parsing HTTP response: invalid character 'Y' looking for beginning of value: "Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one." Build step 'Docker Build and Publish' marked build as failure In the panel for updating the credentials I also get the message: These credentials are valid but do not have access to the "AmazonEC2" service in the region "us-east-1". This message is not a problem if you need to access to other services or to other regions. Message: "You are not authorized to perform this operation. (UnauthorizedOperation)" But I am using the AWS Managed policy "AmazonEC2ContainerRegistryPowerUser" to grant Jenkins access. And should pretty much have full access to all the ECR calls it needs. We do use it in 'eu-west-1' though. But the warning clearly states not to worry about it if we are not in that region. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[kristoffer@codedivision.com (JIRA)]

Kristoffer Peterhänsel commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials So I can't quite manually provoke this error message using the AWS CLI tool and docker. But it does appear that the credentials issued are regional. So it seems this plugin may need a way to set the desired region. But so far I haven't found the place in the AmazonECRClient where you can actually specify region...

Add Comment

[kristoffer@codedivision.com (JIRA)]

Kristoffer Peterhänsel edited a comment on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

So I can't quite manually provoke this error message using the AWS CLI tool and docker. But it does appear that the credentials issued are regional. So it seems this plugin may need a way to set the desired region. But so far I haven't found the place in the AmazonECRClient where you can actually specify region...

Edit: And then of course I did find see it.

Add Comment

[vesa.alho@nordcloud.com (JIRA)]

Vesa Alho commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I can confirm this issue. Works in us-east-1, but not eg. in eu-west-1.

Add Comment

[luis@vvoosh.com (JIRA)]

Luis Silva commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Faced this same issue today with this plugin today. Configuring ECR in us-east-1 region ( **.dkr.ecr.us-east-1.amazonaws.com/** ) the push succeeds. But using ECR in eu-west-1 ( **.dkr.ecr.eu-west-1.amazonaws.com/** ) it fails: with an error: error parsing HTTP 403 response body: unexpected end of JSON input: ""

Build step 'Docker Build and Publish' marked build as failure

It's the exact same job and settings, only changing the registry endpoint in the job. Using the command line to issue a push with the token obtained by the jenkins job (set in ~/.docker/config.json) also fails, but using the aws cli to get the token, issuing a docker login and then push works fine regardless of the region. That leads to me to believe the problem is somewhere in the token / credentials issue code as you mentioned. Any ETA on the fix for this problem? Docker version 1.11.1, build 5604cbe/1.11.1 Amazon ECR plugin: 1.3 CloudBees Docker Build and Publish plugin: 1.2.2 Docker Commons Plugin: 1.3.1

Add Comment

[luis@vvoosh.com (JIRA)]

Luis Silva edited a comment on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Faced this same issue today with this plugin  today

. Configuring ECR in us-east-1 region ( ***.dkr.ecr.us-east-1.amazonaws.com/*** ) the push succeeds. But using ECR in eu-west-1 ( ***.dkr.ecr.eu-west-1.amazonaws.com/***  ) it fails: with an error:

error parsing HTTP 403 response body: unexpected end of JSON input: "" Build step 'Docker Build and Publish' marked build as failure It's the exact same job and settings, only changing the registry endpoint in the job. Using the command line to issue a push with the token obtained by the jenkins job (set in ~/.docker/config.json) also fails, but using the aws cli to get the token, issuing a docker login and then push works fine regardless of the region. That leads to me to believe the problem is somewhere in the token / credentials issue code as you mentioned. Any ETA on the fix for this problem? Docker version 1.11.1, build 5604cbe/1.11.1 Amazon ECR plugin: 1.3 CloudBees Docker Build and Publish plugin: 1.2.2 Docker Commons Plugin: 1.3.1

Add Comment

[tommy@lark-it.com (JIRA)]

Tommy McNeely commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I confirm that we are also having this problem with us-west-2. I even tried doing the docker login command from ecr get-login as the jenkins user (and tested with a manual docker push), expecting that it might "see" that the credentials were already there, but it appears to overwrite the credentials regardless of their age/validity. I am thinking that the region might have to be assigned to the credential? Tommy

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[me@arronwoods.com (JIRA)]

Arron Woods commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

We're using eu-west-1 and experiencing the same. http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ecr/AmazonECRClient.html It looks like the AmazonEcrClient can accept a region ( AmazonEcrClient::configureRegion() ), but not sure where this would need to set in Jenkins? Should the AwsCredentials plugin have a region parameter, or would it be set on the repository somewhere? If using the AWS url, the region could be extracted from the URL. Wouldn't help if using your own domain though.

Add Comment

[alex@evaluagent.net (JIRA)]

Alex Richards commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I'm also getting this issue when trying to use EU-WEST-1. If I change my settings to use US-EAST-1 then it works ok.

Add Comment

[alex@evaluagent.net (JIRA)]

Alex Richards commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

The only way I could get around this issue was to add a shell command to a previous step with: eval `aws ecr get-login --region=eu-west-1`

Add Comment

[alex@evaluagent.net (JIRA)]

Alex Richards edited a comment on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

The only way I could get around this issue was to add a shell command to a previous step with:

{code:java} eval `aws ecr get-login --region = eu-west-1` {code}

Add Comment

[niccolo@olivieriachille.com (JIRA)]

Niccolò Olivieri commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I tryed not to use the aws configuration on the server, so: add the AWS Credential to Jenkins in the build: Build Environment -> Use secret text(s) or file(s) -> AWS Build -> Execute Shell export AWS_DEFAULT_REGION='eu-west-1' eval `aws ecr get-login` Build -> Docker Build & Publish -> do not put the registry credentials In this way my build is ok!!!! Hope it helped, I spent two days searching a solution!!!

Add Comment

[logbon72@gmail.com (JIRA)]

Joseph Orilogbon commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I created a PR for this issue here: https://github.com/jenkinsci/amazon-ecr-plugin/pull/6 I added a new credential format that contains the region. For example, by specifying the following credentials: `ecr:us-west-2:credential-id`, the provider will set the Region of the AWS Client to `us-west-2`, when requesting for Authorisation token. The current credential format is supported and uses the default region (us-east-1) specified in [AmazonECRClient](https://github.com/aws/aws-sdk-java/blob/5ec8907a0342bff23e15fc290f715c66cb9bad03/aws-java-sdk-ecr/src/main/java/com/amazonaws/services/ecr/AmazonECRClient.java#L338)

Add Comment

[logbon72@gmail.com (JIRA)]

Joseph Orilogbon edited a comment on JENKINS-34958

[dguisinger@gmail.com (JIRA)]

Dan Guisinger commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I am having a similar issue, but I am in us-east-1... so what you guys are saying is that it should work... It appears to me like the plug-in isn't even trying to request a new token. I have tried this with both a regular Jenkins project where I can select my credentials from a dropdown and a Jenkins pipeline project where I select it by ID "ecr:aws". I can't get it to work consistently with: sh 'eval `aws ecr get-login`' but every time I'm having trouble, if I ssh into the machine and type "aws ecr get-login", builds work instantly for a few hours. this is very frustrating, its like the code isn't even trying to get a token, or if its failing its not putting it in the log. Ideas? I don't want to have to make get-login a cron job.....

Add Comment

[jenkins-ci.org@cowsgomoo.org (JIRA)]

Coltrey Mather commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

For my pipeline scripts, I've been using get-login before every call that needs it and haven't seen this issue. However, I'm not using the ecr plugin and I am using the machine IAM role for authentication/authorization. ex: sh "`aws ecr get-login --region $ {region} ` && docker push $ {tag} "

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Code changed in jenkins User: Nicolas De loof Path: src/main/java/com/cloudbees/jenkins/plugins/amazonecr/AmazonECSRegistryCredential.java src/main/java/com/cloudbees/jenkins/plugins/amazonecr/AmazonECSRegistryCredentialsProvider.java http://jenkins-ci.org/commit/amazon-ecr-plugin/8e02db93ae9c92bda407e55ecb4fa23ce84986d2 Log: Merge pull request #6 from logbon72/JENKINS-34958-add-region New credential ID with Region Name Compare: https://github.com/jenkinsci/amazon-ecr-plugin/compare/8004d9b6c556...8e02db93ae9c

Add Comment

[byteflinger@gmail.com (JIRA)]

Byte Flinger commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I think this may be an issue with either the AWS library or AWS itself. I am making use of the aws java ecr sdk and the token I get back for login does not work even though the expiration date on it says it has not yet expired

Add Comment

[grevenx@gmail.com (JIRA)]

Even André Fiskvik commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Byte Flinger no, the issue is because you can't set what region you want your login to be valid for and a fix has already been made in the plugin that should fix this when a new version is released.

Add Comment

[stan.domula@smartrac-group.com (JIRA)]

Stan Domula commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

I saw that the fix was merged, is there a schedule for when the plugin gets a new release?

Add Comment

[andr3w@gmail.com (JIRA)]

Drew Halloran commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Adding my voice to the chorus clamoring for a release; this is holding me up from deploying an otherwise fantastically useful little plugin.

Add Comment

[david@cybric.io (JIRA)]

david ficociello commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

This is also preventing us from moving forward with this plugin. We are in us-west-2. Is there any timeline at all for this? It would greatly help to know so we can either wait or move on to other solutions. Thanks

Add Comment

[alex@evaluagent.net (JIRA)]

Alex Richards commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Hi david ficociello, Drew Halloran, We managed to resolve this problem, after almost 3 weeks of conversation with AWS Support, by using the ecr-credential-helper. You can find the helper and documentation here: https://github.com/awslabs/amazon-ecr-credential-helper Good Luck!

Add Comment

[ecentinela@gmail.com (JIRA)]

Javier Martínez commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

When this patch is going to be released? This is a blocking issue for our company.

Add Comment

[micke_ficke@hotmail.com (JIRA)]

CL W edited a comment on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Hi guys, i'ver found that adding the --region $region_name to the aws ecr get-login command fixed a similar issue. would it be possible to add --region to the plugin and deploy. You can see this usage from the AWS - ECS Console click you your repository link. Then click the "View Push Command" button. It shows the use of the --region option. Is this being used by the ECR Jenkins plugin? Although i'm not completely convinced this is just a problem with this plugin. prior to today the plugin worked fine. However I upgraded my AWSCLI client to the latest version as well today (for some stupid reason) and now this plugin doesn't work. #annoying

Add Comment

[micke_ficke@hotmail.com (JIRA)]

CL W commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Hi guys, i'ver found that adding the --region $region_name to the aws ecr get-login command fixed a similar issue. would it be possible to add --region to the plugin and deploy. You can see this usage from the AWS - ECS Console click you repository link. Then click the "View Push Command" button. It shows the use of the --region option. Is this being used by the ECR Jenkins plugin?

Although i'm not completely convinced this is just a problem with this plugin. prior to today the plugin worked fine. However I upgraded my AWSCLI client to the latest version as well today (for some stupid reason) and now this plugin doesn't work. #annoying

Add Comment

[micke_ficke@hotmail.com (JIRA)]

CL W edited a comment on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials

Hi guys, i' ver ve found that adding the --region $region_name to the aws ecr get-login command fixed a similar issue. would it be possible to add --region to the plugin and deploy. You can see this usage from the AWS - ECS Console click your repository link. Then click the "View Push Command" button. It shows the use of the --region option. Is this being used by the ECR Jenkins plugin?

Although i'm not completely convinced this is just a problem with this plugin. prior to today the plugin worked fine. However I upgraded my AWSCLI client to the latest version as well today (for some stupid reason) and now this plugin doesn't work. #annoying

Add Comment

[ifernandezcalvo@cloudbees.com (JIRA)]

Ivan Fernandez Calvo assigned an issue to Ivan Fernandez Calvo

Jenkins / JENKINS-34958

Getting "Your Authorization Token has expired" when using ECR credentials

Change By: Ivan Fernandez Calvo Assignee: Nicolas De Loof Ivan Fernandez Calvo

Add Comment

[ifernandezcalvo@cloudbees.com (JIRA)]

Ivan Fernandez Calvo resolved as Fixed

Jenkins / JENKINS-34958 Getting "Your Authorization Token has expired" when using ECR credentials

Change By: Ivan Fernandez Calvo

Status: Open Resolved Resolution: Fixed

Add Comment

[catufunwa@gmail.com (JIRA)]

Chima Atufunwa commented on JENKINS-34958

Re: Getting "Your Authorization Token has expired" when using ECR credentials Give setting this env a try, AWS_ECR_DISABLE_CACHE. It causes the plugin to not use the local cache. Source, https://github.com/awslabs/amazon-ecr-credential-helper/pull/3

Add Comment