ECS container termination issues

[ashok reddy C k]

Hi Chad,

We have seen a few issues in ecs agents' terminations, and profiles failed to execute.

ecs plugin version: 

8.0.0-775

docker version: 20

base image: CentOS-9


Amazon ECS Agent - v1.84.0 old version


Errors:

Error terminating container with id: GoCD48038f537cff4ecc85d470ea1b457040

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 437fadc6-dfa4-4def-8a61-82cb8f69f913; Proxy: null)

Error terminating container with id: GoCD315a8c8eb0c9410896e5d4c9646f2772

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 8be036f6-2c44-4166-aded-b926f427192e; Proxy: null)

Error terminating container with id: GoCD4eeaed4c9e00455d9ce6fd8cf2283ca6

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: b6024dac-e5d3-40be-a3a8-53ec748cc1bb; Proxy: null)

Error terminating container with id: GoCD959e980be51940a389f04ed1f5bad8dc

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 117d1239-54ef-4530-ac8a-11feaa70201f; Proxy: null)

Error terminating container with id: GoCDb6d294487d514f31a503ff6a67612cd2

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 9e5cbbe2-235f-4109-811e-5aca792a57d4; Proxy: null)

Error terminating container with id: GoCD6a3b736d1e594270af940d6a2771dee1

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: bf18192f-5155-4e7e-9d17-c53918f14797; Proxy: null)

Error terminating container with id: GoCDe9a2b208cae14341a386190dfec47371

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 66c0855e-d66c-4ff7-bd0b-be020296fe9b; Proxy: null)

Error terminating container with id: GoCD9981bcf9774f44939ac2c6d436893b96

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 01dad4e8-2926-4165-85d7-394de204bb17; Proxy: null)

Error terminating container with id: GoCD7f08715f386d481497275388b431e9d5

The task definition could not be deregistered because it is in the process of being deleted. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 3dd406be-bb10-4d08-88c9-e9c5b89dc6d0; Proxy: null)

[Chad Wilson]

The errors seem to imply tasks are trying to be deleted/cleaned up multiple times, which shouldn't normally happen.

Look at the ECS plugin-specific log in the logs folder and try and trace the sequence of events for an individual agent/container. I'd need to see proper logs to see what is happening.

If there is not enough detail, you may want to consider adding -Dplugin.com.thoughtworks.gocd.elasticagent.ecs.log.level=debug to your server startup-arg JVM argos so you can see more of what the plugin is doing.

-Chad

--
You received this message because you are subscribed to the Google Groups "GoCD Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnUs_6aqg41BzWSSBvtm5oE2KWfD%3DBJex-CC-1TA11czvg%40mail.gmail.com.