ECS Plugin issues

[ashok reddy C k]

Hi Chad/GoCD Team,

We are using the following plugins and base image as centos-9,AWS optimized AMI 

GoCD Elastic Agent Plugin for Amazon ECS Version

9.0.0-896

Docker VersionDockerVersion: 20.10.25
ECS Agent Version1.84.0
issues

1) ECS Ec2 servers, we have given 12 as max in the cluster profile

But it's shown max 10 servers in the error logs



2026-02-19 02:05:41,739 INFO  [899991@MessageListener for ServerPingListener] ECSElasticPlugin:72 - [server-ping] Cluster has total 12 LINUX instances which is beyond permissible limit(10). Terminating idle instances.
 

is there any way to limit extend?

due to this max as 12 , ui showed 11 , keep on one ec2 server, span up and terminating.....

Is there any way to limit the extension? 
Due to a maximum of 12, the UI shows like (10),(11),(12)...etc. We need to keep  on one EC2 serverspinning up and terminating instances. 

During this time, jobs or pipelines are being assigned to terminated instances.
2) ec2 servers are not terminated as we expected in profile time intervals

StopLinuxInstanceAfter
60
TerminateStoppedLinuxInstanceAfter
2
manually terminating the ECS Ec2 servers


Production jobs were delayed last week by 40 minutes due to this terminated server registered for the task(out of 12 to 10 )


3)Regarding ECS EC2 tasks, is there a way to assign them equally across each EC2 server? 

For example, if there are 5 servers and 50 tasks, can we configure it so that each server runs 10 tasks?


Please find the attachment screenshots via a separate email

ECS Ec2 server is registered and healthy but container is not assigned



regarding above issues we have connected to AWS support team 
ECS
EC2
these 2 teams said ec2 and ec2 cluster is healthy , issue with plugins (go-agent)

[ashok reddy C k]

Sorry, we are using the below updated  Docker and ecs agent versions

• Docker VersionDockerVersion: 25.0.14

• ECS Agent Version1.101.2

[Chad Wilson]

Sorry, I find this too difficult to follow so I am not sure I understand, but this is almost certainly not an AWS Support issue, and probably a configuration problem on your side.

For the max instances, perhaps you are looking the wrong cluster profile; or perhaps you have two different GoCD server instances creating elastic plugins inside the same ECS cluster with the same elastic profiles and the two GoCD servers are confused about who is responsible for which elastic agent.

In any case, the same advice applies as earlier - you'll need to look at the actual ECS plugin logs (plugin-com.thoughtworks.gocd.elastic-agent.ecs.log.*) on your server in the godata/logs directory. The job console logs are user-facing. If you are having scheduling/termination/registration issues, this is an administrator problem and you need to look at the plugin logs which will tell you what it is doing/why - and you can correlate with what you see in ECS or container logs for individual elastic agent containers.

Those job console logs may mean a number of things, including the container not being started by ECS, or the Gocd agent failing to start correctly.

Jobs are allocated to EC2 instances based on comparing the instance's available capacity of memory/CPU cores not reserved by existing agents, and new instances only created when the existing ones with matching instance-level configuration (InstanceType, AMI etc) don't have sufficient available capacity when deducting existing running tasks/agents. If it's over-allocating agents/tasks to a single EC2 instance, you probably want to review your elastic profile ReservedMemory and CPU to see if it's appropriate to give you the allocation you want. You can't directly configure the # of containers/tasks, but you can configure your elastic profiles such that each task reserves 1/10 of available memory or CPU for whatever instance type you are using, which would prevent it allocating more than 10 tasks to the instances available. It gets more complex when you have lots of different elastic profiles with different memory/CPU requirements, but this is a common container/cluster "packing" optimization problem to consider.

-Chad

--
You received this message because you are subscribed to the Google Groups "GoCD Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnW7AqNbf0C-2CSqrCVqLqKHZDG4p%2B0XRfN_mq7fqce9TA%40mail.gmail.com.

[ashok reddy C k]

Hi Chad,

we are using only one Cluster not  two (2) and we believe there is an issue with plugin version 9.0.0-896

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAEe7TBwNi9jf6obav3XGKU%3D%3DK3L_puT8EqPBF%2Byg5sVv9T%2B00g%40mail.gmail.com.

[Chad Wilson]

As I said, there are multiple reasons this could happen with misconfiguration, not solely multiple GoCD instances.

If you are confident you have found a bug that you can reproduce, please report it properly with versions, logs and cluster/elastic profile configuration and steps/timeline to reproduce at https://github.com/gocd/gocd-ecs-elastic-agent/issues

If you are having problems with container startup, agent registration and termination it looks like your setup is unstable for some reason, and you'll need to dig into logs to determine what is happening and address one problem at a time. The plugin logs, container logs and timeline of events are critical.

The plugin is actively used day-to-day to build GoCD itself on https://build.gocd.org across windows (Core 2022) and Linux (AL2023) instances of various instance types/sizes so any issue is likely not widespread. As with any.cloud container orchestration, it is very sensitive to cluster and AWS configuration, subnets, networking and AMI choice so the details are important and unfortunately there isn't anything that can be done to help with vague descriptions alone.

-Chad

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnWo7nO6fm8pnA4OCZOkphh%3DiJDodX7yjxXDO_x47epLFw%40mail.gmail.com.