Performance issues on long running Agent containers?

[Sofoklis Nikiforos]

We had issues with agents running for long periods of time, affecting performance of the host.  Wondering if anyone experienced this kind of performance issues?

There were 20 agents, running as container on a dedicated host, in addition to other spurious container workloads.  

Over time the responsiveness of other container workloads degraded - which was alleviated when the agents were stopped or restarted.  We had a weak suspicion of a possible memory leak, but did not investigate further.  We started using a periodic restart as a preventative workaround.

Admittedly- we just upgrade to GoCD 19.11, which is still fairly far behind, but the newer JDK has more capabilities for telemetry / profiling, so we may eventually revisit the investigation.