Anyone having docker memory/cpu issues after last couple of days of updates?

[Stephen Hewitt]

I'm just starting to dig into the problem but wanted to see if anyone else is experiencing or has fixed similar issues.

We have a number of docker containers running on a CoreOS EC2 instance. Over the last couple of days we have been experiencing major memory usage and CPU problems on the server. The server is pretty much unresponsive -- I've been able to run top and note that dockerd is consuming the bulk of the memory and CPU. I see that there have been some recent releases of CoreOS, which this server has updated to.

Has anyone else seen this? Any suggestions for how to begin troubleshooting?

Thank you!

[Brandon Philips]

What version of CoreOS Container Linux are you running? cat /etc/os-release?

What software is talking to dockerd? How many containers are you running? How many containers are being created/destroyed at what sort of rate?

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Hewitt]

I’m sorry, I must have forgotten to update the thread. It turns out there were some container issues with the automatic update of CoreOS and Docker (breaking changes in Docker, particularly around a Jenkins plugin.) I’ve resolved these issues and things are behaving again.

Thank you,

Stephen

[Brandon Philips]

Can you link to your fix or plugin to help others who might arrive here via a search?

[Stephen Hewitt]

Great idea, thanks.

We are using the Docker plugin for Jenkins:

http://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

We upgraded to 0.16.2 and that solved the issues we were having with Docker. 

The error message we were seeing in the Jenkins container’s log had this:

{"message":"starting container with HostConfig was deprecated since v1.10 and removed in v1.12"}

More information can be seen here, although the suggestion here seemed to be to install a different plugin. We found that upgrading the Docker Plugin linked above was sufficient.

https://github.com/jenkinsci/docker-plugin/issues/403

Thanks,

Stephen