average memory usage in docker using prometheus

[mohamed...@gmail.com]

I am running 1000 docker containers sequentially. Each container instance runs a specific job. After the execution of the job, I kill the container to release resources and run another job within another instance and so on.

I would like to get the memory usage of each container. One value per container expressing the average memory usage.

How is it possible using prometheus?

I am not familiar with prometheus.

[Björn Rabenstein]

For how long are your containers running? If your containers are
short-lived (like a minute or less), Prometheus will not be able to
monitor them reliably. In that case, you might want to look into the
Pushgateway https://github.com/prometheus/pushgateway . Your jobs
needed to inquire the average memory usage from the container and push
them to the Pushgateway before they exit.

If your containers are longer running (multiple minutes or longer), a
carefully crafted `average_over_time` expression with a long range
selector should do the trick.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Mohamed Boussaa]

Thank you for your reply. Do you have an idea of how using it using docker to get the memory usage per container? Actually my containers run for 8s.

I run 

docker run -d -p 9091:9091 prom/pushgateway

and then ? 

Thank you in advance

[Björn Rabenstein]

On Mon, Nov 23, 2015 at 10:20 AM, Mohamed Boussaa
<mohamed...@gmail.com> wrote:
> Thank you for your reply. Do you have an idea of how using it using docker
> to get the memory usage per container? Actually my containers run for 8s.
> I run
>
> docker run -d -p 9091:9091 prom/pushgateway
>
> and then ?

Yeah, 8s is definitely not enough to get reliably scraped enough for
meaningful memory usage.

So you need to ask a Docker expert how to inquire the memory usage
from within a container and then add a few lines to your job that push
that information to the Pushgateway upon termination of the job.