CoreOS Monitoring (snmp)

[Karan Sikka]

Hey everyone,

What are your thoughts on server monitoring? I would like to be able to install snmpd on my CoreOS boxes, since it provides so much functionality out of the box, but I'm not sure what the easiest way is to do this in CoreOS.

Thanks,

Karan

[Karan Sikka]

Just found this post, I will check it out.

http://www.nuxeo.com/blog/development/2014/03/coreos-monitoring-diamond-graphite/

[Karan Sikka]

Here's the hack they used to run a collector in a container:

The next step is to tell the different Diamond collectors that they should look for metrics in /host_procinstead of /proc. Unfortunately, most of these collectors have the path hard-coded, so for the moment I forked it and hard-coded /host_proc (yes I know, but I was eager to test it). Now that I know it works I will try to parameterize this and send a pull request.

So... That's awkward... I'm not about to fork snmpd, that sounds like a nightmare.

I will look into modifying the image to get snmpd onto it. Then I can configure it using the writable /etc.

Would still love to hear what your thoughts are about this.

Karan

[Андрей Козырев]

Hi CoreOS developers,

I join the question. In our organization we are looking to build a private cluster solution based on coreos/docker. We obviously need monitoring. Currently it is only possible (if at all) through hacks like mounting host's /proc to the running docker container or building a custom CoreOS image. Both variants do not look like a production ready solution to rely upon. Is there any strategic view or plans for natural support of monitoring baked into a CoreOS image?

Thanks,

Andrey.

[Greg KH]

On Tue, Apr 22, 2014 at 02:29:08PM -0700, Андрей Козырев wrote:
> Hi CoreOS developers,
>
> I join the question. In our organization we are looking to build a private
> cluster solution based on coreos/docker. We obviously need monitoring.
> Currently it is only possible (if at all) through hacks like mounting host's /
> proc to the running docker container or building a custom CoreOS image. Both
> variants do not look like a production ready solution to rely upon. Is there
> any strategic view or plans for natural support of monitoring baked into a
> CoreOS image?

Some of us have thrown around the idea of maybe adding something like
sysdig to the image, but once you start using lots of containers,
determining what is happening in an individual container separate from
the system as a whole, is a difficult task.

Do you have any suggestions on things that could be used in CoreOS to
make this work better?

thanks,

greg k-h

[Андрей Козырев]

Greg,

Thanks for response! Monitoring agent inside a container is not an option. The target of monitoring is the ship itself, not a docker container. I would expect something similar to etcd. The need to have a distributed registry on the ship level was obvious - etcd was built into CoreOS. Now the same must be done with monitoring software. Ideally this "monitord" should implement some common protocol, like SNMP that can be commonly consumed. But I'm not an expert here, I'm just trying to see what's possible. Originally I wanted to install a zabbix_agent on CoreOS and found it non-trivial at least. 

Thanks,

Andrey.

[Brandon Philips]

On Tue, Apr 22, 2014 at 6:53 PM, Андрей Козырев
<andrey....@gmail.com> wrote:
> Originally I wanted to install a zabbix_agent on CoreOS and found it
> non-trivial at least.

What sorts of issues did you run into using the zabbix agent on
CoreOS? To get access to host statistics and proc using systemd-nspawn
with flags similar to our "toolbox" script should work well:
https://github.com/coreos/toolbox/blob/master/toolbox#L26

If you hit problems let me know; this is something that we want to work.

Thanks,

Brandon

[Андрей Козырев]

Brandon,

I was not aware of systemd-nspawn. Thanks for mentioning it and toolbox. It looks like what we need!

Thanks,

Andrey. 

[Brandon Philips]

On Wed, Apr 23, 2014 at 2:58 PM, Андрей Козырев
<andrey....@gmail.com> wrote:
> I was not aware of systemd-nspawn. Thanks for mentioning it and toolbox. It
> looks like what we need!

Great, let me know how it goes. It would be great to have a guide for
running zabbix.

[Андрей Козырев]

I have run all zabbix nodes (mysql, zabbix_server, zabbix_frontend, zabbix_agent(s)) on a cluster of 7 CoreOS machines.

All payload daemons run within docker containers. 

zabbix_agent running under docker container is showing the stats about the host machine, not about docker container, which is exactly what we need. The seems to be that cgroup limits are not affecting /proc and sysinfo calls. 

Thanks,

Andrey.

[Kyle Mathews]

Andrey — did you get this working? A blog post about monitoring would be great.

[Jon Wood]

I'm not sure how useful it is, since I'm just installing it myself to have a play, but https://github.com/google/cadvisor looks like it could be an interesting tool in this space if you're interested in which containers are responsible for resource usage.

[pram]

Hi,

Is there a blog post about this? New to CoreOS and Zabbix..

Thanks,

Pat

[pram]

Hi Brandon - Can you please point me to any blog on installing zabbix for coreos host monitoring? Any issues using zabbix on coreOS?