Ubuntu: /run/udev/data full

[Jason]

Hi,

We have a small Nomad installation and one of our nodes is failing to run jobs because /run is full. Specifically it was /run/udev/data where there were hundreds of thousands of files. I've run 'udevadm info --cleanup-db' which has resolved the issue. I'd like to know if this is something that nomad should be cleaning up for me or is it likely a misconfiguration on my side.

Thanks 

[msch...@hashicorp.com]

Hi Jason!

I tested exec and docker drivers on Ubuntu 16.04 and docker creates a new udev database entry for its virtual ethernet interface (something like /run/udev/data/n11 for /sys/subsystem/net/devices/vethe8775d2). The exec driver doesn't create any new entries.

What version of Docker are you running? Are you running a lot of Docker tasks? Perhaps periodic batch jobs? Can you confirm it's network entries (n[0-9]+) filling your udev database? Do you have an unusually small /run filesystem (check df | grep /run)?

My main concern is that there's some interaction between Nomad and Docker that leaks virtual network interfaces. If that's the case it's a Nomad bug that needs fixing (or perhaps a Docker bug that's hopefully been fixed already!).

[Jason]

Oops. I'm sorry. I realize now that my post was a little light on details. Let me try again.

Ubuntu 16.04

Nomad 0.5.4 (We had upgraded from 0.5.1 a couple of days before the incident)

/run = 2GB

Running jobs is less than 10. We use the exec and docker driver. This is simply an evaluation/test setup. The vast majority of the files were prefixed with "+cgroup". I've since cleaned these up with the udmadm command I mentioned. I'll keep an eye on this to see if it reoccurs and report back.

Thanks

[Michael Schurter]

Thanks for the extra details. This is really peculiar. I'm running Ubuntu 16.04 locally and couldn't reproduce it. I see no +cgroup entries when creating exec or docker jobs. Same with the Ubuntu 16.04 vagrant in the root of the Nomad repo.

Do you have any custom udev rules installed or any container/virtualization related packages installed that may have added rules that create these cgroup entries?

Do you have any systemd updates available on this system? Since it seems like these entries are being improperly leaked (since you can fix everything by running cleanup), I'm wondering if there's just a bug that's been fixed on my systems.  

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/b24eae43-cb2c-4aaa-8d85-c8ee0ca589a8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jason]

Hi Michael,

You were right about the system updates. I pulled the latest updates and restarted everything and the +cgroup udev entries are gone.

Many thanks,

Jason

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

[msch...@hashicorp.com]

Goodness that's quite the OS bug. Glad an upgrade fixed it!

[王小锋]

Hi, Jason

I've encountered the similar problem, could you tell me how do you update system?

thanks a lot.

[王智源]

Me too.

在 2018年7月16日星期一 UTC+8下午4:08:55，王小锋写道：