vmtoolsd.service fails after first reboot

[Gary Clayburg]

I'm using this script to provision CoreOS onto VMware ESXi:

http://bit.ly/coreos-vmware

Things look pretty good for the most part, but I noticed a strange issue this morning.  When I first boot a new CoreOS server, vmtoolsd.service seems to load and work fine.  The vSphere client shows extra information about the new CoreOS node, as expected.  However, if I login to the server and reboot it, the vmtoolsd.service stops working.  I get this error in the journal:

Dec 23 16:19:13 queenbee70 vmtoolsd[523]: /usr/share/oem/bin/vmtoolsd: error while loading shared libraries: libvmtools.so.0: cannot open shared object file: No such file or directory


I am using version 536.0.0.  Further investigation:

queenbee70 gclaybur # ldd /usr/share/oem/bin/vmtoolsd
    linux-vdso.so.1 (0x00007fffc15bd000)
    libvmtools.so.0 => not found
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fabd0f29000)
    libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fabd0d25000)
    libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fabd0ac6000)
    libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fabd0776000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fabd03bb000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fabd1146000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fabd01b7000)
    libffi.so.6 => /lib64/libffi.so.6 (0x00007fabcffae000)
queenbee70 gclaybur # find / -name libvmtools.so.0
/usr/share/oem/lib64/libvmtools.so.0

I was able to get it working by adding an EnvironmentFile to the vmtoolsd.service.  Is this the correct way to fix this issue?

queenbee70 gclaybur # cat /etc/systemd/system/vmtoolsd.service
[Unit]
Description=VMware Tools Agent
Documentation=http://open-vm-tools.sourceforge.net/
ConditionVirtualization=vmware

[Service]
EnvironmentFile=-/etc/sysconfig/vmtoolsd
ExecStartPre=/usr/bin/ln -sfT /usr/share/oem/vmware-tools /etc/vmware-tools
ExecStart=/usr/share/oem/bin/vmtoolsd
TimeoutStopSec=5

workerbee70 gclaybur # cat /etc/sysconfig/vmtoolsd
LD_LIBRARY_PATH=/usr/share/oem/lib64

[Ilya Dmitrichenko]

Hi Gary,

On Tue Dec 23 2014 at 5:00:30 PM Gary Clayburg <juggl...@gmail.com> wrote:

I'm using this script to provision CoreOS onto VMware ESXi:

http://bit.ly/coreos-vmware

Things look pretty good for the most part, but I noticed a strange issue this morning.  When I first boot a new CoreOS server, vmtoolsd.service seems to load and work fine.  The vSphere client shows extra information about the new CoreOS node, as expected.  However, if I login to the server and reboot it, the vmtoolsd.service stops working.  I get this error in the journal:

Dec 23 16:19:13 queenbee70 vmtoolsd[523]: /usr/share/oem/bin/vmtoolsd: error while loading shared libraries: libvmtools.so.0: cannot open shared object file: No such file or directory


I am using version 536.0.0.  Further investigation:

queenbee70 gclaybur # ldd /usr/share/oem/bin/vmtoolsd
    linux-vdso.so.1 (0x00007fffc15bd000)
    libvmtools.so.0 => not found
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fabd0f29000)
    libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fabd0d25000)
    libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fabd0ac6000)
    libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fabd0776000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fabd03bb000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fabd1146000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fabd01b7000)
    libffi.so.6 => /lib64/libffi.so.6 (0x00007fabcffae000)
queenbee70 gclaybur # find / -name libvmtools.so.0
/usr/share/oem/lib64/libvmtools.so.0

I was able to get it working by adding an EnvironmentFile to the vmtoolsd.service.  Is this the correct way to fix this issue?

Yes, it's what I would do.

queenbee70 gclaybur # cat /etc/systemd/system/vmtoolsd.service
[Unit]
Description=VMware Tools Agent
Documentation=http://open-vm-tools.sourceforge.net/
ConditionVirtualization=vmware

[Service]
EnvironmentFile=-/etc/sysconfig/vmtoolsd

You can replace that with just this:

EnvironmentFile=LD_LIBRARY_PATH=/usr/share/oem/lib64

 

No need for writing a separate file.

Cheers,

-- 

Ilya 

[Gary Clayburg]

It looks like this issue reappears after rebooting again.  Apparently /etc/systemd/system/vmtoolsd.service is overwritten on every boot?  I see this in the journal:

Dec 23 17:02:56 workerbee70 coreos-cloudinit[485]: 2014/12/23 17:02:56 Writing unit "vmtoolsd.service" to filesystem
Dec 23 17:02:56 workerbee70 coreos-cloudinit[485]: 2014/12/23 17:02:56 Writing file to "/etc/systemd/system/vmtoolsd.service"

[Ilya Dmitrichenko]

Have you implemented the change in cloud config file the vm is provisioned with, or you just dropped those changes on the box?

[Michael Marineau]

A new release to fix this issue should roll out today. https://github.com/coreos/bugs/issues/216

[Gary Clayburg]

OK, thanks Michael.  This github issue sounds exactly like what I am seeing.