IOError: [Errno 11] Resource temporarily unavailable

[Bruno Vilhena]

Trying to run an ansible playbook in Jenkins, which starts a docker container (kubernetes-anywhere) to provision an environment.

I used to be able to run this without issues, by starting the contain detached and the executing the provisioning command line within the container, from jenkins.

Since a couple of days ago started having an issue where the build ends abruptly with this error: 

fatal: [localhost] => Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 589, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 793, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 1125, in _executor_internal_inner
    self.callbacks.on_ok(host, data)
  File "/usr/lib/pymodules/python2.7/ansible/callbacks.py", line 544, in on_ok
    display(msg, color='yellow', runner=self.runner)
  File "/usr/lib/pymodules/python2.7/ansible/callbacks.py", line 148, in display
    print msg2
IOError: [Errno 11] Resource temporarily unavailable

I've tried lots of work around, and the playbook seems to work properly if I run it from the command line, using the same user as Jenkins.

I've also updated nproc and open file limits, and rebooted both jenkins and the box itself, with no success.

Did a strace of the run and I can see the error logging:

lock_gettime(CLOCK_REALTIME, {1477026483, 732994400}) = 0
clock_gettime(CLOCK_REALTIME, {1477026483, 733171300}) = 0
read(3, 0xc2100f9000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {1477026483, 733531600}) = 0
write(3, "POST /v1.18/containers/f0a9b48ba"..., 465) = 465
epoll_wait(4, {{EPOLLIN|EPOLLOUT, {u32=2614059608, u64=139902583792216}}}, 128, 0) = 1
read(3, "HTTP/1.1 404 Not Found\r\nContent-"..., 4096) = 143
clock_gettime(CLOCK_REALTIME, {1477026483, 737483000}) = 0
open("/etc/localtime", O_RDONLY)        = 5
mmap(0xc210100000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc210100000
mmap(0xc20ffe0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc20ffe0000
clock_gettime(CLOCK_REALTIME, {1477026483, 738410500}) = 0
clock_gettime(CLOCK_REALTIME, {1477026483, 738642200}) = 0
read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 3661
clock_gettime(CLOCK_REALTIME, {1477026483, 738984000}) = 0
read(5, "", 4096)                       = 0
close(5)                                = 0
clock_gettime(CLOCK_REALTIME, {1477026483, 739523600}) = 0
clock_gettime(CLOCK_REALTIME, {1477026483, 739702600}) = 0
write(2, "time=\"2016-10-21T06:08:03+01:00\""..., 99) = 99
exit_group(1)                           = ?
+++ exited with 1 +++

Any ideas?

[riscad...@gmail.com]

Ended up redirecting the logs out of the container to a log file in order to avoid this issue.