create: Resource temporarily unavailable

[Beto]

I'm seeing erors similar to this on multiple servers:

   error: Could not open database txn /var/cfengine/state/cf_lock.lmdb: Invalid argument
   error: Unable to create 'last.temp_cf_bundle__var_cfengine_templates_main_cf_template.insert_lines.insert_type.-fulla01.insertline_command_directory____usr_sbin__etc_postfix_main_cf_36_2285_MD5=9fdf5ab7b5f56b0e07860f49e4f7c6a4'. (create: Resource temporarily unavailable)

What resource is it complaining about?  I radically increased nfiles but it had no affect.

root@fulla01-oci [outputs]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 240838
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 221184
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 240838
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[vratislav...@northern.tech]

On Thu, 2022-05-05 at 07:15 -0700, Beto wrote:
> I'm seeing erors similar to this on multiple servers:
>
>    error: Could not open database txn /var/cfengine/state/cf_lock.lmdb: Invalid argument
>    error: Unable to create
> 'last.temp_cf_bundle__var_cfengine_templates_main_cf_template.insert_lines.insert_type.-
> fulla01.insertline_command_directory____usr_sbin__etc_postfix_main_cf_36_2285_MD5=9fdf5ab7b5f56b0e
> 07860f49e4f7c6a4'. (create: Resource temporarily unavailable)
>
> What resource is it complaining about?  I radically increased nfiles but it had no affect.

Hi!
This is a known issue: https://tracker.mender.io/browse/CFE-3807 Unfortunately, we haven't been able
to reliably reproduce the issue (or often enough) to be able to debug and fix it.

Which OS are you running? And does this happen during normal agent runs? (We have mostly seen it
happening during bootstrap).

Thanks!

--
Vratislav

[t...@dix.cz]

Hi,

I can see same issue too.

Often: Debian GNU/Linux 11 (bullseye),
less often: Oracle Linux 7.8.

Althought the workload are different.


Thanks,
Tomas.

[Beto]

Hi!

Thank you for the info.  I'm seeing this on:

Oracle Linux Server release 6.10

Oracle Linux Server release 7.9

Red Hat Enterprise Linux Server release 7.9

On many servers multiple times/day.

[craig.c...@northern.tech]

FYI, we are looking into this issue currently at https://tracker.mender.io/browse/CFE-3807.

If you can think of any correlations or connections to the issue I am working on making as simple of a reproducer as possible.

Thanks,

Craig

[t.d...@servicemusic.org.uk]

We have lots of RHEL7 machines, with community edition 3.15.5, and "control_executor_schedule" set to give a run every 20 minutes.  Across the estate we get a handful of instances like this per day.  Alas, there doesn't seem to be any pattern, and I don't seem able to reproduce it "on demand".

-- David Lee

-- Diamond Light Source

[craig.c...@northern.tech]

Very good, thanks for the feedback.

Yes, we are in much the same boat with our CI systems. We do see the error often enough to annoy us but not enough to see a pattern.

Hopefully with our recent changes we will get some more leads with backtraces in our debug builds in CI automated tests.