Error: Unable to create worker process, insufficient resources

105 views
Skip to first unread message

Sagi Mann

unread,
Dec 5, 2022, 11:31:00 AM12/5/22
to rabbitmq-users
### Name and Version

rabbit 3.11.2

### What steps will reproduce the bug?

* Env: GCP cluster on n2-standard CPU on GKE 1.22, rabbitMQ installed through bitnami/rabbitmq helm chart
* Attached values.yaml (as v.txt) used during the installation of the chart

* Install using `helm install --version 11.1.1 my-rabbitmq -f values.yaml bitnami/rabbitmq -n myproject`

Every once in a while, we're seeing `Does not have minimum availability` error next to the rabbitmq pod in the GKE workload view. When looking at the log explorer around the time of unavailability, we see the following failed startup attempt:

```
Welcome to the Bitnami rabbitmq container
Subscribe to project updates by watching https://github.com/bitnami/containers
Submit issues and feature requests at https://github.com/bitnami/containers/issues
==> ** Starting RabbitMQ setup **
==> Validating settings in RABBITMQ_* env vars..
==> Initializing RabbitMQ...
==> Persisted data detected. Restoring...
==> No custom scripts in /docker-entrypoint-initdb.d
==> ** RabbitMQ setup finished! **
{}
==> ** Starting RabbitMQ **
Failed to create poll thread
Aborted
Welcome to the Bitnami rabbitmq container
Subscribe to project updates by watching https://github.com/bitnami/containers
Submit issues and feature requests at https://github.com/bitnami/containers/issues
==> ** Starting RabbitMQ setup **
==> Validating settings in RABBITMQ_* env vars..
==> Initializing RabbitMQ...
==> Persisted data detected. Restoring...
==> No custom scripts in /docker-entrypoint-initdb.d
==> ** RabbitMQ setup finished! **
==> ** Starting RabbitMQ **
/opt/bitnami/erlang/lib/erlang/erts-13.1.1/bin/inet_gethost[83]: WARNING:Could not fork(), errno = 11
/opt/bitnami/erlang/lib/erlang/erts-13.1.1/bin/inet_gethost[83]: WARNING:Unable to create worker process, insufficient resources
```

Looking at erlang code, this is a result of a `fork()` attempt that fails during [create_worker](https://github.com/erlang/otp/blob/OTP-25.0/erts/etc/common/inet_gethost.c#L1629) function.

limits inside the container are as follows:
```
$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 29789
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
```
But these don't show host limits which I could not find how to access within a k8s node. Where ever I looked, it says direct node access is not available (GCP blocks root access to containerd or so it seems)

### Are you using any custom parameters or values?

attached above

### What is the expected behavior?

rabbitmq server should continue running and not crash the container

### What do you see instead?

the container appears down in the workload view and the log explorer shows the mentioned problem fork()ing inet_host workers

### Additional information

_No response_
v.txt

Luke Bakken

unread,
Dec 5, 2022, 5:39:04 PM12/5/22
to rabbitmq-users
Hello,

Please, when cross-posting all over the internet, link to where you have asked this question -




All the information you have provided so far points to a limit imposed by the host system running the container.

The following links seem promising:
At any rate, it's not a RabbitMQ or Erlang issue but something lower level.

Good luck!
Luke

On Monday, December 5, 2022 at 8:31:00 AM UTC-8 sagi...@gmail.com wrote:
### Name and Version

rabbit 3.11.2

### What steps will reproduce the bug?

* Env: GCP cluster on n2-standard CPU on GKE 1.22, rabbitMQ installed through bitnami/rabbitmq helm chart
Reply all
Reply to author
Forward
0 new messages