erl_crash.dump | Slogan: Kernel pid terminated (application

Scott E. MacKenzie

unread,

Jul 8, 2022, 10:04:20 AM7/8/22

to rabbitmq-users

Any ideas?

Fresh clean installation of OS

Problem:

apt install -f (to complete apt installation produces below results)

================

root@thinktank:~# uname -a
Linux thinktank 5.4.0-1080-aws-fips #87+fips1-Ubuntu SMP Fri Jun 10 20:13:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

================

root@thinktank:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

================

root@thinktank:~# apt update && apt upgrade -y && apt full-upgrade -y && apt autoremove -y
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:5 http://ppa.launchpad.net/maxmind/ppa/ubuntu focal InRelease
Hit:6 http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu focal InRelease
Get:8 https://esm.ubuntu.com/cis/ubuntu focal InRelease [3138 B]
Hit:9 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease
Hit:10 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease
Hit:7 https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu focal InRelease
Get:11 https://esm.ubuntu.com/fips-updates/ubuntu focal-updates InRelease [3918 B]
Fetched 121 kB in 10s (12.2 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up rabbitmq-server (3.10.5-1) ...
Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2022-07-08 10:01:01 EDT; 114ms ago
Process: 158381 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
Main PID: 158381 (code=exited, status=1/FAILURE)
dpkg: error processing package rabbitmq-server (--configure):
installed rabbitmq-server package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

===============

root@thinktank:~# cat /etc/apt/sources.list.d/rabbitmq.list
## Provides modern Erlang/OTP releases
##
## "focal" as distribution name should work for any reasonably recent Ubuntu or Debian release.
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
deb http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu focal main
deb-src http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu focal main

## Provides RabbitMQ
##
## "focal" as distribution name should work for any reasonably recent Ubuntu or Debian release.
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ focal main
deb-src https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ focal main

================

APT Keys installed no issues.

================

root@thinktank:~# head -n 8 /var/lib/rabbitmq/erl_crash.dump

=erl_crash_dump:0.5
Fri Jul 8 09:48:49 2022
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{low_entropy,[{crypto,strong_rand_bytes,1,[{file,"crypto.erl"},{line,1138}]},{credentials_obfuscation_svc,check,3,[{file,"src/credentials_obfuscation_svc.erl"},{line,197}]},{credentials_obfuscation_svc,init_state,0,[{file,"src/credentials_obfuscation_svc.erl"},{line,169}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,848}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}})
System version: Erlang/OTP 25 [erts-13.0.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Taints: crypto
Atoms: 10383
Calling Thread: scheduler:1
=scheduler:1

===================================

sudo systemctl start rabbitmq-server
Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

=================================

root@thinktank:~# journalctl -xe
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {line,169}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {gen_server,init_it,2,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"gen_server.erl"},{line,848}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {gen_server,init_it,6,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"gen_server.erl"},{line,811}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {proc_lib,init_p_do_apply,3,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"proc_lib.erl"},{line,240}]}]}}},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {credentials_obfuscation_app,start,[normal,[]]}}
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: in function application_master:init/4 (application_master.erl, line 142)
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: ancestors: [<0.120.0>]
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: message_queue_len: 0
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: messages: []
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: links: [<0.120.0>,<0.122.0>,<0.44.0>]
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: dictionary: []
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: trap_exit: true
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: status: running
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: heap_size: 376
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: stack_size: 28
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: reductions: 194
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: neighbours:
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: =INFO REPORT==== 8-Jul-2022::09:55:06.573481 ===
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: application: credentials_obfuscation
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: exited: {{shutdown,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {failed_to_start_child,credentials_obfuscaton_svc,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {low_entropy,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{crypto,strong_rand_bytes,1,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"crypto.erl"},{line,1138}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {credentials_obfuscation_svc,check,3,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"src/credentials_obfuscation_svc.erl"},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {line,197}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {credentials_obfuscation_svc,init_state,0,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"src/credentials_obfuscation_svc.erl"},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {line,169}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {gen_server,init_it,2,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"gen_server.erl"},{line,848}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {gen_server,init_it,6,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"gen_server.erl"},{line,811}]},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {proc_lib,init_p_do_apply,3,
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: [{file,"proc_lib.erl"},{line,240}]}]}}},
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: {credentials_obfuscation_app,start,[normal,[]]}}
Jul 08 09:55:06 thinktank rabbitmq-server[156016]: type: transient
Jul 08 09:55:08 thinktank rabbitmq-server[156016]: {"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_st>
Jul 08 09:55:08 thinktank rabbitmq-server[156016]: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_sta>
Jul 08 09:55:08 thinktank rabbitmq-server[156016]:
Jul 08 09:55:08 thinktank rabbitmq-server[156016]: Crash dump is being written to: erl_crash.dump...done
Jul 08 09:55:08 thinktank systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit rabbitmq-server.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Jul 08 09:55:08 thinktank systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit rabbitmq-server.service has entered the 'failed' state with result 'exit-code'.
Jul 08 09:55:08 thinktank systemd[1]: Failed to start RabbitMQ broker.
-- Subject: A start job for unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit rabbitmq-server.service has finished with a failure.
--
-- The job identifier is 612294 and the job result is failed.
Jul 08 09:55:08 thinktank audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rabbitmq-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=?>
lines 1304-1371/1371 (END)

Scott E. MacKenzie

unread,

Jul 8, 2022, 10:08:32 AM7/8/22

to rabbitmq-users

root@thinktank:~# erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell
"25"

Not erlang version:

RabbitMQ Erlang Version Requirements — RabbitMQ

Scott E. MacKenzie

unread,

Jul 8, 2022, 10:10:14 AM7/8/22

to rabbitmq-users

root@thinktank:~# rabbitmqctl version
3.10.5

Just in case you are wondering binary versions.

Scott E. MacKenzie

unread,

Jul 8, 2022, 10:48:26 AM7/8/22

to rabbitmq-users

Daemon in a nice crash loop.

Jul 8 10:47:53 thinktank rabbitmq-server[169994]:
Jul 8 10:47:53 thinktank rabbitmq-server[169994]: Crash dump is being written to: erl_crash.dump...done
Jul 8 10:47:53 thinktank systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Jul 8 10:47:53 thinktank systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Jul 8 10:47:53 thinktank systemd[1]: Failed to start RabbitMQ broker.

jo...@cloudamqp.com

unread,

Jul 8, 2022, 5:42:00 PM7/8/22

to rabbitmq-users

I think the critical line is {failed_to_start_child,credentials_obfuscaton_svc,{low_entropy,[{crypto,strong_rand_bytes,1,

Reading from the Erlang manual it fails due to "due to lack of secure "randomness"." [1]

Might be that the OpenSSL (which OpenSSL do you have?) doesn't reply in the right way/isn't installed.

/Johan

[1] https://www.erlang.org/docs/25/man/crypto.html#strong_rand_bytes-1

Reply all

Reply to author

Forward

erl_crash.dump | Slogan: Kernel pid terminated (application_controller)

Scott E. MacKenzie

Scott E. MacKenzie

Scott E. MacKenzie

Scott E. MacKenzie

jo...@cloudamqp.com