How to generate erl_crash.dump in docker

130 views
Skip to first unread message

Guangyu Suo

unread,
Dec 6, 2021, 10:53:27 PM12/6/21
to rabbitmq-users
Hi, all

I have a rabbitmq cluster running in docker, which the version is 3.8.26, I want to test if there will be an erl_crash.dump file generated when rabbitmq crashes in docker for some reason. 

I sent a SIGUSR1 signal to erlang beam process as described here https://www.erlang.org/doc/apps/erts/crash_dump.html, expected to get a erl_crash.dump,  but got nothing, except that the docker container exists and goes back again, but if test without container, it will generate erl_crash.dump as expected, can anyone has any clue about this?

the container test procedure is like follows:

(rabbitmq)[root@control1 ~]$
(rabbitmq)[root@control1 ~]$ ps -ef | grep erlang
rabbitmq      25       7 53 11:29 ?        00:00:19 /usr/lib64/erlang/erts-11.2.2.7/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -sbwt none -B i -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -epmd_port 4369 -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -kernel inetrc '/etc/rabbitmq/erl_inetrc' -lager crash_log false -lager handlers []
rabbitmq      78       1  0 11:29 ?        00:00:00 /usr/lib64/erlang/erts-11.2.2.7/bin/epmd -daemon
root         288     123  0 11:30 pts/0    00:00:00 grep --color=auto erlang
(rabbitmq)[root@control1 ~]$
(rabbitmq)[root@control1 ~]$
(rabbitmq)[root@control1 ~]$ kill -s SIGUSR1 25
(rabbitmq)[root@control1 ~]$ [root@control1 ~]#
[root@control1 ~]#
[root@control1 ~]# docker exec -uroot -it rabbitmq bash
(rabbitmq)[root@control1 /]$ cd
(rabbitmq)[root@control1 ~]$ ls
anaconda-ks.cfg  anaconda-post.log  original-ks.cfg
(rabbitmq)[root@control1 ~]$ ls /var/lib/rabbitmq/
mnesia
(rabbitmq)[root@control1 ~]$ env
KOLLA_DISTRO_PYTHON_VERSION=3.6
LANG=zh_CN.UTF-8
HOSTNAME=control1
OLDPWD=/
RABBITMQ_CLUSTER_COOKIE=km8b5IpEqndxyTmeDLaiQ5iiTokM8c13MhPBBmBB
KOLLA_BASE_ARCH=x86_64
which_declare=declare -f
RABBITMQ_LOG_DIR=/var/log/kolla/rabbitmq
KOLLA_SERVICE_NAME=rabbitmq
PWD=/root
HOME=/root
KOLLA_INSTALL_METATYPE=rdo
KOLLA_INSTALL_TYPE=binary
KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PS1=\[\033[01;31m\](rabbitmq)\[\033[00m\][\[\033[00;33m\]\u\[\033[00m\]@\[\033[02;32m\]\H\[\033[00m\] \[\033[02;31m\]\w\[\033[00m\]]$
KOLLA_BASE_DISTRO=centos
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
}
_=/usr/bin/env
(rabbitmq)[root@control1 ~]$

the docker inspect hostconfig output is as follows:

        "HostConfig": {
            "Binds": [
                "/etc/kolla/rabbitmq/:/var/lib/kolla/config_files/:ro",
                "/etc/localtime:/etc/localtime:ro",
                "rabbitmq:/var/lib/rabbitmq/:rw",
                "kolla_logs:/var/log/kolla/:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },

I also test to configure core dumps in docker as describe here: https://ddanilov.me/how-to-configure-core-dump-in-docker-container, but this will not help to generate erl_crash.dump, except indeed will generate linux core dump.


yon...@laposte.net

unread,
Dec 7, 2021, 12:04:13 AM12/7/21
to rabbitm...@googlegroups.com
‌may I think this is a docker issue?
 
De : "Guangyu Suo"
A : "rabbitmq-users"
Envoyé: mardi 7 Décembre 2021 11:53
Objet : [rabbitmq-users] How to generate erl_crash.dump in docker
 
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/be71daa2-6148-4c4a-9739-bae30284352en%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages