目前使用 apisix:2.15.0-centos 程序出现死锁，CPU 跑到 100%，这个是什么问题啊？

[Yachao Zhang]

openresty 里面有大量报错日志如下

prometheus.lua:920: log_error(): Error while setting 'node_info{hostname=""}' to '1': 'no memory', client: xxx.xxxx.xxxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"

prometheus.lua:920: log_error(): Error while setting 'etcd_modify_indexes{key="x_etcd_index"}' to '8553921': 'no memory', client: xxx.xxxx.xxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"

通过 strace 发现程序在大量调用  futex

futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1)    = 0
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1)    = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1)    = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1)    = 0

perf 追踪结果如下：

Samples: 31K of event 'cycles', Event count (approx.): 18419237386
  Children      Self  Command    Shared Object           Symbol
+   87.95%     0.00%  openresty  [unknown]               [k] 0x0000000000000080
+   84.44%    83.62%  openresty  nginx                   [.] ngx_shmtx_lock
+    4.85%     0.10%  openresty  [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
+    4.57%     0.03%  openresty  [kernel.kallsyms]       [k] do_syscall_64
+    4.36%     0.07%  openresty  [kernel.kallsyms]       [k] __x64_sys_futex
+    4.30%     0.08%  openresty  [kernel.kallsyms]       [k] do_futex

pstack 结果如下

#0  0x0000000000449bf3 in ngx_shmtx_lock (mtx=0x7fc9d4ded068) at src/core/ngx_shmtx.c:78
#1  0x0000000000572caf in ngx_meta_lua_ffi_shdict_get (zone=0x1281138, key=0x7fc9d1d69080 "__ngx_prom__delete_count", key_len=24, value_type=0x7fc9de85f828, str_value_buf=0x7fc9de844380, str_value_len=0x7fc9de875ef8, num_value=0x7fc9de85f868, user_flags=0x7fc9de85f848, get_stale=0, is_stale=0x7fc9de8636f8, err=0x7fc9de8723a8) at /tmp/tmp.YN7822nAJt/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta/ngx_meta_lua_shdict.c:1631
#2  0x00007fca19caa8d7 in ?? ()
#3  0x00007fc9de85f868 in ?? ()
#4  0x00007fc9de85f848 in ?? ()
#5  0x0000000000000000 in ?? ()

这个是什么问题啊？

还有就是 pstack 的有大量栈帧没有符号表，这个有解决办法么？

[Yachao Zhang]

程序运行 13 天之后出现死锁，重启之后暂时不会出现

[Junlong Li]

这个可以使用  orxray 来进行分析。

https://xray.openresty.com

[xue]

为什么在这里问，为什么不去apisix.

我的发帖子看不到吗？