nginx worker eating a lot of virtual memory

35 views
Skip to first unread message

louis....@gmail.com

unread,
Feb 13, 2023, 12:12:05 AM2/13/23
to openresty-en
Hello,

I'm running openresty on a server which has 16GB RAM but nginx worker eating 130GB  virtual memory.

My build
libmaxminddb_version="1.5.2"
pcre_version="8.44"
openssl_version="1.1.1j"
openssl_patch_version="1.1.1j"
openresty_version="1.19.3.1"


%Cpu(s):  0.8 us,  0.2 sy,  0.0 ni, 98.8 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 16292728 total,  1499480 free, 12218212 used,  2575036 buff/cache
KiB Swap:  8191996 total,  3968020 free,  4223976 used.  2826724 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND    
 9575 nginx  20   0  131.3g 848244  53072 S   2.7  5.2   1:19.72 nginx      
18912 root      20   0 1548356  41152   7844 S   2.3  0.3   4816:54 filebeat  
 9569 nginx  20   0  131.3g 845640  50472 S   1.3  5.2   1:00.61 nginx      
 9566 nginx  20   0  131.3g 840052  45020 S   1.0  5.2   0:20.07 nginx      
 9571 nginx  20   0  131.3g 841448  46280 S   1.0  5.2   0:32.63 nginx      
 9573 nginx  20   0  131.3g 843320  48152 S   1.0  5.2   0:37.47 nginx      
 9574 nginx  20   0  131.3g 844060  48892 S   1.0  5.2   0:40.68 nginx      
 9564 nginx  20   0  131.3g 834644  39612 S   0.7  5.1   0:12.78 nginx      
 9570 nginx  20   0  131.3g 839748  44596 S   0.7  5.2   0:25.13 nginx  
   
   12 root      20   0       0      0      0 I   0.3  0.0   1335:31 rcu_sched  
 9565 nginx  20   0  131.3g 834116  39080 S   0.3  5.1   0:14.27 nginx      
 9567 nginx  20   0  131.3g 839184  44148 S   0.3  5.2   0:22.49 nginx      
 9568 nginx  20   0  131.3g 839264  44232 S   0.3  5.2   0:25.51 nginx      
 9572 nginx  20   0  131.3g 844220  49048 S   0.3  5.2   0:53.67 nginx      
11783 nginx  20   0  140.4g 212460  49072 S   0.3  1.3  51:25.81 nginx
     
14224 root      20   0  159076   8952   7508 S   0.3  0.1   0:00.08 sshd      
14471 root      20   0       0      0      0 I   0.3  0.0   0:00.53 kworker/3:+

I see there a lot of 102400  rw-s- zero (deleted)
Exactly 2676 lines rw-s- zero (deleted) showing when I exec pmap -x 9572

# pmap -x 9572
9572:   nginx: worker process              
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000    4480    3536       0 r-x-- edge
0000000000a5f000     196     196       4 r---- edge
0000000000a90000     172     172     140 rw--- edge
0000000000abb000     164      48      48 rw---   [ anon ]
0000000002993000  347500  347496  347496 rw---   [ anon ]
0000000017cee000  410892  410680  410680 rw---   [ anon ]
00007f1c48e75000  102400       0       0 rw-s- zero (deleted)
00007f1c4f275000  102400       0       0 rw-s- zero (deleted)
00007f1c55675000  102400       0       0 rw-s- zero (deleted)
00007f1c5ba75000  102400       0       0 rw-s- zero (deleted)
00007f1c61e75000  102400       0       0 rw-s- zero (deleted)
00007f1c68275000  102400       0       0 rw-s- zero (deleted)
00007f1c6e675000  102400       0       0 rw-s- zero (deleted)
00007f1c74a75000  102400       0       0 rw-s- zero (deleted)
00007f1c7ae75000  102400       0       0 rw-s- zero (deleted)
00007f1c81275000  102400       0       0 rw-s- zero (deleted)
00007f1c87675000  102400       0       0 rw-s- zero (deleted)
00007f1c8da75000  102400       0       0 rw-s- zero (deleted)
00007f1c93e75000  102400       0       0 rw-s- zero (deleted)
00007f1c9a275000  102400       0       0 rw-s- zero (deleted)
00007f1ca0675000  102400       0       0 rw-s- zero (deleted)
00007f1ca6a75000  102400       0       0 rw-s- zero (deleted)
00007f1cace75000  102400       0       0 rw-s- zero (deleted)
00007f1cb3275000  102400       0       0 rw-s- zero (deleted)
00007f1cb9675000  102400       0       0 rw-s- zero (deleted)
00007f1cbfa75000  102400       0       0 rw-s- zero (deleted)
00007f1cc5e75000  102400       0       0 rw-s- zero (deleted)
00007f1ccc275000  102400       0       0 rw-s- zero (deleted)
00007f1cd2675000  102400       0       0 rw-s- zero (deleted)
00007f1cd8a75000  102400       0       0 rw-s- zero (deleted)
00007f1cdee75000  102400       0       0 rw-s- zero (deleted)
00007f1ce5275000  102400       0       0 rw-s- zero (deleted)
00007f1ceb675000  102400       0       0 rw-s- zero (deleted)
00007f1cf1a75000  102400       0       0 rw-s- zero (deleted)
00007f1cf7e75000  102400       0       0 rw-s- zero (deleted)
00007f1cfe275000  102400       0       0 rw-s- zero (deleted)
00007f1d04675000  102400       0       0 rw-s- zero (deleted)
00007f1d0aa75000  102400       0       0 rw-s- zero (deleted)
00007f1d10e75000  102400       0       0 rw-s- zero (deleted)
00007f1d17275000  102400       0       0 rw-s- zero (deleted)
00007f1d1d675000  102400       0       0 rw-s- zero (deleted)
00007f1d23a75000  102400       0       0 rw-s- zero (deleted)
00007f1d29e75000  102400       0       0 rw-s- zero (deleted)
00007f1d30275000  102400       0       0 rw-s- zero (deleted)
00007f1d36675000  102400       0       0 rw-s- zero (deleted)
00007f1d3ca75000  102400       0       0 rw-s- zero (deleted)
00007f1d42e75000  102400     116     116 rw-s- zero (deleted)
00007f1d49275000  102400       0       0 rw-s- zero (deleted)
00007f1d4f675000  102400       0       0 rw-s- zero (deleted)
00007f1d55a75000  102400       0       0 rw-s- zero (deleted)
00007f1d5be75000  102400       0       0 rw-s- zero (deleted)
00007f1d62275000  102400       0       0 rw-s- zero (deleted)
00007f1d68675000  102400       0       0 rw-s- zero (deleted)
00007f1d6ea75000  102400       0       0 rw-s- zero (deleted)
00007f1d74e75000  102400       0       0 rw-s- zero (deleted)
00007f1d7b275000  102400       0       0 rw-s- zero (deleted)
00007f1d81675000  102400       0       0 rw-s- zero (deleted)
00007f1d87a75000  102400       0       0 rw-s- zero (deleted)



thanks for reaching out

Junlong Li

unread,
Feb 13, 2023, 1:33:26 AM2/13/23
to openresty-en
you can try openresty xray to analyze this issue.

louis....@gmail.com

unread,
Feb 13, 2023, 4:08:44 AM2/13/23
to openresty-en
Thank you for your recommend but We did not purchase OpenResty XRay so cannot use this tool

louis....@gmail.com

unread,
Feb 13, 2023, 4:49:09 AM2/13/23
to openresty-en
I found this issue, it was due to using separate keys_zone for each vhost causing heavy usage of memory.
Reply all
Reply to author
Forward
0 new messages