NGINX lua crash on migrating to openssl 1.1.0

[rajesh singh]

Hi,

I am migrating to openssl 1.1.0 from openssl 1.0.2 on nginx.

I am using ngx-lua-module-0.9.8 with nginx version 1.4.4

I am seeing LUA crash as soon as i send http/https request.

Does ngx-lua-module-0.9.8 has any openssl 1.1.0 dependancy?? I did not see any compilation error.

In my understanding I should not see crash if i send http request as i have only SSL related changes.

I have done NGINX code changes to avoid compilation error and using appropriate openssl 1.1.0 APIs. Taken reference of latest NGINX code.

I am not sure why i am seeing crash on http request too. look like lua intrerface is broken.

I am using LuaJIT 2.1.0-beta3.

I am sharing logs snipped and traceback below:

===============================================================

2018/06/05 05:04:07 [debug] 7724#0: *4 lua rewrite handler, uri:"/webui/index.html" c:1                                                                         
2018/06/05 05:04:07 [debug] 7724#0: *4 lua post read for rewrite/access phases
2018/06/05 05:04:07 [debug] 7724#0: *4 lua creating new thread
2018/06/05 05:04:07 [debug] 7724#0: *4 lua reset ctx         
2018/06/05 05:04:07 [debug] 7724#0: *4 http cleanup add: 000000AAB64D0830
2018/06/05 05:04:07 [debug] 7724#0: *4 lua run thread, top:0 c:1        
2018/06/05 05:04:07 [notice] 7411#0: signal 16 (SIGUSR1) received, reopening logs
2018/06/05 05:04:07 [debug] 7411#0: wake up, sigio 0
2018/06/05 05:04:07 [notice] 7411#0: reopening logs
2018/06/05 05:04:07 [debug] 7411#0: reopen file "/tmp/rp/trace/ngx_error.log", old:34 new:33
2018/06/05 05:04:07 [debug] 7411#0: reopen file "/dev/null", old:5 new:34
2018/06/05 05:04:07 [debug] 7411#0: child: 0 7724 e:0 t:0 d:0 r:1 j:0
2018/06/05 05:04:07 [debug] 7411#0: sigsuspend
2018/06/05 05:04:12 [notice] 7411#0: signal 18 (SIGCHLD) received
2018/06/05 05:04:12 [alert] 7411#0: worker process 7724 exited on signal 10 (core dumped)
2018/06/05 05:04:12 [debug] 7411#0: wake up, sigio 0
2018/06/05 05:04:12 [debug] 7411#0: reap children
2018/06/05 05:04:12 [debug] 7411#0: child: 0 7724 e:0 t:1 d:0 r:1 j:0
2018/06/05 05:04:12 [debug] 7411#0: channel 5:7
2018/06/05 05:04:12 [notice] 7411#0: start worker process 11722
2018/06/05 05:04:12 [debug] 7411#0: sigsuspend
2018/06/05 05:04:12 [debug] 11722#0: aaa eventfd: 28
2018/06/05 05:04:12 [debug] 11722#0: malloc: 000000AAB646A418:8192
2018/06/05 05:04:12 [debug] 11722#0: malloc: 000000AAB646C458:143360
2018/06/05 05:04:12 [debug] 11722#0: malloc: 000000AAB648F498:53248

2018/06/05 05:04:12 [debug] 11722#0: malloc: 000000AAB649C4D8:53248
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:12 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:13 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:10 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:15 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:16 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:11 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:18 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:19 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:14 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:21 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:22 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:17 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:20 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:23 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: epoll add event: fd:7 op:1 ev:00000001
2018/06/05 05:04:12 [debug] 11722#0: setproctitle: "nginx: worker process"
2018/06/05 05:04:12 [debug] 11722#0: worker cycle
2018/06/05 05:04:12 [debug] 11722#0: epoll timer: -1
2018/06/05 05:04:12 [debug] 11722#0: epoll: fd:28 ev:0001 d:000000AAB5A49080
2018/06/05 05:04:12 [debug] 11722#0: aaa eventfd handler
2018/06/05 05:04:12 [debug] 11722#0: timer delta: 33
2018/06/05 05:04:12 [debug] 11722#0: posted events 0000000000000000
2018/06/05 05:04:12 [debug] 11722#0: worker cycle
2018/06/05 05:04:12 [debug] 11722#0: epoll timer: -1

========================================================

Backtrace:

========================================================

(gdb) bt                                                                 
#0  0x636f6f63501317ac in ?? ()                                          
#1  0x000000aaada35778 in ngx_http_lua_run_thread (L=<optimized out>, r=<optimized out>, ctx=<optimized out>, nrets=<optimized out>) at contrib/nginx/src/../lua-nginx-module-0.9.8/src/ngx_http_lua_util.c:1097                     
#2  0x000000aaada3dc74 in ngx_http_lua_rewrite_by_chunk (L=0xffed3ca378, r=0xaab7902950) at nginx/src/../lua-nginx-module-0.9.8/src/ngx_http_lua_rewriteby.c:303                                                             
#3  0x000000aaada3df18 in ngx_http_lua_rewrite_handler_file (r=0xaab7902950) at nginx/src/../lua-nginx-module-0.9.8/src/ngx_http_lua_rewriteby.c:221                                                                         
#4  0x000000aaada3d8f0 in ngx_http_lua_rewrite_handler (r=<optimized out>) at nginx/src/../lua-nginx-module-0.9.8/src/ngx_http_lua_rewriteby.c:162                                                                           
#5  0x000000aaadac45b8 in ngx_http_core_rewrite_phase (r=<optimized out>, ph=<optimized out>) at contrib/nginx/src/http/ngx_http_core_module.c:946                                                                                   
#6  0x000000aaadac4328 in ngx_http_core_run_phases (r=0xaab7902950) at nginx/src/http/ngx_http_core_module.c:892                                                                                                             
#7  ngx_http_handler (r=<optimized out>) at contrib/nginx/src/http/ngx_http_core_module.c:875                     
#8  0x000000aaadad2570 in ngx_http_process_request (r=<optimized out>) at nginx/src/http/ngx_http_request.c:1854                                                                                                             
#9  0x000000aaadad532c in ngx_http_process_request_headers (rev=<optimized out>) at nginx/src/http/ngx_http_request.c:1285                                                                                                   
#10 0x000000aaadad4c24 in ngx_http_process_request_line (rev=<optimized out>) at nginx/src/http/ngx_http_request.c:963                                                                                                       
#11 0x000000aaadacf1b8 in ngx_http_wait_request_handler (rev=<optimized out>) at nginx/src/http/ngx_http_request.c:486                                                                                                       
#12 0x000000aaadabc090 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at nginx/src/event/modules/ngx_epoll_module.c:718                                                   
#13 0x000000aaada9f9f4 in ngx_process_events_and_timers (cycle=0xaab769ced0) at nginx/src/event/ngx_event.c:249                                                                                                              
#14 0x000000aaadab91fc in ngx_worker_process_cycle (cycle=0xaab769ced0, data=<optimized out>) at contrib/nginx/src/os/unix/ngx_process_cycle.c:838                                                                                   
#15 0x000000aaadab5938 in ngx_spawn_process (cycle=0xaab769ced0, proc=<optimized out>, data=<optimized out>, name=<optimized out>, respawn=<optimized out>) at nginx/src/os/unix/ngx_process.c:198                           
#16 0x000000aaadab7c3c in ngx_reap_children (cycle=<optimized out>) at nginx/src/os/unix/ngx_process_cycle.c:635                                                                                                             
#17 ngx_master_process_cycle (cycle=<optimized out>) at nginx/src/os/unix/ngx_process_cycle.c:196

Thanks,

Rajesh Singh

[rajesh singh]

I found the root cause of the issue.

Issue is there because i am using luacrypto-0.3.2 in my lua file.

since luacrypto-0.3.2 does not support openssl 1.1.0 its crashing as soon as lua file is getting loaded.

I tried searching for luacrypto that support openssl 1.1.0 but i could not find.

Is there any openssl 1.1.0  wrapper available for lua??

Please suggest.