Nginx failing after many requests
39 views
Skip to first unread message
Matthew Kenigsberg
Feb 3, 2021, 11:47:13 AM2/3/21
to OSv Development
Hi,
I'm trying to benchmark nginx with weighttp, but it appears to crash after too many requests are sent. As soon as requests start failing, it seems all following requests will fail. Any ideas why?
$ curl ip
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed, 0 errored
...
$ curl 129.114.109.194
curl: (52) Empty reply from server
$ sleep 60
$ curl 129.114.109.194
curl: (52) Empty reply from server
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed, 0 errored
...
$ curl 129.114.109.194
curl: (52) Empty reply from server
$ sleep 60
$ curl 129.114.109.194
curl: (52) Empty reply from server
Not sure if this is related to either of:
Thanks,
Matthew
Waldek Kozaczuk
Feb 8, 2021, 11:00:29 AM2/8/21
to OSv Development
Hi,
On Wednesday, February 3, 2021 at 11:47:13 AM UTC-5 Matthew Kenigsberg wrote:
Hi,I'm trying to benchmark nginx with weighttp, but it appears to crash after too many requests are sent. As soon as requests start failing, it seems all following requests will fail. Any ideas why?
Not really but I think I may have seen similar behavior with nginx.
$ curl ip
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed, 0 errored
...
$ curl 129.114.109.194
curl: (52) Empty reply from server
$ sleep 60
$ curl 129.114.109.194
curl: (52) Empty reply from serverNot sure if this is related to either of:
Maybe.
It would be best if you could connect with gdb to OSv when it crashes/hangs and send us a thread list and ideally stack traces of interesting threads (most likely those part of TCP stack). I suggest you build and run ROFS image as it will have fewer threads. See https://github.com/cloudius-systems/osv/wiki/Debugging-OSv.
Does this crash happen with single cpu or with many?
Thanks,Matthew
Fotis Xenakis
Feb 10, 2021, 4:44:56 AM2/10/21
to OSv Development
On Monday, February 8, 2021 at 6:00:29 PM UTC+2 jwkoz...@gmail.com wrote:
Hi,On Wednesday, February 3, 2021 at 11:47:13 AM UTC-5 Matthew Kenigsberg wrote:Hi,I'm trying to benchmark nginx with weighttp, but it appears to crash after too many requests are sent. As soon as requests start failing, it seems all following requests will fail. Any ideas why?Not really but I think I may have seen similar behavior with nginx.
I have recently tried a very similar scenario: nginx on OSv, stressed with vegeta on the host. While I didn't face the same issue, there was a problem when using NFS for the underlying files (the benchmark target was the fs, so I was using some static files, not the default nginx response page). I didn't get to the root of it at the time, but is seemed like a deadlock, in OSv.
Out of curiosity:
- Have you tried to e.g. use lighttpd instead of nginx?
- Have you tried to e.g. use lighttpd instead of nginx?
- What file system are you running (might be irrelevant)?
- Does OSv seem to "hang"? Is it accepting connections or not?
$ curl ip
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored
...
$ weighttp -t 1 -n 1000 -c 1 http://ip
...
requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed, 0 errored
...
$ curl 129.114.109.194
curl: (52) Empty reply from server
$ sleep 60
$ curl 129.114.109.194
curl: (52) Empty reply from serverNot sure if this is related to either of:Maybe.It would be best if you could connect with gdb to OSv when it crashes/hangs and send us a thread list and ideally stack traces of interesting threads (most likely those part of TCP stack). I suggest you build and run ROFS image as it will have fewer threads. See https://github.com/cloudius-systems/osv/wiki/Debugging-OSv.
+1
The ability to debug with GDB has proved invaluable!
Matthew Kenigsberg
Feb 10, 2021, 12:25:51 PM2/10/21
to OSv Development
Hi,
Thanks for the response!
On Wednesday, February 10, 2021 at 3:44:56 AM UTC-6 Fotis Xenakis wrote:
On Monday, February 8, 2021 at 6:00:29 PM UTC+2 jwkoz...@gmail.com wrote:Hi,On Wednesday, February 3, 2021 at 11:47:13 AM UTC-5 Matthew Kenigsberg wrote:Hi,I'm trying to benchmark nginx with weighttp, but it appears to crash after too many requests are sent. As soon as requests start failing, it seems all following requests will fail. Any ideas why?Not really but I think I may have seen similar behavior with nginx.I have recently tried a very similar scenario: nginx on OSv, stressed with vegeta on the host. While I didn't face the same issue, there was a problem when using NFS for the underlying files (the benchmark target was the fs, so I was using some static files, not the default nginx response page). I didn't get to the root of it at the time, but is seemed like a deadlock, in OSv.Out of curiosity:
- Have you tried to e.g. use lighttpd instead of nginx?
- What file system are you running (might be irrelevant)?
The default which I believe is zfs
- Does OSv seem to "hang"? Is it accepting connections or not?
Doesn't seem to hang, but maybe not accepting connections? A simple curl localhost gives
Tried to use rofs but it's not working immediately. It was still
trying to open error log files in read mode even after I changed
nginx.conf. If that's important I could work on that more.
+1The ability to debug with GDB has proved invaluable!Does this crash happen with single cpu or with many?
I'm running on a server with many CPUs, I think up to 64 threads.
Here's some debugging info from immediately before
weighttp gets back 1000 failed requests (weighttp gets responses when I
finish virtio::net::receiver). Debugging this kind of stuff is definitely outside of my comfort zone, so if there's more specific things I should be doing let me know!
(gdb) bt
#0 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000) at drivers/virtio-net.cc:539
#1 0x000000004036d8ac in virtio::net::<lambda()>::operator() (__closure=<optimized out>)
at drivers/virtio-net.cc:242
#2 std::_Function_handler<void(), virtio::net::net(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300
#3 0x00000000403f9e18 in sched::thread_main_c (t=0xffff80007ff54040) at arch/x64/arch-switch.hh:325
#4 0x000000004038cf13 in thread_main () at arch/x64/entry.S:113
#0 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000) at drivers/virtio-net.cc:539
#1 0x000000004036d8ac in virtio::net::<lambda()>::operator() (__closure=<optimized out>)
at drivers/virtio-net.cc:242
#2 std::_Function_handler<void(), virtio::net::net(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300
#3 0x00000000403f9e18 in sched::thread_main_c (t=0xffff80007ff54040) at arch/x64/arch-switch.hh:325
#4 0x000000004038cf13 in thread_main () at arch/x64/entry.S:113
(gdb) info threads
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000)
at drivers/virtio-net.cc:539
2 Thread 1.2 (CPU#1 [running]) std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed,
this=0xffff8000000567e0)
at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/atomic_base.h:413
3 Thread 1.3 (CPU#2 [halted ]) sched::cpu::do_idle (this=0xffff800000088040) at core/sched.cc:405
4 Thread 1.4 (CPU#3 [running]) sched::cpu::do_idle (this=0xffff80000008c040)
at include/boost/intrusive/bstree.hpp:945
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000)
at drivers/virtio-net.cc:539
2 Thread 1.2 (CPU#1 [running]) std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed,
this=0xffff8000000567e0)
at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/atomic_base.h:413
3 Thread 1.3 (CPU#2 [halted ]) sched::cpu::do_idle (this=0xffff800000088040) at core/sched.cc:405
4 Thread 1.4 (CPU#3 [running]) sched::cpu::do_idle (this=0xffff80000008c040)
at include/boost/intrusive/bstree.hpp:945
(gdb) osv info threads
1 (0xffff800000017040) reclaimer cpu0 status::waiting
condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
vruntime 5.55762e-24
2 (0xffff800000091040) kvm_wall_clock_ cpu0 status::waiting std::_Function_handler<void(), kvmclock::kvmclock()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 47.8555
3 (0xffff800000173040) page-access-sca cpu1 status::waiting pagecache::access_scanner::run() at core/pagecache.cc:695 vruntime 41259.8
4 (0xffff80000018c040) page_pool_l2 cpu0 status::waiting memory::page_pool::l2::fill_thread() at core/mempool.cc:1437 vruntime 3.18475e-05
5 (0xffff8000001a7040) itimer-real cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0
6 (0xffff8000001bf040) itimer-virt cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0
7 (0xffff800000a22040) balancer0 cpu0 status::waiting ?? at arch/x64/entry.S:113 vruntime 260.086
8 (0xffff800000a39040) rcu0 cpu0 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 1.0958
9 (0xffff800000a50040) page_pool_l1_0 cpu0 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 0.0484772
10 (0xffff800000a67040) percpu0 cpu0 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 5.55762e-24
11 (0xffff800000a7e040) async_worker0 cpu0 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 280.215
12 (0xffff800000a95040) idle0 cpu0 status::queued std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
13 (0xffff800000aac040) idle1 cpu1 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
14 (0xffff800000adf040) idle2 cpu2 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
15 (0xffff800000b12040) idle3 cpu3 status::running sched::cpu::do_idle() at include/boost/intrusive/bstree.hpp:945 vruntime inf
16 (0xffff800000b46040) rcu2 cpu2 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0608673
17 (0xffff800000b5f040) page_pool_l1_2 cpu2 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1346 vruntime 1.11152e-23
18 (0xffff800000b77040) percpu2 cpu2 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11023e-23
19 (0xffff800000b8f040) async_worker2 cpu2 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 1.11023e-23
20 (0xffff800000ba7040) rcu1 cpu1 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0379105
21 (0xffff800000bc1040) page_pool_l1_1 cpu1 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 1.19201e-10
22 (0xffff800000bd9040) percpu1 cpu1 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11152e-23
23 (0xffff800000bf1040) async_worker1 cpu1 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 3.38554e-05
24 (0xffff800000c09040) >init cpu0 status::waiting std::_Function_handler<void (), osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 5.86781e-24
25 (0xffff800000c21040) rcu3 cpu3 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0317421
26 (0xffff800000c3b040) page_pool_l1_3 cpu3 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 2.42855e-22
--Type <RET> for more, q to quit, c to continue without paging--c
27 (0xffff800000cf3040) percpu3 cpu3 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 6.44942e-24
28 (0xffff800000e4a040) async_worker3 cpu3 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 5.89366e-24
29 (0xffff800001201040) >init cpu0 status::waiting std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 6.15991e-23
30 (0xffff800001218040) thread taskq cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.04819e-23
31 (0xffff800001223040) callout cpu1 status::waiting _callout_thread() at bsd/porting/callout.cc:138 vruntime 196.727
32 (0xffff800001242040) system_taskq_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.63867e-23
33 (0xffff80000124d040) system_taskq_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57589e-23
34 (0xffff800001258040) system_taskq_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.5578e-23
35 (0xffff800001263040) system_taskq_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57848e-23
36 (0xffff80000126e040) system_taskq_4 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.56555e-23
37 (0xffff800001279040) system_taskq_5 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24
38 (0xffff800001284040) system_taskq_6 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24
39 (0xffff80000128f040) system_taskq_7 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24
40 (0xffff80007fe41040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 52.043
41 (0xffff80007fe4c040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 18.793
42 (0xffff80007fe9d040) netisr cpu0 status::waiting netisr_osv_thread_wrapper(void (*)(void*), void*) at bsd/sys/net/netisr1.cc:26 vruntime 1.75647e-23
43 (0xffff80007ff3a040) >init cpu0 status::waiting sched::thread::join() at core/sched.cc:1313 vruntime 3.59201e-20
joining on <optimized out>
44 (0xffff80007ff54040) virtio-net-rx cpu0 status::running virtio::net::receiver() at drivers/virtio-net.cc:539 vruntime 1.72028
45 (0xffff80007ff7c040) virtio-tx-0 cpu0 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 8.71137e-29
46 (0xffff80007ffa4040) virtio-tx-1 cpu1 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 2.22299e-29
47 (0xffff80007ffcc040) virtio-tx-2 cpu2 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.46063e-29
48 (0xffff80007ffd2040) virtio-tx-3 cpu3 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.43474e-29
49 (0xffff80000154e040) virtio-blk cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:158 vruntime 1.34348e-06
50 (0xffff800001565040) virtio-rng cpu1 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 1446.79
51 (0xffff80000157f040) isa-serial-inpu cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.67375e-23
52 (0xffff800001596040) kbd-input cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.47859e-23
53 (0xffff8000015d6040) rand_harvestq cpu1 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5295.02
136 (0xffff8000015e4040) zio_null_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.72184e-07
137 (0xffff8000015ef040) zio_null_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.70787e-08
138 (0xffff8000015fa040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.63833e-14
139 (0xffff800001605040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.4607e-14
140 (0xffff800001610040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.28306e-14
141 (0xffff80000161b040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14
142 (0xffff800001626040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.20792e-13
143 (0xffff800001631040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.96046e-08
144 (0xffff80000163c040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.81597e-14
145 (0xffff800001647040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00646973
146 (0xffff800001652040) zio_read_intr_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.21541e-08
147 (0xffff80000165d040) zio_read_intr_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98549e-08
148 (0xffff800001668040) zio_read_intr_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.72982e-08
149 (0xffff800001673040) zio_read_intr_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.33299e-08
150 (0xffff80000167e040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17907e-06
151 (0xffff800001689040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19583e-06
152 (0xffff800001694040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.03512e-05
153 (0xffff80000169f040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.04239e-06
154 (0xffff8000016aa040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
155 (0xffff8000016b5040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24
156 (0xffff8000016c0040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24
157 (0xffff8000016cb040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
158 (0xffff8000016d6040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
159 (0xffff8000016e1040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.3276e-06
160 (0xffff8000016ec040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.87896e-07
161 (0xffff8000016f7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.16532e-07
162 (0xffff800001702040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.92901e-07
163 (0xffff80000170d040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.26311e-07
164 (0xffff800001718040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.29454e-07
165 (0xffff800001723040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.82545e-08
166 (0xffff80000172e040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.86268e-07
167 (0xffff800001739040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.74975e-08
168 (0xffff800001744040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.46804e-07
169 (0xffff80000174f040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54371e-08
170 (0xffff80000175a040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.01794e-06
171 (0xffff800001765040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.3609e-07
172 (0xffff800001770040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17581e-07
173 (0xffff80000177b040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.16184e-07
174 (0xffff800001786040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.9139e-08
175 (0xffff800001791040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24
176 (0xffff80000179c040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.89219e-24
177 (0xffff8000017a7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24
178 (0xffff8000017b2040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24
179 (0xffff8000017bd040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
180 (0xffff8000017c8040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.19564e-08
181 (0xffff8000017d3040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.09548e-08
182 (0xffff8000017de040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
183 (0xffff8000017e9040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.90921e-08
184 (0xffff8000017f4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.27993e-24
185 (0xffff8000017ff040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.37956e-08
186 (0xffff80000180a040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.00234e-08
187 (0xffff800001815040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14
188 (0xffff800001820040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.76951e-08
189 (0xffff80000182b040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.68575e-08
190 (0xffff800001836040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0022583
191 (0xffff800001841040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98258e-08
192 (0xffff80000184c040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209
193 (0xffff800001857040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.28408e-08
194 (0xffff800001862040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209
195 (0xffff80000186d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
196 (0xffff800001878040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
197 (0xffff800001883040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.18861e-08
198 (0xffff80000188e040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.21072e-07
199 (0xffff800001899040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.28643e-08
200 (0xffff8000018a4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.45661e-13
201 (0xffff8000018af040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19744e-14
202 (0xffff8000018ba040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.10862e-14
203 (0xffff8000018c5040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00738525
204 (0xffff8000018d0040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.40984e-07
205 (0xffff8000018db040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.77186e-08
206 (0xffff8000018e6040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0148926
207 (0xffff8000018f1040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00628662
208 (0xffff8000018fc040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00830078
209 (0xffff800001907040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.70318e-08
210 (0xffff800001912040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0640259
211 (0xffff80000191d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0196533
212 (0xffff800001928040) zio_free_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24
213 (0xffff800001933040) zio_claim_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24
214 (0xffff80000193e040) zio_claim_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.95682e-24
215 (0xffff800001949040) zio_ioctl_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24
216 (0xffff800001954040) zio_ioctl_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.14907e-08
217 (0xffff800001be9040) zfs_vn_rele_tas cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.79692e-24
218 (0xffff800001c83040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 3.93018e-07
219 (0xffff800001c8e040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 9.81754e-06
220 (0xffff800001cc6040) zil_clean cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.50479
221 (0xffff800001cda040) dhcp cpu0 status::waiting dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:828 vruntime 5.60518e-22
222 (0xffff800001d5c040) /nginx.so cpu0 status::waiting epoll_file::wait(epoll_event*, int, int) at core/epoll.cc:142 vruntime 93.4503
Number of threads: 140
2 (0xffff800000091040) kvm_wall_clock_ cpu0 status::waiting std::_Function_handler<void(), kvmclock::kvmclock()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 47.8555
3 (0xffff800000173040) page-access-sca cpu1 status::waiting pagecache::access_scanner::run() at core/pagecache.cc:695 vruntime 41259.8
4 (0xffff80000018c040) page_pool_l2 cpu0 status::waiting memory::page_pool::l2::fill_thread() at core/mempool.cc:1437 vruntime 3.18475e-05
5 (0xffff8000001a7040) itimer-real cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0
6 (0xffff8000001bf040) itimer-virt cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0
7 (0xffff800000a22040) balancer0 cpu0 status::waiting ?? at arch/x64/entry.S:113 vruntime 260.086
8 (0xffff800000a39040) rcu0 cpu0 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 1.0958
9 (0xffff800000a50040) page_pool_l1_0 cpu0 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 0.0484772
10 (0xffff800000a67040) percpu0 cpu0 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 5.55762e-24
11 (0xffff800000a7e040) async_worker0 cpu0 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 280.215
12 (0xffff800000a95040) idle0 cpu0 status::queued std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
13 (0xffff800000aac040) idle1 cpu1 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
14 (0xffff800000adf040) idle2 cpu2 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf
15 (0xffff800000b12040) idle3 cpu3 status::running sched::cpu::do_idle() at include/boost/intrusive/bstree.hpp:945 vruntime inf
16 (0xffff800000b46040) rcu2 cpu2 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0608673
17 (0xffff800000b5f040) page_pool_l1_2 cpu2 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1346 vruntime 1.11152e-23
18 (0xffff800000b77040) percpu2 cpu2 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11023e-23
19 (0xffff800000b8f040) async_worker2 cpu2 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 1.11023e-23
20 (0xffff800000ba7040) rcu1 cpu1 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0379105
21 (0xffff800000bc1040) page_pool_l1_1 cpu1 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 1.19201e-10
22 (0xffff800000bd9040) percpu1 cpu1 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11152e-23
23 (0xffff800000bf1040) async_worker1 cpu1 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 3.38554e-05
24 (0xffff800000c09040) >init cpu0 status::waiting std::_Function_handler<void (), osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 5.86781e-24
25 (0xffff800000c21040) rcu3 cpu3 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0317421
26 (0xffff800000c3b040) page_pool_l1_3 cpu3 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 2.42855e-22
--Type <RET> for more, q to quit, c to continue without paging--c
27 (0xffff800000cf3040) percpu3 cpu3 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 6.44942e-24
28 (0xffff800000e4a040) async_worker3 cpu3 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 5.89366e-24
29 (0xffff800001201040) >init cpu0 status::waiting std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 6.15991e-23
30 (0xffff800001218040) thread taskq cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.04819e-23
31 (0xffff800001223040) callout cpu1 status::waiting _callout_thread() at bsd/porting/callout.cc:138 vruntime 196.727
32 (0xffff800001242040) system_taskq_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.63867e-23
33 (0xffff80000124d040) system_taskq_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57589e-23
34 (0xffff800001258040) system_taskq_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.5578e-23
35 (0xffff800001263040) system_taskq_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57848e-23
36 (0xffff80000126e040) system_taskq_4 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.56555e-23
37 (0xffff800001279040) system_taskq_5 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24
38 (0xffff800001284040) system_taskq_6 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24
39 (0xffff80000128f040) system_taskq_7 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24
40 (0xffff80007fe41040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 52.043
41 (0xffff80007fe4c040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 18.793
42 (0xffff80007fe9d040) netisr cpu0 status::waiting netisr_osv_thread_wrapper(void (*)(void*), void*) at bsd/sys/net/netisr1.cc:26 vruntime 1.75647e-23
43 (0xffff80007ff3a040) >init cpu0 status::waiting sched::thread::join() at core/sched.cc:1313 vruntime 3.59201e-20
joining on <optimized out>
44 (0xffff80007ff54040) virtio-net-rx cpu0 status::running virtio::net::receiver() at drivers/virtio-net.cc:539 vruntime 1.72028
45 (0xffff80007ff7c040) virtio-tx-0 cpu0 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 8.71137e-29
46 (0xffff80007ffa4040) virtio-tx-1 cpu1 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 2.22299e-29
47 (0xffff80007ffcc040) virtio-tx-2 cpu2 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.46063e-29
48 (0xffff80007ffd2040) virtio-tx-3 cpu3 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.43474e-29
49 (0xffff80000154e040) virtio-blk cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:158 vruntime 1.34348e-06
50 (0xffff800001565040) virtio-rng cpu1 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 1446.79
51 (0xffff80000157f040) isa-serial-inpu cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.67375e-23
52 (0xffff800001596040) kbd-input cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.47859e-23
53 (0xffff8000015d6040) rand_harvestq cpu1 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5295.02
136 (0xffff8000015e4040) zio_null_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.72184e-07
137 (0xffff8000015ef040) zio_null_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.70787e-08
138 (0xffff8000015fa040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.63833e-14
139 (0xffff800001605040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.4607e-14
140 (0xffff800001610040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.28306e-14
141 (0xffff80000161b040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14
142 (0xffff800001626040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.20792e-13
143 (0xffff800001631040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.96046e-08
144 (0xffff80000163c040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.81597e-14
145 (0xffff800001647040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00646973
146 (0xffff800001652040) zio_read_intr_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.21541e-08
147 (0xffff80000165d040) zio_read_intr_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98549e-08
148 (0xffff800001668040) zio_read_intr_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.72982e-08
149 (0xffff800001673040) zio_read_intr_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.33299e-08
150 (0xffff80000167e040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17907e-06
151 (0xffff800001689040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19583e-06
152 (0xffff800001694040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.03512e-05
153 (0xffff80000169f040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.04239e-06
154 (0xffff8000016aa040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
155 (0xffff8000016b5040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24
156 (0xffff8000016c0040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24
157 (0xffff8000016cb040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
158 (0xffff8000016d6040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
159 (0xffff8000016e1040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.3276e-06
160 (0xffff8000016ec040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.87896e-07
161 (0xffff8000016f7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.16532e-07
162 (0xffff800001702040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.92901e-07
163 (0xffff80000170d040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.26311e-07
164 (0xffff800001718040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.29454e-07
165 (0xffff800001723040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.82545e-08
166 (0xffff80000172e040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.86268e-07
167 (0xffff800001739040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.74975e-08
168 (0xffff800001744040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.46804e-07
169 (0xffff80000174f040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54371e-08
170 (0xffff80000175a040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.01794e-06
171 (0xffff800001765040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.3609e-07
172 (0xffff800001770040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17581e-07
173 (0xffff80000177b040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.16184e-07
174 (0xffff800001786040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.9139e-08
175 (0xffff800001791040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24
176 (0xffff80000179c040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.89219e-24
177 (0xffff8000017a7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24
178 (0xffff8000017b2040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24
179 (0xffff8000017bd040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24
180 (0xffff8000017c8040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.19564e-08
181 (0xffff8000017d3040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.09548e-08
182 (0xffff8000017de040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
183 (0xffff8000017e9040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.90921e-08
184 (0xffff8000017f4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.27993e-24
185 (0xffff8000017ff040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.37956e-08
186 (0xffff80000180a040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.00234e-08
187 (0xffff800001815040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14
188 (0xffff800001820040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.76951e-08
189 (0xffff80000182b040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.68575e-08
190 (0xffff800001836040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0022583
191 (0xffff800001841040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98258e-08
192 (0xffff80000184c040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209
193 (0xffff800001857040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.28408e-08
194 (0xffff800001862040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209
195 (0xffff80000186d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
196 (0xffff800001878040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08
197 (0xffff800001883040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.18861e-08
198 (0xffff80000188e040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.21072e-07
199 (0xffff800001899040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.28643e-08
200 (0xffff8000018a4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.45661e-13
201 (0xffff8000018af040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19744e-14
202 (0xffff8000018ba040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.10862e-14
203 (0xffff8000018c5040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00738525
204 (0xffff8000018d0040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.40984e-07
205 (0xffff8000018db040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.77186e-08
206 (0xffff8000018e6040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0148926
207 (0xffff8000018f1040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00628662
208 (0xffff8000018fc040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00830078
209 (0xffff800001907040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.70318e-08
210 (0xffff800001912040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0640259
211 (0xffff80000191d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0196533
212 (0xffff800001928040) zio_free_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24
213 (0xffff800001933040) zio_claim_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24
214 (0xffff80000193e040) zio_claim_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.95682e-24
215 (0xffff800001949040) zio_ioctl_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24
216 (0xffff800001954040) zio_ioctl_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.14907e-08
217 (0xffff800001be9040) zfs_vn_rele_tas cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.79692e-24
218 (0xffff800001c83040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 3.93018e-07
219 (0xffff800001c8e040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 9.81754e-06
220 (0xffff800001cc6040) zil_clean cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.50479
221 (0xffff800001cda040) dhcp cpu0 status::waiting dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:828 vruntime 5.60518e-22
222 (0xffff800001d5c040) /nginx.so cpu0 status::waiting epoll_file::wait(epoll_event*, int, int) at core/epoll.cc:142 vruntime 93.4503
Number of threads: 140
Thanks,Matthew
Reply all
Reply to author
Forward
0 new messages