Potential deadlock when running latest Java 12
19 views
Skip to first unread message
Waldek Kozaczuk
Jul 11, 2019, 9:01:59 AM7/11/19
to OSv Development
I tried to run Java 12 on OSv and simple apps like java-example and java-httpserver work. But more complicated like spring-boot-example or jetty seem to hang early on.
Here is the output:
./scripts/build image=openjdk12-zulu,jetty fs=rofs
./scripts/run.py -V -c 1OSv v0.53.0-53-ge9ef51b21 CPUs detectedFirmware vendor: SeaBIOSbsd: initializing - doneVFS: mounting ramfs at /VFS: mounting devfs at /devnet: initializing - donevga: Add VGA device instanceeth0: ethernet address: 52:54:00:12:34:56virtio-blk: Add blk device instances 0 as vblk0, devsize=75520512random: virtio-rng registered as a source.random: intel drng, rdrand registered as a source.random: <Software, Yarrow> initializedVFS: unmounting /devVFS: mounting rofs at /rofsVFS: mounting devfs at /devVFS: mounting procfs at /procVFS: mounting ramfs at /tmp[I/27 dhcp]: Broadcasting DHCPDISCOVER message with xid: [662656670][I/27 dhcp]: Waiting for IP...[I/35 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15[I/35 dhcp]: Broadcasting DHCPREQUEST message with xid: [662656670] to SELECT offered IP: 192.168.122.15[I/35 dhcp]: Received DHCPACK message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15[I/35 dhcp]: Server acknowledged IP 192.168.122.15 for interface eth0 with time to lease in seconds: 86400eth0: 192.168.122.15[I/35 dhcp]: Configuring eth0: ip 192.168.122.15 subnet mask 255.255.255.0 gateway 192.168.122.1 MTU 1500Booted up in 152.21 msjava.so: Starting JVM app using: io/osv/nonisolated/RunNonIsolatedJvmAppjava.so: Setting Java system classloader to NonIsolatingOsvSystemClassLoaderrandom: device unblocked.
and gdb threads and the initial thread stacktrace I get after connecting with gdb:
(gdb) bt#0 0x00000000403e1acc in lockfree::mutex::lock (this=this@entry=0xffffa0000431b410) at /usr/include/c++/8/bits/atomic_base.h:512#1 0x00000000403e36a0 in std::lock_guard<lockfree::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /usr/include/c++/8/bits/std_mutex.h:161#2 lock_guard_for_with_lock<lockfree::mutex>::lock_guard_for_with_lock (lock=..., this=<synthetic pointer>) at include/osv/mutex.h:89#3 semaphore::trywait (this=0xffffa0000431b400, units=units@entry=1) at core/semaphore.cc:64#4 0x00000000404656c1 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#5 0x00001000012feb22 in PosixSemaphore::trywait() ()#6 0x0000100000e19de4 in VM_HandshakeAllThreads::doit() ()#7 0x0000100001575fd7 in VM_Operation::evaluate() ()#8 0x000010000157bdb6 in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.67] ()#9 0x000010000157c50a in VMThread::loop() ()#10 0x000010000157c6e3 in VMThread::run() ()#11 0x000010000150479f in Thread::call_run() ()#12 0x000010000122cbae in thread_native_entry(Thread*) ()#13 0x000000004045edb6 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00002ca9d00) at libc/pthread.cc:114#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#15 0x0000000040401117 in sched::thread_main_c (t=0xffff800003175040) at arch/x64/arch-switch.hh:271#16 0x00000000403a7083 in thread_main () at arch/x64/entry.S:113
1 (0xffff80000000c040) reclaimer cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 4.3944e-25 2 (0xffff800000050040) page-access-sca cpu0 status::waiting pagecache::access_scanner::run() at core/pagecache.cc:593 vruntime 127576 timers: 63.727309402 3 (0xffff800000068040) page_pool_l2 cpu0 status::waiting memory::page_pool::l2::fill_thread() at core/mempool.cc:1346 vruntime 6.05067e-21 4 (0xffff800000085040) itimer-real cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0 5 (0xffff80000008b040) itimer-virt cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0 6 (0xffff80000013f040) balancer0 cpu0 status::waiting ?? at arch/x64/entry.S:113 vruntime 1414.8 timers: 63.807163741 7 (0xffff800000156040) rcu0 cpu0 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 5.29913e-25 8 (0xffff80000016d040) page_pool_l1_0 cpu0 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1258 vruntime 5.84375e-21 9 (0xffff800000184040) percpu0 cpu0 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 4.65289e-25 10 (0xffff80000019b040) async_worker0 cpu0 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 3.87741e-25 11 (0xffff8000001b2040) idle0 cpu0 status::queued std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /usr/include/c++/8/bits/std_function.h:297 vruntime inf 12 (0xffff800000afa040) >init cpu0 status::waiting std::_Function_handler<void (), osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/8/bits/std_function.h:297 vruntime 5.95829e-24 13 (0xffff800000b11040) >init cpu0 status::waiting std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) at /usr/include/c++/8/bits/std_function.h:297 vruntime 5.88074e-24 14 (0xffff800000b28040) thread taskq cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.04876e-24 15 (0xffff800000b33040) callout cpu0 status::waiting _callout_thread() at bsd/porting/callout.cc:138 vruntime 936.234 timers: 63.807143836 16 (0xffff800000b4c040) system_taskq_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.00998e-24 17 (0xffff800000b57040) system_taskq_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24 18 (0xffff800000b62040) system_taskq_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24 19 (0xffff800000b6d040) system_taskq_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24 20 (0xffff800000b78040) system_taskq_4 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.85489e-24 21 (0xffff800000b83040) system_taskq_5 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bs
22 (0xffff800000b8e040) system_taskq_6 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.84196e-24 23 (0xffff800000b99040) system_taskq_7 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24 24 (0xffff800000fe8040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 22.707 timers: 64.002825055 25 (0xffff800000ff3040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 2.35645 timers: 64.002826619 26 (0xffff80000101a040) netisr cpu0 status::waiting netisr_osv_thread_wrapper(void (*)(void*), void*) at bsd/sys/net/netisr1.cc:26 vruntime 1.68797e-23 27 (0xffff800001181040) >init cpu0 status::waiting sched::thread::join() at core/sched.cc:1311 vruntime 6.39851e-20 joining on <optimized out> 28 (0xffff80000119b040) virtio-net-rx cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:160 vruntime 2.39361e-28 29 (0xffff8000011c3040) virtio-tx-0 cpu0 status::waiting sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}>(sched::thread::dummy_lock&, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}) at /usr/include/c++/8/bits/atomic_base.h:390 vruntime 9.29284e-29 30 (0xffff8000012dd040) virtio-blk cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:160 vruntime 4.61277e-21 31 (0xffff8000012f4040) virtio-rng cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:160 vruntime 0.941162 32 (0xffff80000130e040) isa-serial-inpu cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.38294e-23 33 (0xffff800001325040) kbd-input cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.40233e-23 34 (0xffff800001347040) rand_harvestq cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4507 timers: 63.730925282 35 (0xffff800001367040) dhcp cpu0 status::waiting dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:828 vruntime 7.61456e-21 36 (0xffff8000014b3040) java.so cpu0 status::waiting sched::thread::join() at core/sched.cc:1311 vruntime 1.39354e-22 joining on <optimized out> 37 (0xffff8000014b9040) >java.so cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 4.15911e-19 38 (0xffff800003175040) VM Thread cpu0 status::running lockfree::mutex::lock() at /usr/include/c++/8/bits/atomic_base.h:512 vruntime 1.12722e+07 39 (0xffff800003480040) Reference Handl cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 2.93597e-22 40 (0xffff80000358f040) Finalizer cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 2.90547e-22 41 (0xffff80000372e040) Signal Dispatch cpu0 status::waiting semaphore::wait(unsigned int, sched::timer*) at core/semaphore.cc:46 vruntime 1.49254e-22 42 (0xffff800003834040) C2 CompilerThre cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 6.14524e-05 timers: 65.498438192 43 (0xffff800003a55040) C1 CompilerThre cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 6.00219e-05 timers: 65.501513376 44 (0xffff800003b7f040) Sweeper thread cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 1.0315e-20 45 (0xffff80000425f040) Common-Cleaner cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 0.000185609 timers: 120.420613278 46 (0xffff8000045cd040) Service Thread cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 3.28391e-22 47 (0xffff8000046dc040) VM Periodic Tas cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 2725.28 timers: 63.727339031 48 (0xffff800004a5b040) Thread-1 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 4.80877e-20Number of threads: 48This smells like a deadlock, somewhere early in JIT compiler (C1/C2 compiler threads). Any suggestions on how to debug it further? Enable some tracepoints?
The same app (jetty in this case) works fine with previous Java 11.
Waldek
Waldek Kozaczuk
Jul 15, 2019, 5:29:08 PM7/15/19
to OSv Development
Here is the full thread dump:
(gdb) osv thread apply all btthread 0xffff80000000c040 reclaimer
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000068040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000000c040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0xffff900000012f00) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0x40904470 <memory::reclaimer_thread+16>, user_mutex=user_mutex@entry=0x40904520 <memory::free_page_ranges_lock>, tmr=tmr@entry=0x0) at core/condvar.cc:43#8 0x00000000403edb3a in condvar::wait (tmr=0x0, user_mutex=..., this=0x40904470 <memory::reclaimer_thread+16>) at include/osv/condvar.h:226#9 memory::reclaimer::_do_reclaim (this=0x40904460 <memory::reclaimer_thread>) at core/mempool.cc:1005#10 0x00000000403edc2c in memory::reclaimer::<lambda()>::operator() (__closure=<optimized out>) at core/mempool.cc:960#11 std::_Function_handler<void(), memory::reclaimer::reclaimer()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#12 0x0000000040401187 in sched::thread_main_c (t=0xffff80000000c040) at arch/x64/arch-switch.hh:321#13 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000050040 page-access-sca
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400c5a in sched::thread::wait (this=0xffff800000050040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::sleep_impl(sched::timer&)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::thread::sleep_impl(sched::timer&)::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 sched::thread::sleep_impl (t=...) at core/sched.cc:1370#7 0x00000000403ebbb7 in sched::thread::sleep<long, std::ratio<1l, 1000000000l> > (duration=...) at /usr/include/c++/8/chrono:715#8 pagecache::access_scanner::run (this=0x40903600 <pagecache::s_access_scanner>) at core/pagecache.cc:593#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000050040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000068040 page_pool_l2
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000068040) at core/sched.cc:1214#4 0x00000000403ef614 in sched::thread::do_wait_for<sched::thread::dummy_lock, sched::wait_object<memory::page_pool::l2::fill_thread()::<lambda()> > > (mtx=<synthetic pointer>...) at core/mempool.cc:1348#5 sched::thread::wait_for<memory::page_pool::l2::fill_thread()::<lambda()> > () at include/osv/sched.hh:1228#6 memory::page_pool::l2::fill_thread (this=0x40904280 <memory::page_pool::global_l2>) at core/mempool.cc:1346#7 0x00000000403f2ebc in memory::page_pool::l2::l2()::{lambda()#1}::operator()() const (__closure=<optimized out>) at core/mempool.cc:1165#8 std::_Function_handler<void (), memory::page_pool::l2::l2()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000068040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000085040 itimer-real
#0 thread_main () at arch/x64/entry.S:112
thread 0xffff80000008c040 itimer-virt
#0 thread_main () at arch/x64/entry.S:112
thread 0xffff80000013f040 balancer0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404013fa in sched::thread::wait (this=0xffff80000013f040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::cpu::load_balance()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::cpu::load_balance()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 sched::cpu::load_balance (this=<optimized out>) at core/sched.cc:677#7 0x0000000040401187 in sched::thread_main_c (t=0xffff80000013f040) at arch/x64/arch-switch.hh:321#8 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000156040 rcu0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff80000016d040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000156040) at core/sched.cc:1214#4 0x00000000403e5908 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, osv::rcu::cpu_quiescent_state_thread::do_work()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/percpu.hh:47#5 sched::thread::wait_until<osv::rcu::cpu_quiescent_state_thread::do_work()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 osv::rcu::cpu_quiescent_state_thread::do_work (this=0xffffa00000095920) at core/rcu.cc:178#7 0x00000000403e5c39 in osv::rcu::cpu_quiescent_state_thread::work (this=<optimized out>) at core/rcu.cc:113#8 0x00000000403e5c4c in osv::rcu::cpu_quiescent_state_thread::<lambda()>::operator() (__closure=<optimized out>) at core/rcu.cc:67#9 std::_Function_handler<void(), osv::rcu::cpu_quiescent_state_thread::cpu_quiescent_state_thread(sched::cpu*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#10 0x0000000040401187 in sched::thread_main_c (t=0xffff800000156040) at arch/x64/arch-switch.hh:321#11 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000016d040 page_pool_l1_0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800004a7b040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000016d040) at core/sched.cc:1214#4 0x00000000403ef361 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, memory::page_pool::l1::fill_thread()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at arch/x64/arch.hh:98#5 sched::thread::wait_until<memory::page_pool::l1::fill_thread()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 memory::page_pool::l1::fill_thread () at core/mempool.cc:1258#7 0x00000000403f2ea9 in memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}::operator()() const (__closure=<optimized out>) at core/mempool.cc:1087#8 std::_Function_handler<void (), memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff80000016d040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000184040 percpu0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff80000019b040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000184040) at core/sched.cc:1214#4 0x00000000404227ad in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, workman::call_of_duty()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until<workman::call_of_duty()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 workman::call_of_duty () at core/percpu-worker.cc:92#7 0x0000000040422959 in workman::<lambda()>::operator() (__closure=<optimized out>) at core/percpu-worker.cc:124#8 std::_Function_handler<void(), workman::pcpu_init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000184040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000019b040 async_worker0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000019b040) at core/sched.cc:1214#4 0x000000004042c1f4 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, async::async_worker::run()::{lambda()#1}>(sched::thread::dummy_lock&, async::async_worker::run(#5 sched::thread::wait_until<async::async_worker::run()::{lambda()#1}>(async::async_worker::run()::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 async::async_worker::run (this=0xffffa0000009c040) at core/async.cc:158#7 0x0000000040401187 in sched::thread_main_c (t=0xffff80000019b040) at arch/x64/arch-switch.hh:321#8 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000001b2040 idle0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000012de040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x000000004040102d in sched::cpu::idle (this=0xffff800000013040) at core/sched.cc:425#4 0x000000004040104c in sched::cpu::<lambda()>::operator() (__closure=<optimized out>) at core/sched.cc:165#5 std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#6 0x0000000040401187 in sched::thread_main_c (t=0xffff8000001b2040) at arch/x64/arch-switch.hh:321#7 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000afb040 >init
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000afb040) at core/sched.cc:1214#4 0x0000000040388ff8 in sched::thread::do_wait_for<sched::thread::dummy_lock, sched::wait_object<osv::acpi_interrupt::process_interrupts()::{lambda()#1}>, sched::wait_object<osv::acpi_interrupt::proce#5 sched::thread::wait_for<osv::acpi_interrupt::process_interrupts()::{lambda()#1}, osv::acpi_interrupt::process_interrupts()::{lambda()#2}>(osv::acpi_interrupt::process_interrupts()::{lambda()#1}&&, o#6 osv::acpi_interrupt::process_interrupts (this=0xffffa00000958380) at drivers/acpi.cc:242#7 osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}::operator()() const (__closure=<optimized out>) at drivers/acpi.cc:228#8 std::_Function_handler<void (), osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000afb040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b12040 >init
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400dd2 in sched::thread::wait (this=0xffff800000b12040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, sched::thread::reaper::reap()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::thread::reaper::reap()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 sched::thread::reaper::reap (this=0xffffa000001ff400) at core/sched.cc:1553#7 0x0000000040400f4c in sched::thread::reaper::<lambda()>::operator() (__closure=<optimized out>) at core/sched.cc:1544#8 std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b12040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b29040 thread taskq
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b29040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff4b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff480, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff4b0, p=0xffffa000001ff480) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b29040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b34040 callout
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b34040) at core/sched.cc:1214#4 0x00000000402625e9 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, _callout_thread()::<lambda()> > (mtx=..., pred=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<_callout_thread()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 _callout_thread () at bsd/porting/callout.cc:138#7 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b34040) at arch/x64/arch-switch.hh:321#8 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b4d040 system_taskq_0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b4d040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b4d040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b58040 system_taskq_1
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b58040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b58040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b63040 system_taskq_2
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b63040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b63040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b6e040 system_taskq_3
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b6e040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b6e040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b79040 system_taskq_4
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b79040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b79040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b84040 system_taskq_5
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b84040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b84040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b8f040 system_taskq_6
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b8f040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b8f040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000b9a040 system_taskq_7
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0x408acdf0) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000b9a040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0xffffa000001ff630, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101#7 0x000000004025db0b in TQ_SLEEP (tq=0xffffa000001ff600, t=0, wm=0x406ab8fd "-", pri=0, m=0xffffa000001ff630, p=0xffffa000001ff600) at bsd/sys/kern/subr_taskqueue.c:474#8 taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474#9 0x0000000040401187 in sched::thread_main_c (t=0xffff800000b9a040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000fe9040 solthread-0x402
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000ff4040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000fe9040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0xffff800000ff2e80, this=0xffff800000ff2e30) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0x408e8820 <arc_reclaim_thr_cv>, user_mutex=user_mutex@entry=0x408e8860 <arc_reclaim_thr_lock>, tmr=tmr@entry=0xffff800000ff2e80) at core/condvar.cc:43#8 0x0000000040260554 in condvar::wait<long, std::ratio<1l, 1000000000l> > (duration=..., user_mutex=0x408e8860 <arc_reclaim_thr_lock>, this=0x408e8820 <arc_reclaim_thr_cv>) at /usr/include/c++/8/chrono:715#9 cv_timedwait (cv=cv@entry=0x408e8820 <arc_reclaim_thr_cv>, mutex=mutex@entry=0x408e8860 <arc_reclaim_thr_lock>, tmo=tmo@entry=1000) at bsd/porting/netport1.cc:66#10 0x00000000402cdd55 in arc_reclaim_thread (dummy=<optimized out>) at bsd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2539#11 0x0000000040401187 in sched::thread_main_c (t=0xffff800000fe9040) at arch/x64/arch-switch.hh:321#12 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800000ff4040 solthread-0x402
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000b34040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800000ff4040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0xffff800000ffde20, this=0xffff800000ffddd0) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0x408d88a0 <l2arc_feed_thr_cv>, user_mutex=user_mutex@entry=0x408d88e0 <l2arc_feed_thr_lock>, tmr=tmr@entry=0xffff800000ffde20) at core/condvar.cc:43#8 0x0000000040260554 in condvar::wait<long, std::ratio<1l, 1000000000l> > (duration=..., user_mutex=0x408d88e0 <l2arc_feed_thr_lock>, this=0x408d88a0 <l2arc_feed_thr_cv>) at /usr/include/c++/8/chrono:715#9 cv_timedwait (cv=cv@entry=0x408d88a0 <l2arc_feed_thr_cv>, mutex=mutex@entry=0x408d88e0 <l2arc_feed_thr_lock>, tmo=tmo@entry=1000) at bsd/porting/netport1.cc:66#10 0x00000000402cc379 in l2arc_feed_thread (dummy=<optimized out>) at bsd/sys/cddl/compat/opensolaris/sys/time.h:72#11 0x0000000040401187 in sched::thread_main_c (t=0xffff800000ff4040) at arch/x64/arch-switch.hh:321#12 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000101b040 netisr
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff80000135f040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000101b040) at core/sched.cc:1214#4 0x0000000040272272 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, netisr_osv_thread_wrapper(netisr_osv_handler_t, void*)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at bsd/sys/net/netisr1.cc:26#5 sched::thread::wait_until<netisr_osv_thread_wrapper(netisr_osv_handler_t, void*)::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 netisr_osv_thread_wrapper (handler=0x40270c30 <swi_net(void*)>, arg=0x408b41c0 <main_nws>) at bsd/sys/net/netisr1.cc:26#7 0x0000000040272320 in <lambda()>::operator() (__closure=<optimized out>, __closure=<optimized out>) at bsd/sys/net/netisr1.cc:39#8 std::_Function_handler<void(), netisr_osv_start_thread(netisr_osv_handler_t, void*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff80000101b040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800001182040 >init
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000001b2040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400702 in sched::thread::wait (this=0xffff800001182040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::join()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::thread::join()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 sched::thread::join (this=<optimized out>) at core/sched.cc:1311#7 0x000000004045f626 in pthread_private::pthread::join (retval=0x0, this=0xffffa00000983720) at /usr/include/c++/8/bits/unique_ptr.h:342#8 pthread_join (thread=18446638520603260704, retval=retval@entry=0x0) at libc/pthread.cc:245#9 0x000000004042e68a in osv::application::join (this=0xffffa0000115f210) at core/app.cc:259#10 0x000000004022df1a in do_main_thread (_main_args=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1018#11 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00000952d00) at libc/pthread.cc:114#12 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#13 0x0000000040401187 in sched::thread_main_c (t=0xffff800001182040) at arch/x64/arch-switch.hh:321#14 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000119c040 virtio-net-rx
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff80000135f040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000119c040) at core/sched.cc:1214#4 0x00000000403752f1 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda#5 sched::thread::wait_until<virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 virtio::virtio_driver::wait_for_queue (this=this@entry=0xffff90000112d000, queue=queue@entry=0xffffa00000952f00, pred=<optimized out>) at drivers/virtio.cc:160#7 0x0000000040379c6c in virtio::net::receiver (this=0xffff90000112d000) at drivers/virtio-net.cc:451#8 0x000000004037a1bc in virtio::net::<lambda()>::operator() (__closure=<optimized out>) at drivers/virtio-net.cc:242#9 std::_Function_handler<void(), virtio::net::net(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#10 0x0000000040401187 in sched::thread_main_c (t=0xffff80000119c040) at arch/x64/arch-switch.hh:321#11 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000011c4040 virtio-tx-0
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800001182040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000011c4040) at core/sched.cc:1214#4 0x000000004037b489 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output#5 0x000000004037d0bc in sched::thread::wait_until<osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::p#6 osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() (this=0xffff90000112d3c0) at include/osv/percpu_xmit.hh:395#7 0x000000004037d1ec in osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, #8 std::_Function_handler<void (), osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::n#9 0x0000000040401187 in sched::thread_main_c (t=0xffff8000011c4040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000012de040 virtio-blk
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000050040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000012de040) at core/sched.cc:1214#4 0x00000000403752f1 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda#5 sched::thread::wait_until<virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 virtio::virtio_driver::wait_for_queue (this=this@entry=0xffffa000012b9400, queue=queue@entry=0xffffa00001130500, pred=<optimized out>) at drivers/virtio.cc:160#7 0x00000000403841e5 in virtio::blk::req_done (this=<optimized out>) at drivers/virtio-blk.cc:227#8 0x000000004038448c in virtio::blk::<lambda()>::operator() (__closure=<optimized out>) at drivers/virtio-blk.cc:130#9 std::_Function_handler<void(), virtio::blk::blk(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#10 0x0000000040401187 in sched::thread_main_c (t=0xffff8000012de040) at arch/x64/arch-switch.hh:321#11 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000012f5040 virtio-rng
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000050040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000012f5040) at core/sched.cc:1214#4 0x00000000403752f1 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda#5 sched::thread::wait_until<virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 virtio::virtio_driver::wait_for_queue (this=this@entry=0xffffa000012b9c00, queue=0xffffa00001130600, pred=<optimized out>) at drivers/virtio.cc:160#7 0x0000000040387074 in virtio::rng::refill (this=0xffffa000012b9c00) at drivers/virtio-rng.cc:132#8 0x0000000040387471 in virtio::rng::worker (this=0xffffa000012b9c00) at drivers/virtio-rng.cc:107#9 0x000000004038749c in virtio::rng::<lambda()>::operator() (__closure=<optimized out>) at drivers/virtio-rng.cc:41#10 std::_Function_handler<void(), virtio::rng::rng(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#11 0x0000000040401187 in sched::thread_main_c (t=0xffff8000012f5040) at arch/x64/arch-switch.hh:321#12 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000130f040 isa-serial-inpu
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800001182040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000130f040) at core/sched.cc:1214#4 0x0000000040358b70 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 console::LineDiscipline::read_poll (this=0xffffa00001130700, driver=0x408ec7c0 <console::arch_early_console>) at drivers/line-discipline.cc:86#7 0x0000000040401187 in sched::thread_main_c (t=0xffff80000130f040) at arch/x64/arch-switch.hh:321#8 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800001326040 kbd-input
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800001182040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800001326040) at core/sched.cc:1214#4 0x0000000040358b70 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 console::LineDiscipline::read_poll (this=0xffffa00001130700, driver=0xffff800001188040) at drivers/line-discipline.cc:86#7 0x0000000040401187 in sched::thread_main_c (t=0xffff800001326040) at arch/x64/arch-switch.hh:321#8 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800001348040 rand_harvestq
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800001348040) at core/sched.cc:1214#4 0x000000004026355a in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > ( mtx=<synthetic pointer>..., pred=...) at /usr/include/c++/8/bits/atomic_base.h:390#5 sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1099#6 synch_port::_msleep (this=0xffffa000001ff500, chan=<optimized out>, mtx=0x0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=100) at bsd/porting/synch.cc:101#7 0x00000000402b327c in random_kthread (arg=0x402b2b80 <random_process_event(harvest*)>) at bsd/sys/dev/random/random_harvestq.cc:112#8 0x0000000040401187 in sched::thread_main_c (t=0xffff800001348040) at arch/x64/arch-switch.hh:321#9 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000135f040 dhcp
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800001182040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000135f040) at core/sched.cc:1214#4 0x00000000404271e0 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, dhcp::dhcp_worker::dhcp_worker_fn()::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<dhcp::dhcp_worker::dhcp_worker_fn()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 dhcp::dhcp_worker::dhcp_worker_fn (this=0x40908820 <net_dhcp_worker>) at core/dhcp.cc:828#7 0x000000004042734c in dhcp::dhcp_worker::<lambda()>::operator() (__closure=<optimized out>) at core/dhcp.cc:773#8 std::_Function_handler<void(), dhcp::dhcp_worker::init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#9 0x0000000040401187 in sched::thread_main_c (t=0xffff80000135f040) at arch/x64/arch-switch.hh:321#10 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000014af040 /java.so
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400702 in sched::thread::wait (this=0xffff8000014af040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::join()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::thread::join()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 sched::thread::join (this=<optimized out>) at core/sched.cc:1311#7 0x000000004045f626 in pthread_private::pthread::join (retval=0x0, this=0xffffa00000983730) at /usr/include/c++/8/bits/unique_ptr.h:342#8 pthread_join (thread=18446638520603260720, retval=0x0) at libc/pthread.cc:245#9 0x00000000404d06d3 in std::thread::join() ()#10 0x0000100000019386 in main (argc=<optimized out>, argv=0xffffa0000115f800) at /home/wkozaczuk/projects/osv/modules/java-base/java.cc:270#11 0x000000004043096d in osv::application::run_main (this=0xffffa0000115f210) at /usr/include/c++/8/bits/stl_vector.h:805#12 0x0000000040226b61 in osv::application::main (this=0xffffa0000115f210) at core/app.cc:320#13 0x0000000040430b19 in osv::application::<lambda(void*)>::operator() (__closure=0x0, app=<optimized out>) at core/app.cc:233#14 osv::application::<lambda(void*)>::_FUN(void *) () at core/app.cc:235#15 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000129c400) at libc/pthread.cc:114#16 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#17 0x0000000040401187 in sched::thread_main_c (t=0xffff8000014af040) at arch/x64/arch-switch.hh:321#18 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000014b5040 >/java.so
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000001b2040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000014b5040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x2000002ff190) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa00001ee7500, user_mutex=0xffffa0000478a380, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011fedd5 in ObjectMonitor::wait(long, bool, Thread*) ()#10 0x00001000014ad3e2 in ObjectSynchronizer::wait(Handle, long, Thread*) ()#11 0x0000100000f44268 in JVM_MonitorWait ()#12 0x0000200001c0ffb7 in ?? ()#13 0x0000000000000000 in ?? ()
thread 0xffff800003195040 VM Thread
#0 lockfree::mutex::unlock (this=this@entry=0xffffa000012d4690) at core/lfmutex.cc:221#1 0x00000000403e36bc in std::lock_guard<lockfree::mutex>::~lock_guard (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:167#2 lock_guard_for_with_lock<lockfree::mutex>::~lock_guard_for_with_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at include/osv/mutex.h:88#3 semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64#4 0x0000000040465731 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#5 0x00001000012feb22 in PosixSemaphore::trywait() ()#6 0x0000100000e19ebb in VM_HandshakeAllThreads::doit() ()#7 0x0000100001575fd7 in VM_Operation::evaluate() ()#8 0x000010000157bdb6 in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.67] ()#9 0x000010000157c50a in VMThread::loop() ()#10 0x000010000157c6e3 in VMThread::run() ()#11 0x000010000150479f in Thread::call_run() ()#12 0x000010000122cbae in thread_native_entry(Thread*) ()#13 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00002c7cf00) at libc/pthread.cc:114#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#15 0x0000000040401187 in sched::thread_main_c (t=0xffff800003195040) at arch/x64/arch-switch.hh:321#16 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000348e040 Reference Handl
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000348e040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x200011600870) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa000030b6500, user_mutex=0xffffa00002f5adc0, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011d31c8 in Monitor::IWait(Thread*, long) ()#10 0x00001000011d4386 in Monitor::wait(bool, long, bool) ()#11 0x0000100000f4b88e in JVM_WaitForReferencePendingList ()#12 0x0000200001c0ffb7 in ?? ()#13 0x0000200001c0fc76 in ?? ()#14 0x00002000116009a8 in ?? ()#15 0x0000200010c314f0 in ?? ()#16 0x0000200011600a08 in ?? ()#17 0x0000200010ce3f20 in ?? ()#18 0x0000000000000000 in ?? ()
thread 0xffff80000359d040 Finalizer
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000359d040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x2000117006c0) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa000030b6980, user_mutex=0xffffa00003428040, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011fedd5 in ObjectMonitor::wait(long, bool, Thread*) ()#10 0x00001000014ad3e2 in ObjectSynchronizer::wait(Handle, long, Thread*) ()#11 0x0000100000f44268 in JVM_MonitorWait ()#12 0x0000200001c0ffb7 in ?? ()#13 0x0000200011700940 in ?? ()#14 0xffff900003228000 in ?? ()#15 0x0000000000000000 in ?? ()
thread 0xffff80000383c040 Signal Dispatch
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000383c040) at core/sched.cc:1214#4 0x00000000403e35af in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46#7 0x0000000040465643 in sem_wait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#8 0x00001000012feaf2 in PosixSemaphore::wait() ()#9 0x00001000012263db in os::signal_wait() ()#10 0x000010000121a66c in signal_thread_entry(JavaThread*, Thread*) ()#11 0x0000100001507cdf in JavaThread::run() ()#12 0x000010000150479f in Thread::call_run() ()#13 0x000010000122cbae in thread_native_entry(Thread*) ()#14 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000365b200) at libc/pthread.cc:114#15 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#16 0x0000000040401187 in sched::thread_main_c (t=0xffff80000383c040) at arch/x64/arch-switch.hh:321#17 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800003942040 C2 CompilerThre
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800000050040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800003942040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x200011900b90, this=0x200011900b30) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0xffffa0000357f700, user_mutex=0xffffa00003428a80, tmr=tmr@entry=0x200011900b90) at core/condvar.cc:43#8 0x00000000404601c0 in pthread_cond_timedwait (cond=0xffffa0000357ed50, mutex=0xffffa0000357ed28, ts=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:707#9 0x0000100001235d0e in os::PlatformEvent::park(long) ()#10 0x00001000011d311b in Monitor::IWait(Thread*, long) ()#11 0x00001000011d4386 in Monitor::wait(bool, long, bool) ()#12 0x0000100000c2884d in CompileQueue::get() ()#13 0x0000100000c3188f in CompileBroker::compiler_thread_loop() ()#14 0x0000100001507cdf in JavaThread::run() ()#15 0x000010000150479f in Thread::call_run() ()#16 0x000010000122cbae in thread_native_entry(Thread*) ()#17 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000365b800) at libc/pthread.cc:114#18 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#19 0x0000000040401187 in sched::thread_main_c (t=0xffff800003942040) at arch/x64/arch-switch.hh:321#20 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800003a63040 C1 CompilerThre
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800003a63040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x200011a00b90, this=0x200011a00b30) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0xffffa0000357fc80, user_mutex=0xffffa00003428d00, tmr=tmr@entry=0x200011a00b90) at core/condvar.cc:43#8 0x00000000404601c0 in pthread_cond_timedwait (cond=0xffffa000034bcb50, mutex=0xffffa000034bcb28, ts=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:707#9 0x0000100001235d0e in os::PlatformEvent::park(long) ()#10 0x00001000011d311b in Monitor::IWait(Thread*, long) ()#11 0x00001000011d4386 in Monitor::wait(bool, long, bool) ()#12 0x0000100000c2884d in CompileQueue::get() ()#13 0x0000100000c3188f in CompileBroker::compiler_thread_loop() ()#14 0x0000100001507cdf in JavaThread::run() ()#15 0x000010000150479f in Thread::call_run() ()#16 0x000010000122cbae in thread_native_entry(Thread*) ()#17 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000365bf00) at libc/pthread.cc:114#18 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#19 0x0000000040401187 in sched::thread_main_c (t=0xffff800003a63040) at arch/x64/arch-switch.hh:321#20 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800003b8d040 Sweeper thread
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800004a7b040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800003b8d040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x200011b00a90) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa00003850200, user_mutex=0xffffa00003428f40, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011d31c8 in Monitor::IWait(Thread*, long) ()#10 0x00001000011d4386 in Monitor::wait(bool, long, bool) ()#11 0x000010000157b8e7 in VMThread::execute(VM_Operation*) ()#12 0x0000100000e18d78 in Handshake::execute(ThreadClosure*) ()#13 0x00001000014a0d0c in NMethodSweeper::possibly_sweep() ()#14 0x00001000014a0f25 in NMethodSweeper::sweeper_loop() ()#15 0x0000100001507cdf in JavaThread::run() ()#16 0x000010000150479f in Thread::call_run() ()#17 0x000010000122cbae in thread_native_entry(Thread*) ()#18 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00003851500) at libc/pthread.cc:114#19 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#20 0x0000000040401187 in sched::thread_main_c (t=0xffff800003b8d040) at arch/x64/arch-switch.hh:321#21 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800004276040 Common-Cleaner
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800004276040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x200011c00610, this=0x200011c005b0) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0xffffa00003ffe400, user_mutex=0xffffa00003428f80, tmr=tmr@entry=0x200011c00610) at core/condvar.cc:43#8 0x00000000404601c0 in pthread_cond_timedwait (cond=0xffffa0000401f750, mutex=0xffffa0000401f728, ts=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:707#9 0x0000100001235d0e in os::PlatformEvent::park(long) ()#10 0x00001000011fec90 in ObjectMonitor::wait(long, bool, Thread*) ()#11 0x00001000014ad3e2 in ObjectSynchronizer::wait(Handle, long, Thread*) ()#12 0x0000100000f44268 in JVM_MonitorWait ()#13 0x0000200001c0ffb7 in ?? ()#14 0x0000200011c00880 in ?? ()#15 0x0000100001224196 in os::javaTimeNanos() ()#16 0x0000200001c09c63 in ?? ()#17 0x0000000000000000 in ?? ()
thread 0xffff8000045c0040 Service Thread
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000045c0040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x200011d00c60) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa000043aaf00, user_mutex=0xffffa000043381c0, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011d31c8 in Monitor::IWait(Thread*, long) ()#10 0x00001000011d4349 in Monitor::wait(bool, long, bool) ()#11 0x00001000012ff7f2 in ServiceThread::service_thread_entry(JavaThread*, Thread*) ()#12 0x0000100001507cdf in JavaThread::run() ()#13 0x000010000150479f in Thread::call_run() ()#14 0x000010000122cbae in thread_native_entry(Thread*) ()#15 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa000043a9b00) at libc/pthread.cc:114#16 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#17 0x0000000040401187 in sched::thread_main_c (t=0xffff8000045c0040) at arch/x64/arch-switch.hh:321#18 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff8000046cf040 VM Periodic Tas
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff8000046cf040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x200011e01ce0, this=0x200011e01c80) at include/osv/wait_record.hh:46#7 condvar::wait (this=this@entry=0xffffa00004335500, user_mutex=0xffffa00004338980, tmr=tmr@entry=0x200011e01ce0) at core/condvar.cc:43#8 0x00000000404601c0 in pthread_cond_timedwait (cond=0xffffa0000457d550, mutex=0xffffa0000457d528, ts=<optimized out>) at /usr/include/c++/8/bits/atomic_base.h:707#9 0x0000100001235d0e in os::PlatformEvent::park(long) ()#10 0x00001000011d311b in Monitor::IWait(Thread*, long) ()#11 0x00001000011d4349 in Monitor::wait(bool, long, bool) ()#12 0x00001000014ff113 in WatcherThread::sleep() const ()#13 0x00001000014ff23c in WatcherThread::run() ()#14 0x000010000150479f in Thread::call_run() ()#15 0x000010000122cbae in thread_native_entry(Thread*) ()#16 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa000043a9f00) at libc/pthread.cc:114#17 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#18 0x0000000040401187 in sched::thread_main_c (t=0xffff8000046cf040) at arch/x64/arch-switch.hh:321#19 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff800004a7b040 Thread-1
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff800003195040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff800004a7b040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x200011f00030) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa0000474f680, user_mutex=0xffffa0000478a300, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001235b73 in os::PlatformEvent::park() ()#9 0x00001000011d31c8 in Monitor::IWait(Thread*, long) ()#10 0x00001000011d4386 in Monitor::wait(bool, long, bool) ()#11 0x000010000157b8e7 in VMThread::execute(VM_Operation*) ()#12 0x0000100000a6a21e in BiasedLocking::revoke_and_rebias(Handle, bool, Thread*) ()#13 0x00001000014adb6d in ObjectSynchronizer::fast_enter(Handle, BasicLock*, bool, Thread*) ()#14 0x0000100000e8a7a2 in InterpreterRuntime::monitorenter(JavaThread*, BasicObjectLock*) ()#15 0x0000200001c20ba9 in ?? ()#16 0x00000000fc15edf8 in ?? ()#17 0x0000200001c20b6a in ?? ()#18 0x00000000fc0fae60 in ?? ()#19 0x00000000fc0fae60 in ?? ()#20 0x0000200011f00460 in ?? ()#21 0x0000200010cbe108 in ?? ()#22 0x0000200011f00500 in ?? ()#23 0x0000200010e03150 in ?? ()#24 0x0000000000000000 in ?? ()Waldek Kozaczuk
Jul 15, 2019, 6:05:58 PM7/15/19
to OSv Development
There is this thread that seems to be doing something related to signals:
thread 0xffff80000383c040 Signal Dispatch
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000383c040) at core/sched.cc:1214#4 0x00000000403e35af in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41#5 sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084#6 semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46#7 0x0000000040465643 in sem_wait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#8 0x00001000012feaf2 in PosixSemaphore::wait() ()#9 0x00001000012263db in os::signal_wait() ()#10 0x000010000121a66c in signal_thread_entry(JavaThread*, Thread*) ()#11 0x0000100001507cdf in JavaThread::run() ()#12 0x000010000150479f in Thread::call_run() ()#13 0x000010000122cbae in thread_native_entry(Thread*) ()#14 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000365b200) at libc/pthread.cc:114#15 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) ( __functor=...) at /usr/include/c++/8/bits/std_function.h:297#16 0x0000000040401187 in sched::thread_main_c (t=0xffff80000383c040) at arch/x64/arch-switch.hh:321#17 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113Could it be caused by some limitations of signal handling in OSv?
...
Waldek Kozaczuk
Jul 20, 2019, 1:15:59 AM7/20/19
to OSv Development
I am thinking these 2 threads are somehow entangled:
Thread 0xffff800003195040 VM Thread
#0 lockfree::mutex::unlock (this=this@entry=0xffffa000012d4690) at core/lfmutex.cc:221
#1 0x00000000403e36bc in std::lock_guard<lockfree::mutex>::~lock_guard (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:167
#2 lock_guard_for_with_lock<lockfree::mutex>::~lock_guard_for_with_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at include/osv/mutex.h:88
#3 semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64
#4 0x0000000040465731 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342
#5 0x00001000012feb22 in PosixSemaphore::trywait() ()
#6 0x0000100000e19ebb in VM_HandshakeAllThreads::doit() ()
#7 0x0000100001575fd7 in VM_Operation::evaluate() ()
#8 0x000010000157bdb6 in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.67] ()
#9 0x000010000157c50a in VMThread::loop() ()
#10 0x000010000157c6e3 in VMThread::run() ()
#11 0x000010000150479f in Thread::call_run() ()
#12 0x000010000122cbae in thread_native_entry(Thread*) ()
#13 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00002c7cf00) at libc/pthread.cc:114
#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (
__functor=...) at /usr/include/c++/8/bits/std_function.h:297
#15 0x0000000040401187 in sched::thread_main_c (t=0xffff800003195040) at arch/x64/arch-switch.hh:321
#16 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
thread 0xffff80000383c040 Signal Dispatch
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108
#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...)
at core/sched.cc:339
#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310
#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000383c040) at core/sched.cc:1214
#4 0x00000000403e35af in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41
#5 sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084
#6 semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46
#7 0x0000000040465643 in sem_wait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342
#8 0x00001000012feaf2 in PosixSemaphore::wait() ()
#9 0x00001000012263db in os::signal_wait() ()
#10 0x000010000121a66c in signal_thread_entry(JavaThread*, Thread*) ()
#11 0x0000100001507cdf in JavaThread::run() ()
#12 0x000010000150479f in Thread::call_run() ()
#13 0x000010000122cbae in thread_native_entry(Thread*) ()
#14 0x000000004045ee26 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa0000365b200) at libc/pthread.cc:114
#15 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (
__functor=...) at /usr/include/c++/8/bits/std_function.h:297
#16 0x0000000040401187 in sched::thread_main_c (t=0xffff80000383c040) at arch/x64/arch-switch.hh:321
#17 0x00000000403a7113 in thread_main () at arch/x64/entry.S:113
Shouldn't sem_trywait() return immediately per https://linux.die.net/man/3/sem_trywait, instead it seems to block?
--
You received this message because you are subscribed to a topic in the Google Groups "OSv Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osv-dev/wwMWBtWFcEQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/8b54996f-e03e-4462-bbf2-a2076727c5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Waldek Kozaczuk
Jul 20, 2019, 1:41:00 AM7/20/19
to OSv Development
Are we properly handling the EINTR scenario, where thread blocked on sem_wait() or sem_timed_wait() should get interrupted if signal sent to it?
...
Waldek Kozaczuk
Jul 20, 2019, 1:01:39 PM7/20/19
to OSv Development
I am thinking we might simply have a bug in core/semaphore.cc. Based on these two lines:
semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64 -> from VM Thread
and
semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46 -> from Signal Dispatch
it seems that VM Thread is waiting on mutex _mtx while the Signal Dispatch is waiting on the timer to expire or be interrupted (?) which never happens (why?). But what is weirder is that
Signal Dispatch seems to be indefinitely stuck waiting for the mutex to be released. Is it because the core around waiting does not release the mutex?
This address seems to be different for each thread so these seem to be different semaphores, right? But then I do not see other threads having stack traces that involve semaphore code.
What am I missing? Which thread is holding the mutex VM Thread is waiting for in try_wait()?
Waldek
Nadav Har'El
Jul 21, 2019, 4:30:51 AM7/21/19
to Waldek Kozaczuk, OSv Development
On Sat, Jul 20, 2019 at 8:01 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:
I am thinking we might simply have a bug in core/semaphore.cc. Based on these two lines:semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64 -> from VM Thread
This is interesting. trywait() isn't supposed to block for any significant time - it takes a mutex, but this mutex is only supposed
to be held by other functions for very short durations. If it's taken for a long duration then I think you indeed found a bug -
a deadlock.
But I don't know what can possible cause _mtx to be taken for a long time... The other functions taking it are post() and wait().
post() only holds it for a short time.
wait() uses:
sched::thread::wait_until(_mtx,
[&] { return (tmr && tmr->expired()) || !wr.owner; });
[&] { return (tmr && tmr->expired()) || !wr.owner; });
The idea of this mutex parameter of wait_until is that the mutex is released while waiting, and taken back again when done waiting.
There could of course be a bug in wait_until(), but it's been heavily used and tested over the years, so it would be surprising if that's
the problem.
Are you sure you see a thread hung on a try_wait? Maybe it's just a try_wait() loop, so there is a high probability of seeing it?
You can do something like add a tracepoint for try_wait() to see if it's called in a loop, or hung in one call.
andsemaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46 -> from Signal Dispatchit seems that VM Thread is waiting on mutex _mtx while the Signal Dispatch is waiting on the timer to expire or be interrupted (?) which never happens (why?). But what is weirder is thatSignal Dispatch seems to be indefinitely stuck waiting for the mutex to be released. Is it because the core around waiting does not release the mutex?This address seems to be different for each thread so these seem to be different semaphores, right?
Right
--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e24a4a3a-17be-4013-9d7b-491558092743%40googlegroups.com.
Nadav Har'El
Jul 21, 2019, 5:55:56 AM7/21/19
to Waldek Kozaczuk, OSv Development
On Sun, Jul 21, 2019 at 11:30 AM Nadav Har'El <n...@scylladb.com> wrote:
On Sat, Jul 20, 2019 at 8:01 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:I am thinking we might simply have a bug in core/semaphore.cc. Based on these two lines:semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64 -> from VM ThreadThis is interesting. trywait() isn't supposed to block for any significant time - it takes a mutex, but this mutex is only supposedto be held by other functions for very short durations. If it's taken for a long duration then I think you indeed found a bug -a deadlock.
But I don't know what can possible cause _mtx to be taken for a long time... The other functions taking it are post() and wait().post() only holds it for a short time.wait() uses:sched::thread::wait_until(_mtx,
[&] { return (tmr && tmr->expired()) || !wr.owner; });The idea of this mutex parameter of wait_until is that the mutex is released while waiting, and taken back again when done waiting.There could of course be a bug in wait_until(), but it's been heavily used and tested over the years, so it would be surprising if that'sthe problem.Are you sure you see a thread hung on a try_wait? Maybe it's just a try_wait() loop, so there is a high probability of seeing it?You can do something like add a tracepoint for try_wait() to see if it's called in a loop, or hung in one call.
Unfortunately, I don't have too much time to devote to this issue, but I took a cursory look. I think may have been right that you caught a try_wait() loop, not a hang on try_wait(), because I see this in the full thread backtrace (thanks for supplying it):
thread 0xffff800003195040 VM Thread
#0 lockfree::mutex::unlock (this=this@entry=0xffffa000012d4690) at core/lfmutex.cc:221
#1 0x00000000403e36bc in std::lock_guard<lockfree::mutex>::~lock_guard (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:167
#2 lock_guard_for_with_lock<lockfree::mutex>::~lock_guard_for_with_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at include/osv/mutex.h:88
#3 semaphore::trywait (this=0xffffa000012d4680, units=units@entry=1) at core/semaphore.cc:64
#4 0x0000000040465731 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342
#5 0x00001000012feb22 in PosixSemaphore::trywait() ()
#6 0x0000100000e19ebb in VM_HandshakeAllThreads::doit() ()
You can see here that the thread is not hung - it isn't sleeping, it wasn't scheduled a way, you simply happened to catch it in an mutex::unlock() call which takes some time, but cannot block. I am guessing that if you run the same thing again and look at the same thread, you'll find it in a different place - either in some other line of code inside sem_trywait() if this is a trywait loop, or, in something completely different. If it's a trywait loop, I would take a look at VM_HandshakeAllThreads::doit() (I guess you have the JVM source code) to see what it is that it is trying to wait for.
Your other guess is that the problem was in:
thread 0xffff80000383c040 Signal Dispatch
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108
#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...)
at core/sched.cc:339
#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310
#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000383c040) at core/sched.cc:1214
#4 0x00000000403e35af in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41
#5 sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084
#6 semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46
#7 0x0000000040465643 in sem_wait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342
#8 0x00001000012feaf2 in PosixSemaphore::wait() ()
#9 0x00001000012263db in os::signal_wait() ()
#10 0x000010000121a66c in signal_thread_entry(JavaThread*, Thread*) ()
#0 sched::thread::switch_to (this=0xffffffffffffffb0, this@entry=0xffff8000014b5040) at arch/x64/arch-switch.hh:108
#1 0x00000000403ff804 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...)
at core/sched.cc:339
#2 0x00000000403ffcfc in sched::cpu::schedule () at include/osv/sched.hh:1310
#3 0x00000000404003e2 in sched::thread::wait (this=this@entry=0xffff80000383c040) at core/sched.cc:1214
#4 0x00000000403e35af in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/mutex.h:41
#5 sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1084
#6 semaphore::wait (this=0xffffa00001f95680, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46
#7 0x0000000040465643 in sem_wait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342
#8 0x00001000012feaf2 in PosixSemaphore::wait() ()
#9 0x00001000012263db in os::signal_wait() ()
#10 0x000010000121a66c in signal_thread_entry(JavaThread*, Thread*) ()
Note that a different semaphore address is involved here, so this is completely unrelated to the other try_wait issue, if it is an issue.
Please take a look at the os::signal_wait() code and what it tries to do.
If I had to guess without looking at the code, and just seeing the function's name, Java may be waiting on some random semaphore but really caring about this wait being interrupted by a system call. But our sem_wait() apparently cannot interrupted by a signal. To do that, sem_wait() would need to use instead of wait_until something like:
try
{
sched::thread::wait_until_interruptible([&] {
return ( (timo_hz && t.expired()) ||
(wait._awake) );
});
}
catch (int e)
{
assert(e == EINTR);
interrupted = true;
}
{
sched::thread::wait_until_interruptible([&] {
return ( (timo_hz && t.expired()) ||
(wait._awake) );
});
}
catch (int e)
{
assert(e == EINTR);
interrupted = true;
}
(this is a snippet from our msleep() code).
But worse, it appears that not all signals currently cause this sort of interruption, only SIGALRM does - see commit 8b5d9994acb7cb48dcf1db5fb46c9d5de74d4cd8. Maybe we'll need to change kill() to do that on every signal.
But I would try by looking at what Java's os::signal_wait() is actually trying to do (and could it be blocking the run?), and if anything changed in this in Java 12, maybe this is just a red herring and not a real problem.
Nadav Har'El
Jul 21, 2019, 5:58:41 AM7/21/19
to Waldek Kozaczuk, OSv Development
On Sat, Jul 20, 2019 at 8:41 AM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:
Are we properly handling the EINTR scenario, where thread blocked on sem_wait() or sem_timed_wait() should get interrupted if signal sent to it?
No, I believe we do not support this correctly. Please see my other mail on some pointers on what we need to change to properly support EINTR for sem_wait() and random signals (except SIGALRM).
But please try to understand whether it is possible that this code changed in Java 12, or why Java hangs waiting for it. If not, then maybe it's a problem that always existed but never caused any problems?
Waldek Kozaczuk
Aug 5, 2019, 11:10:32 PM8/5/19
to OSv Development
I have not gotten to the bottom of this issue but I think I have narrowed it down and found a workaround.
First of all, I discovered a JVM option that limits the usage of signals: `-Xrs' (please see for details this article https://docs.oracle.com/en/java/javase/12/troubleshoot/handle-signals-and-exceptions.html#GUID-57C048F6-0D4B-43BD-B27C-06A613435360).
So after I added -Xrs to the command line I would still see the almost exact same type of 'deadlock-like' threads state except the 'Signal Dispatch' is no longer present. Here is a part of the thread dump:
thread 0xffff8000014ae040 /usr/lib/jvm/jd
#0 sched::thread::switch_to (this=0xffffa00003b3b6f0, this@entry=0xffff800001fdd040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff794 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffc8c in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400692 in sched::thread::wait (this=0xffff8000014ae040) at core/sched.cc:1214#4 sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::join()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1066#5 sched::thread::wait_until<sched::thread::join()::<lambda()> > (pred=...) at include/osv/sched.hh:1077#6 sched::thread::join (this=<optimized out>) at core/sched.cc:1311#7 0x000000004045f5b6 in pthread_private::pthread::join (retval=0x2000001fbbc8, this=0xffffa00000982d70) at /usr/include/c++/8/bits/unique_ptr.h:342#8 pthread_join (thread=18446638520603258224, retval=0x2000001fbbc8) at libc/pthread.cc:245#9 0x0000100000609efd in ContinueInNewThread0 ()#10 0x0000100000606552 in ContinueInNewThread ()#11 0x0000100000607b68 in JLI_Launch ()#12 0x00000000004009f3 in main ()#13 0x00000000404308fd in osv::application::run_main (this=0xffffa0000115e210) at /usr/include/c++/8/bits/stl_vector.h:805#14 0x0000000040430a4f in __libc_start_main (main=0x4008d0 <main>) at core/app.cc:37#15 0x0000000000400a91 in _start ()#16 0x00002000001fff48 in ?? ()#17 0xfffffffffffff998 in ?? ()#18 0xffffa000012ae4c0 in ?? ()#19 0x000000000000003c in ?? ()#20 0x000000000000003c in ?? ()#21 0x0000000000000040 in ?? ()#22 0x00002000001fffa0 in ?? ()#23 0xffffa0000129d900 in ?? ()#24 0xffffa00000982d50 in ?? ()#25 0xffff800000013040 in ?? ()#26 0x00002000001fffa0 in ?? ()#27 0x0000000040430aa9 in osv::application::<lambda(void*)>::operator() (__closure=0x0, app=<optimized out>) at core/app.cc:233#28 osv::application::<lambda(void*)>::_FUN(void *) () at core/app.cc:235Backtrace stopped: Cannot access memory at address 0x8
thread 0xffff800001fdd040 >/usr/lib/jvm/j
#0 sched::thread::switch_to (this=0xffffa00003b3b6f0, this@entry=0xffff800003722040) at arch/x64/arch-switch.hh:108
#1 0x00000000403ff794 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffc8c in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400372 in sched::thread::wait (this=this@entry=0xffff800001fdd040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*--Type <RET> for more, q to quit, c to continue without paging--) const::{lambda()#1}) (pred=..., mtx=<synthetic pointer>...) at include/osv/sched.hh:939#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x2000002fdb40) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa0000202bb00, user_mutex=0xffffa0000382ca80, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001637213 in os::PlatformEvent::park() ()#9 0x00001000015d48e8 in Monitor::IWait(Thread*, long) ()#10 0x00001000015d5aa6 in Monitor::wait(bool, long, bool) ()#11 0x0000100001990237 in VMThread::execute(VM_Operation*) ()#12 0x0000100000fef3ca in CodeCache::flush_dependents_on(InstanceKlass*) ()#13 0x00001000018c8f30 in SystemDictionary::define_instance_class(InstanceKlass*, Thread*) ()#14 0x00001000018c92f2 in SystemDictionary::find_or_define_instance_class(Symbol*, Handle, InstanceKlass*, Thread*) ()#15 0x00001000018cb2fc in SystemDictionary::load_instance_class(Symbol*, Handle, Thread*) ()#16 0x00001000018ca1c2 in SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*) ()#17 0x00001000018cc9e4 in SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*) ()#18 0x0000100001077a8d in ConstantPool::klass_at_impl(constantPoolHandle const&, int, bool, Thread*) ()#19 0x00001000012871cc in InterpreterRuntime::_new(JavaThread*, ConstantPool*, int) ()#20 0x0000200001c201c5 in ?? ()#21 0x0000200001c20170 in ?? ()#22 0x00002000002fe408 in ?? ()#23 0x0000200010fec980 in ?? ()#24 0x00002000002fe460 in ?? ()#25 0x0000200010fed4c8 in ?? ()#26 0x0000000000000000 in ?? ()
thread 0xffff80000313a040 VM Thread
#0 0x00000000403e236c in lockfree::mutex::unlock (this=this@entry=0xffffa00003553e90) at /usr/include/c++/8/bits/atomic_base.h:512#1 0x00000000403e36bc in std::lock_guard<lockfree::mutex>::~lock_guard (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:167#2 lock_guard_for_with_lock<lockfree::mutex>::~lock_guard_for_with_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at include/osv/mutex.h:88#3 semaphore::trywait (this=0xffffa00003553e80, units=units@entry=1) at core/semaphore.cc:64#4 0x00000000404656c1 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#5 0x0000100001702bf2 in PosixSemaphore::trywait() ()#6 0x000010000121aab4 in VM_HandshakeAllThreads::doit() ()#7 0x000010000198a8d7 in VM_Operation::evaluate() ()#8 0x0000100001990706 in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.64] ()#9 0x0000100001990e7a in VMThread::loop() ()#10 0x0000100001991073 in VMThread::run() ()#11 0x00001000019186ff in Thread::call_run() ()#12 0x000010000162e1be in thread_native_entry(Thread*) ()#13 0x000000004045edb6 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00002c3df00) at libc/pthread.cc:114#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#15 0x0000000040401117 in sched::thread_main_c (t=0xffff80000313a040) at arch/x64/arch-switch.hh:271#16 0x00000000403a71d3 in thread_main () at arch/x64/entry.S:113
thread 0xffff800003462040 Reference Handl
#0 sched::thread::switch_to (this=0xffffa00003b3b6f0, this@entry=0xffff800001fdd040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff794 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffc8c in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400372 in sched::thread::wait (this=this@entry=0xffff800003462040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (pred=..., mtx=<synthetic pointer>...) at include/osv/sched.hh:939#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x200011800870) at include/osv/wait_record.hh:46--Type <RET> for more, q to quit, c to continue without paging--#7 condvar::wait (this=0xffffa00003046680, user_mutex=0xffffa000030f9a40, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001637213 in os::PlatformEvent::park() ()#9 0x00001000015d48e8 in Monitor::IWait(Thread*, long) ()#10 0x00001000015d5aa6 in Monitor::wait(bool, long, bool) ()#11 0x000010000134cd7e in JVM_WaitForReferencePendingList ()#12 0x0000200001c0ffb7 in ?? ()#13 0x00002000118009f8 in ?? ()#14 0x00002000118009a8 in ?? ()#15 0x0000000000000000 in ?? ()
thread 0xffff800003583040 Finalizer
#0 sched::thread::switch_to (this=0xffffa00003b3b6f0, this@entry=0xffff80000442d040) at arch/x64/arch-switch.hh:108#1 0x00000000403ff794 in sched::cpu::reschedule_from_interrupt (this=0xffff800000013040, called_from_yield=called_from_yield@entry=false, preempt_after=..., preempt_after@entry=...) at core/sched.cc:339#2 0x00000000403ffc8c in sched::cpu::schedule () at include/osv/sched.hh:1310#3 0x0000000040400372 in sched::thread::wait (this=this@entry=0xffff800003583040) at core/sched.cc:1214#4 0x00000000403e38df in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (pred=..., mtx=<synthetic pointer>...) at include/osv/sched.hh:939#5 sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1077#6 waiter::wait (tmr=0x0, this=0x2000119006c0) at include/osv/wait_record.hh:46#7 condvar::wait (this=0xffffa00003046b00, user_mutex=0xffffa000030f9d00, tmr=<optimized out>) at core/condvar.cc:43#8 0x0000100001637213 in os::PlatformEvent::park() ()#9 0x00001000016002d5 in ObjectMonitor::wait(long, bool, Thread*) ()#10 0x00001000018c1132 in ObjectSynchronizer::wait(Handle, long, Thread*) ()#11 0x0000100001345778 in JVM_MonitorWait ()#12 0x0000200001c0ffb7 in ?? ()#13 0x0000200011900968 in ?? ()#14 0xffff900003240000 in ?? ()#15 0x0000200011900910 in ?? ()#16 0x00002000119008e8 in ?? ()#17 0x0000000000000000 in ?? ()There is still running thread VM Thread calling sem_trywait.
After more research, I have found about this JEP - http://openjdk.java.net/jeps/312 - which was gradually added to Java 10 and probably enhanced in next versions. As I understand in essence it replaces so called 'safepoints (global lock for "stop the world events") with a more granular mechanism - "thread local handshakes". Here is the fragment from the doc:
"A handshake operation is a callback that is executed for each JavaThread while that thread is in a safepoint safe state. The callback is executed either by the thread itself or by the VM thread while keeping the thread in a blocked state. The big difference between safepointing and handshaking is that the per thread operation will be performed on all threads as soon as possible and they will continue to execute as soon as it’s own operation is completed. If a JavaThread is known to be running, then a handshake can be performed with that single JavaThread as well."
I think the core of the logic is in https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/handshake.cpp and one of the methods is in this thread stacktrace:
#0 0x00000000403e236c in lockfree::mutex::unlock (this=this@entry=0xffffa00003553e90) at /usr/include/c++/8/bits/atomic_base.h:512#1 0x00000000403e36bc in std::lock_guard<lockfree::mutex>::~lock_guard (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/std_mutex.h:167#2 lock_guard_for_with_lock<lockfree::mutex>::~lock_guard_for_with_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at include/osv/mutex.h:88#3 semaphore::trywait (this=0xffffa00003553e80, units=units@entry=1) at core/semaphore.cc:64#4 0x00000000404656c1 in sem_trywait (s=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:342#5 0x0000100001702bf2 in PosixSemaphore::trywait() ()#6 0x000010000121aab4 in VM_HandshakeAllThreads::doit() ()#7 0x000010000198a8d7 in VM_Operation::evaluate() ()#8 0x0000100001990706 in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.64] ()#9 0x0000100001990e7a in VMThread::loop() ()#10 0x0000100001991073 in VMThread::run() ()#11 0x00001000019186ff in Thread::call_run() ()#12 0x000010000162e1be in thread_native_entry(Thread*) ()#13 0x000000004045edb6 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00002c3df00) at libc/pthread.cc:114#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297#15 0x0000000040401117 in sched::thread_main_c (t=0xffff80000313a040) at arch/x64/arch-switch.hh:271#16 0x00000000403a71d3 in thread_main () at arch/x64/entry.S:113Now, there is a way to switch off this new mechanism (which clearly is a default in Java 12) and get back to the "safepoints" one by adding this option: -XX:-ThreadLocalHandshakes
When I add this option the jetty app that used to hang, boots just fine.
So there is either a bug on Java side (but why it works on Linux?) or there is some bug on OSv side possibly due to some difference with Linux how some of the related calls are implemented.
Lastly, I have found this code in Java which seems, even more, to confirm that signal-related difference in sem_trywait implementation does not play a role here:
bool PosixSemaphore::trywait() { int ret;
do { ret = sem_trywait(&_semaphore); } while (ret != 0 && errno == EINTR);
assert_with_errno(ret == 0 || errno == EAGAIN, "trywait failed");
return ret == 0;}Waldek Kozaczuk
Aug 9, 2019, 10:23:33 AM8/9/19
to OSv Development
Mystery solved!!! One character makes a difference (it is is the 3rd bug I have found in last 2 weeks where the fix is a single line of code ;-).
It turns one we have code in OSv that uses sem_post() and sem_wait() (or semaphore.cc methods directly) but there only single-use of sem_trywait() in ACPI (https://github.com/cloudius-systems/osv/blob/master/drivers/acpi.cc#L135) which I am sure is even actively used. There are no direct unit tests around semaphore.cc or sem_* methods. And Java has just started using semaphores (sem_* functions; learned from an email on Hotspot OpenJDK mailing list).
So the is this bug in trywait():
diff --git a/core/semaphore.cc b/core/semaphore.cc
index 31fda35e..e2ccc801 100644
--- a/core/semaphore.cc
+++ b/core/semaphore.cc
@@ -62,7 +62,7 @@ bool semaphore::trywait(unsigned units)
{
bool ok = false;
WITH_LOCK(_mtx) {
- if (_val > units) {
+ if (_val >= units) {
_val -= units;
ok = true;
}
With this simple fix Java 12 starts properly.
Nadav Har'El
Aug 11, 2019, 9:08:42 AM8/11/19
to Waldek Kozaczuk, OSv Development
On Fri, Aug 9, 2019 at 5:23 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:
Mystery solved!!! One character makes a difference (it is is the 3rd bug I have found in last 2 weeks where the fix is a single line of code ;-).It turns one we have code in OSv that uses sem_post() and sem_wait() (or semaphore.cc methods directly) but there only single-use of sem_trywait() in ACPI (https://github.com/cloudius-systems/osv/blob/master/drivers/acpi.cc#L135) which I am sure is even actively used. There are no direct unit tests around semaphore.cc or sem_* methods. And Java has just started using semaphores (sem_* functions; learned from an email on Hotspot OpenJDK mailing list).So the is this bug in trywait():diff --git a/core/semaphore.cc b/core/semaphore.cc
index 31fda35e..e2ccc801 100644
--- a/core/semaphore.cc
+++ b/core/semaphore.cc
@@ -62,7 +62,7 @@ bool semaphore::trywait(unsigned units)
{
bool ok = false;
WITH_LOCK(_mtx) {
- if (_val > units) {
+ if (_val >= units) {
_val -= units;
ok = true;
}
With this simple fix Java 12 starts properly.
Fantastic. I just committed your patch with this fix. Thanks.
--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/d9098e04-ceca-4752-a476-6a31cdb0c5c8%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages