rr 5.4.0 Crashes

[Ben Stanley]

I've got a problem trying to debug a program using rr 5.4.0 on RHEL7.8.

I installed rr using the EPEL RPM package 5.4.0.

I have no idea how to proceed with diagnosing this problem. I did attempt to run gdb as suggested at the end of the debugging output, but unfortunately it did not enlighten me at all.

Is there any useful information that can be gleaned from the following debug output? Unfortunately, I will only be able to provide limited further information.

Ben.

[stanb@host DataDir]$ /usr/bin/rr record /home/stanb/TargetProgram

rr: Saving execution to trace directory `/home/stanb/.local/share/rr/TargetProgram-3'.

[FATAL /builddir/build/BUILD/rr-5.4.0/src/AutoRemoteSyscalls.cc:592:check_syscall_result()] 

 (task 28629 (rec:28629) at time 550)

 -> Assertion `false' failed to hold. Syscall mmap failed with errno ESRCH

Tail of trace dump:

{

  real_time:2442396.873524 global_time:530, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1069358

rax:0x1ee6000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1ec5000 rsi:0x1ee6000 rdi:0x1ee6000 rbp:0xc rsp:0x681ffd70 r8:0x0 r9:0x8008 r10:0x21000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x7a10 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1ec5000, length:0x21000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x21000 }

}

{

  real_time:2442396.873679 global_time:531, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1071843

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x28000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0xc00c r10:0x28000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4990 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.873715 global_time:532, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1071843

rax:0x1ee6000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x28000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0xc00c r10:0x28000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4990 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1ee6000, length:(nil), prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x0 }

}

{

  real_time:2442396.873767 global_time:533, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1071858

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1ee6000 rsi:0x1f0e000 rdi:0x1f0e000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0xc00c r10:0x28000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4990 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.873882 global_time:534, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1071858

rax:0x1f0e000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1ee6000 rsi:0x1f0e000 rdi:0x1f0e000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0xc00c r10:0x28000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4990 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1ee6000, length:0x28000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x28000 }

}

{

  real_time:2442396.874030 global_time:535, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1074630

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x48d0 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874063 global_time:536, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1074630

rax:0x1f0e000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x48d0 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1f0e000, length:(nil), prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x0 }

}

{

  real_time:2442396.874108 global_time:537, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1074645

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1f0e000 rsi:0x1f32000 rdi:0x1f32000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x48d0 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874203 global_time:538, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1074645

rax:0x1f32000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1f0e000 rsi:0x1f32000 rdi:0x1f32000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x48d0 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1f0e000, length:0x24000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x24000 }

}

{

  real_time:2442396.874312 global_time:539, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1077129

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4850 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874336 global_time:540, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1077129

rax:0x1f32000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24000 rsi:0x0 rdi:0x0 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4850 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1f32000, length:(nil), prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x0 }

}

{

  real_time:2442396.874370 global_time:541, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:28628, ticks:1077144

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1f32000 rsi:0x1f56000 rdi:0x1f56000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4850 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874442 global_time:542, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:28628, ticks:1077144

rax:0x1f56000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1f32000 rsi:0x1f56000 rdi:0x1f56000 rbp:0xc rsp:0x681ffd70 r8:0x1 r9:0x8008 r10:0x24000 r11:0x246 r12:0x7fd2bc14a4a0 r13:0x4850 r14:0xfff r15:0x7fd2bc147760 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x1f32000, length:0x24000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x24000 }

}

{

  real_time:2442396.874522 global_time:543, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:28628, ticks:1078511

rax:0xffffffffffffffda rbx:0xb6c554 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0xb6c554 rbp:0x41f802 rsp:0x7fff1c0a3b30 r8:0x1 r9:0x28 r10:0x3 r11:0x246 r12:0xb68bf0 r13:0x7fff1c0a3ec8 r14:0x7fff1c0a3e98 r15:0x5 rip:0x7fd2bc154230 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874544 global_time:544, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:28628, ticks:1078511

rax:0x0 rbx:0xb6c554 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0xb6c554 rbp:0x41f802 rsp:0x7fff1c0a3b30 r8:0x1 r9:0x28 r10:0x3 r11:0x246 r12:0xb68bf0 r13:0x7fff1c0a3ec8 r14:0x7fff1c0a3e98 r15:0x5 rip:0x7fd2bc154230 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874634 global_time:545, event:`PATCH_SYSCALL' tid:28628, ticks:1080910

rax:0x9 rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x801000 rdi:0x0 rbp:0x801000 rsp:0x7fff1c0a3c38 r8:0xffffffffffffffff r9:0x0 r10:0x20022 r11:0x246 r12:0x20022 r13:0x0 r14:0xffffffffffffffff r15:0x3 rip:0x7fd2bbe78d98 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fd2be148740 gs_base:0x0

  { tid:28628, addr:0x7fd2bdf7c04f, length:0x4f }

  { tid:28628, addr:0x7fd2bbe78d98, length:0x5 }

  { tid:28628, addr:0x7fd2bbe78d9d, length:0x3 }

}

{

  real_time:2442396.874674 global_time:546, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:28628, ticks:1080916

rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x801000 rdi:0x0 rbp:0x9 rsp:0x681ffd70 r8:0xffffffffffffffff r9:0x0 r10:0x20022 r11:0x246 r12:0x20022 r13:0x0 r14:0xffffffffffffffff r15:0x3 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fd2be148740 gs_base:0x0

}

{

  real_time:2442396.874702 global_time:547, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:28628, ticks:1080916

rax:0x7fd2b9630000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x801000 rdi:0x0 rbp:0x9 rsp:0x681ffd70 r8:0xffffffffffffffff r9:0x0 r10:0x20022 r11:0x246 r12:0x20022 r13:0x0 r14:0xffffffffffffffff r15:0x3 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fd2be148740 gs_base:0x0

  { map_file:"<ZERO>", addr:0x7fd2b9630000, length:0x801000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x801000 }

}

{

  real_time:2442396.874777 global_time:548, event:`PATCH_SYSCALL' tid:28628, ticks:1080996

rax:0xa rbx:0x7fd2bc369400 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x7fd2b9630000 rbp:0x0 rsp:0x7fff1c0a3c68 r8:0x7fd2b9e309c0 r9:0x7fd2b9e309c0 r10:0x3 r11:0x246 r12:0x1000 r13:0x801000 r14:0x0 r15:0x7fd2b9e30700 rip:0x7fd2bbe78e55 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fd2be148740 gs_base:0x0

  { tid:28628, addr:0x7fd2bdf7c09e, length:0x4f }

  { tid:28628, addr:0x7fd2bbe78e55, length:0x5 }

  { tid:28628, addr:0x7fd2bbe78e5a, length:0x3 }

}

{

  real_time:2442396.874839 global_time:549, event:`SYSCALLBUF_FLUSH' tid:28628, ticks:1081027

  { syscall:'mprotect', ret:0x0, size:0x10 }

}

=== Start rr backtrace:

/usr/bin/rr(_ZN2rr13dump_rr_stackEv+0x3b)[0x5b247b]

/usr/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x30d)[0x4a836d]

/usr/bin/rr(_ZN2rr21EmergencyDebugOstreamD2Ev+0x26f)[0x4b515f]

/usr/bin/rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0xe4)[0x4637c4]

/usr/bin/rr(_ZN2rr18AutoRemoteSyscalls22infallible_syscall_ptrIJNS_10remote_ptrIvEEmiiimEEES3_iDpT_+0xa1)[0x466d31]

/usr/bin/rr(_ZN2rr18AutoRemoteSyscalls23infallible_mmap_syscallENS_10remote_ptrIvEEmiiim+0x90)[0x4642d0]

/usr/bin/rr[0x4f2948]

/usr/bin/rr[0x4f3156]

/usr/bin/rr[0x518970]

/usr/bin/rr[0x52652d]

/usr/bin/rr(_ZN2rr19rec_prepare_syscallEPNS_10RecordTaskE+0xe7)[0x526657]

/usr/bin/rr(_ZN2rr13RecordSession21syscall_state_changedEPNS_10RecordTaskEPNS0_9StepStateE+0x4bf)[0x4df6df]

/usr/bin/rr(_ZN2rr13RecordSession11record_stepEv+0x4a1)[0x4e4b61]

/usr/bin/rr(_ZN2rr13RecordCommand3runERSt6vectorISsSaISsEE+0x13b5)[0x4d8c45]

/usr/bin/rr(main+0x1ee)[0x44e03e]

/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f8ff3c3f555]

/usr/bin/rr[0x44e19f]

=== End rr backtrace

Launch gdb with

  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:28629' /home/stanb/TargetProgram

[Robert O'Callahan]

Try building rr from master. If you still see the problem, when it occurs and rr is paused, attach gdb *to the rr process* and then give us a full stack trace from gdb.

Rob

--

Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht. Efil fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew hcihw, gninnigeb eht morf saw hcihw taht.