Can't build rr master - fail at 37%

[Ben Stanley]

I'm trying to build rr from git at commit 29048db7c24d78038135e8a2a9fe5c6acd4b548e .

I'm working on RHEL7.8.

[stanb@host Mozilla_rr]$ mkdir obj && cd obj
[stanb@host obj]$ cmake3 ../rr

...
[stanb@host obj]$ make -j8
...

Scanning dependencies of target reverse_step_threads

[ 37%] Linking C executable bin/ptrace_remote_unmap

[ 37%] Building C object CMakeFiles/reverse_step_threads.dir/src/test/reverse_step_threads.c.o

make[1]: *** [CMakeFiles/rr.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

[ 37%] Built target ptrace_remote_unmap

[ 37%] Linking C executable bin/fcntl_notify

[ 37%] Linking C executable bin/alarm_32

[ 37%] Built target fcntl_notify

[ 37%] Linking C executable bin/remove_latest_trace

[ 37%] Built target alarm_32

[ 37%] Linking C executable bin/watchpoint

[ 37%] Built target remove_latest_trace

[ 37%] Linking C executable bin/reverse_continue_multiprocess

[ 37%] Built target watchpoint

[ 37%] Built target reverse_continue_multiprocess

[ 37%] Linking C executable bin/reverse_step_threads

[ 37%] Built target reverse_step_threads

make: *** [all] Error 2

[stanb@host obj]$ 

Unfortunately I can't see what is going wrong here because the compiler messages are suppressed. What do I do next?

Thanks,

Ben Stanley.

[Robert O'Callahan]

They aren't normally suppressed. I'm not sure what's going on.

Rob

--

Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht. Efil fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew hcihw, gninnigeb eht morf saw hcihw taht.

[Kyle Huey]

Perhaps you're running with a high enough -jN that the error messages
are further back in the terminal output?

- Kyle

[Ben Stanley]

I made a CentOS 7.8 virtual machine for testing this a bit more. Built
it clean from scratch. Installed all the rr build dependencies as listed
in the rr spec file from the EPEL RPM package.

I downloaded the git repo and checked out master at commit
e9ec388d95ae05bc2eceb0361aea39510f08e17e, which is 6 commits ahead of
where I tested last time.

In this configuration I get a different error:

Scanning dependencies of target fxregs_32
[  5%] Building C object CMakeFiles/fxregs_32.dir/32/x86/fxregs.c.o
[  5%] Linking C executable bin/fxregs_32
CMakeFiles/fxregs_32.dir/32/x86/fxregs.c.o: In function `main':
/home/ben/rr/obj/32/x86/fxregs.c:64: undefined reference to
`__x86.get_pc_thunk.ax'

I have attached the complete build log. This is the only error that is
displayed.

If I check out commit 29048db7c24d78038135e8a2a9fe5c6acd4b548e (which is
the same commit that I tested before on RHEL 7.8), I obtain the same
result - the build stops at 5% on fxregs.c. It seems that there is
something wrong with this new CentOS 7.8 machine that I set up. I
haven't managed to reproduce the problem I had on the RHEL7.8 machine
yet - I expect that problem is being masked by the fxregs.c problem.

Any advice on fixing the fxregs.c problem?

Thanks,
Ben Stanley

[Ben Stanley]

TL;DR

I have now figured out how to build main latest rr-debugger on RHEL7.

I have submitted a pull request with one code change, replacing a direct call to memfd_create with syscall instead. Details of how I completed the build are included in the details below.

Running tests reveals many failures. I don't have numbers at this time. I request assistance to get these test cases sorted out, as I would rather like to have rr-debugger working properly on RHEL7.

I have not figured out how to fix the 32bit build on RHEL7. For now I have just disabled it. Unfortunately, 32bit remains important and I would also like to see that working.

Ben Stanley.

-------------------------------------------------

Further experiments reveal:

* Upgrading the VM to CentOS 7.9 latest patches does not fix the get_pc_thunk problem at 5% (glibc remains at glibc-2.17-324.el7_9.i686)

* Disabling 32 bit support in the cmake3 stage (as is done by the EPEL spec file) allows the build to proceed further to 26%..
cmake3 ../rr -Ddisable32bit=ON

Build then proceeds as follows:

[ben@rr-7 obj]$ cmake3 ../rr -Ddisable32bit=ON
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/lib64/ccache/c++
-- Check for working CXX compiler: /usr/lib64/ccache/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORTS_CXX14
-- Performing Test SUPPORTS_CXX14 - Failed
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Checking for module 'capnp'
--   Found capnp, version 0.5.3.1
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.8", minimum required is "3")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ben/rr/obj
[ben@rr-7 obj]$ make
Scanning dependencies of target ftrace_helper
[  0%] Building C object CMakeFiles/ftrace_helper.dir/src/ftrace/ftrace_helper.c.o
[  0%] Linking C executable bin/ftrace_helper
[  0%] Built target ftrace_helper
...
[ 26%] Building CXX object CMakeFiles/rr.dir/src/Task.cc.o
/home/ben/rr/rr/src/Task.cc: In member function ‘void rr::Task::write_zeroes(std::unique_ptr<rr::AutoRemoteSyscalls>*, rr::remote_ptr<void>, size_t)’:
/home/ben/rr/rr/src/Task.cc:3004:23: error: ‘make_unique’ was not declared in this scope
             *remote = make_unique<AutoRemoteSyscalls>(this);
                       ^
/home/ben/rr/rr/src/Task.cc:3004:53: error: expected primary-expression before ‘>’ token
             *remote = make_unique<AutoRemoteSyscalls>(this);
                                                     ^
make[2]: *** [CMakeFiles/rr.dir/src/Task.cc.o] Error 1

make[1]: *** [CMakeFiles/rr.dir/all] Error 2

make: *** [all] Error 2
[ben@rr-7 obj]$

It seems that C++14 is required to build rr at master commit e9ec388d95ae05bc2eceb0361aea39510f08e17e.

Checking out tag 5.4.0 and building on the same VM in the same way allows the build to complete.

C++14 support on CentOS 7.x is available through the devtoolset-7 package, from the Software Collections repository.

https://wiki.centos.org/AdditionalResources/Repositories/SCL
https://stackoverflow.com/questions/47214415/how-to-install-c14-on-rhel-7-4

yum install centos-release-scl
yum install devtoolset-7
scl enable devtoolset-7 bash
cmake3 ../rr -Ddisable32bit=ON
make

The cmake3 output is encouraging:

[ben@rr-7-8 obj]$ cmake3 ../rr -Ddisable32bit=ON
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- The ASM compiler identification is GNU
-- Found assembler: /opt/rh/devtoolset-7/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-7/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-7/root/usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-7/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-7/root/usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORTS_CXX14
-- Performing Test SUPPORTS_CXX14 - Success
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Checking for module 'capnp'
--   Found capnp, version 0.5.3.1
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.8", minimum required is "3")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ben/rr/obj
[ben@rr-7-8 obj]$

The build proceeds to 73% before any problems are found:

/home/ben/rr/rr/src/test/shared_monitor.c: In function ‘main’:
/home/ben/rr/rr/src/test/shared_monitor.c:6:12: error: implicit declaration of function ‘memfd_create’; did you mean ‘timerfd_create’? [-Werror=implicit-function-declaration]
   int fd = memfd_create("temp", 0);
            ^~~~~~~~~~~~
            timerfd_create

Note that RHEL7.8 is based on kernel 3.10.0 which apparently doesn't have memfd_create (introduced linux 3.17 and glibc 2.27).

So it looks like I can't have latest rr on RHEL7.8 unless it can be built without requiring memfd_create.

It looks like this has been discussed before.
https://github.com/rr-debugger/rr/issues/2442

and fixed

https://github.com/rr-debugger/rr/pull/2443 commit abd344288878c9b4046e0b8664927992947a46eb

Applying the same fix to rr/src/test/shared_monitor.c allows the build to complete 100%.

Running tests is a disaster though. Every test appears to fail. It appears that I need to do some work to make rr work in this VM.

https://github.com/rr-debugger/rr/wiki/Building-And-Installing

I have now fixed kernel.perf_event_paranoid=1 (was =2).

I have set <cpu mode='host-passthrough'/> in the domain XML definition and restarted the virtual machine. Let me shut it down completely and start it again to ensure that took effect, as rebooting the virtual machine doesn't allow XML configuration changes to take effect.

I had to install perf (wasn't installed).

https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-system

[ben@rr obj]$ perf stat -e br_inst_retired.conditional true

 Performance counter stats for 'true':

            68,190      br_inst_retired.conditional                                  

       0.016617908 seconds time elapsed

       0.000000000 seconds user
       0.016775000 seconds sys


[ben@rr obj]$

Running tests reveals a significant number of failures. The log contains many stack dumps and is quite large. I'm not in a position to enumerate the pass/fails or to attempt to repair broken test cases. However, I would rather like rr-debugger to work robustly on RHEL7.

Ben.

[Kyle Huey]

Can you copy/paste the list of failing tests? rr contains a lot of
tests for various kernel APIs (e.g. that memfd_create issue you ran
into) that aren't going to pass on an old kernel. But there are other
tests that, if failing, point at something much more fundamental being
broken (the `alarm` test comes to mind). Seeing the list of what fails
would be helpful.

- Kyle

> --
> You received this message because you are subscribed to the Google Groups "rr-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rr-devel+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/rr-devel/76f335c7-e1ea-8ba2-b92b-52fc72fa0dd4%40gmail.com.

[Ben Stanley]

Tests only just finished.

It looks like something fundamental is broken.

----------------------------------------------

19% tests passed, 1053 tests failed out of 1299

Total Test time (real) = 38265.14 sec

The following tests FAILED:
      2 - 64bit_child (Failed)
      3 - 64bit_child-no-syscallbuf (Failed)
      4 - _llseek (Failed)
      5 - _llseek-no-syscallbuf (Failed)
      8 - accept (Failed)
      9 - accept-no-syscallbuf (Failed)
     10 - acct (Failed)
     11 - acct-no-syscallbuf (Failed)
     12 - adjtimex (Failed)
     13 - adjtimex-no-syscallbuf (Failed)
     14 - aio (Failed)
     15 - aio-no-syscallbuf (Failed)
     16 - alarm (Failed)
     17 - alarm-no-syscallbuf (Failed)
     18 - alarm2 (Failed)
     19 - alarm2-no-syscallbuf (Failed)
     20 - alsa_ioctl (Failed)
     21 - alsa_ioctl-no-syscallbuf (Failed)
     22 - arch_prctl (Failed)
     23 - arch_prctl-no-syscallbuf (Failed)
     24 - async_segv_ignored (Failed)
     25 - async_segv_ignored-no-syscallbuf (Failed)
     26 - at_threadexit (Failed)
     27 - at_threadexit-no-syscallbuf (Failed)
     28 - bad_ip (Failed)
     29 - bad_ip-no-syscallbuf (Failed)
     30 - bad_syscall (Failed)
     31 - bad_syscall-no-syscallbuf (Failed)
     32 - barrier (Failed)
     33 - barrier-no-syscallbuf (Failed)
     34 - big_buffers (Failed)
     35 - big_buffers-no-syscallbuf (Failed)
     36 - block (Failed)
     37 - block-no-syscallbuf (Failed)
     38 - block_open (Failed)
     39 - block_open-no-syscallbuf (Failed)
     40 - bpf (Failed)
     41 - bpf-no-syscallbuf (Failed)
     42 - brk (Failed)
     43 - brk-no-syscallbuf (Failed)
     44 - brk2 (Failed)
     45 - brk2-no-syscallbuf (Failed)
     46 - capget (Failed)
     47 - capget-no-syscallbuf (Failed)
     48 - chew_cpu (Failed)
     49 - chew_cpu-no-syscallbuf (Failed)
     50 - x86/chew_cpu_cpuid (Failed)
     51 - x86/chew_cpu_cpuid-no-syscallbuf (Failed)
     52 - chmod (Failed)
     53 - chmod-no-syscallbuf (Failed)
     54 - chown (Failed)
     55 - chown-no-syscallbuf (Failed)
     56 - clock (Failed)
     57 - clock-no-syscallbuf (Failed)
     58 - clock_adjtime (Failed)
     59 - clock_adjtime-no-syscallbuf (Failed)
     60 - clock_nanosleep (Failed)
     61 - clock_nanosleep-no-syscallbuf (Failed)
     62 - clock_time64 (Failed)
     63 - clock_time64-no-syscallbuf (Failed)
     64 - clone (Failed)
     65 - clone-no-syscallbuf (Failed)
     66 - clone_bad_stack (Failed)
     67 - clone_bad_stack-no-syscallbuf (Failed)
     68 - clone_bad_tls (Failed)
     69 - clone_bad_tls-no-syscallbuf (Failed)
     70 - clone_cleartid_coredump (Timeout)
     71 - clone_cleartid_coredump-no-syscallbuf (Failed)
     72 - clone_fail (Failed)
     73 - clone_fail-no-syscallbuf (Failed)
     74 - clone_file_range (Failed)
     75 - clone_file_range-no-syscallbuf (Failed)
     76 - clone_immediate_exit (Failed)
     77 - clone_immediate_exit-no-syscallbuf (Failed)
     78 - clone_newflags (Failed)
     79 - clone_newflags-no-syscallbuf (Failed)
     80 - clone_parent (Failed)
     81 - clone_parent-no-syscallbuf (Failed)
     82 - clone_untraced (Timeout)
     83 - clone_untraced-no-syscallbuf (Timeout)
     84 - clone_vfork_pidfd (Failed)
     85 - clone_vfork_pidfd-no-syscallbuf (Failed)
     86 - cloned_sigmask (Failed)
     87 - cloned_sigmask-no-syscallbuf (Failed)
     88 - constructor (Failed)
     89 - constructor-no-syscallbuf (Failed)
     90 - copy_file_range (Failed)
     91 - copy_file_range-no-syscallbuf (Failed)
     92 - x86/cpuid_same_state (Failed)
     93 - x86/cpuid_same_state-no-syscallbuf (Failed)
     94 - creat_address_not_truncated (Failed)
     95 - creat_address_not_truncated-no-syscallbuf (Failed)
     96 - x86/cross_arch (Failed)
     97 - x86/cross_arch-no-syscallbuf (Failed)
     98 - cwd_inaccessible (Failed)
     99 - cwd_inaccessible-no-syscallbuf (Failed)
    100 - daemon (Failed)
    101 - daemon-no-syscallbuf (Failed)
    102 - desched_blocking_poll (Failed)
    103 - desched_blocking_poll-no-syscallbuf (Failed)
    104 - detach_state (Failed)
    105 - detach_state-no-syscallbuf (Failed)
    106 - detach_threads (Failed)
    107 - detach_threads-no-syscallbuf (Failed)
    108 - deterministic_sigsys (Failed)
    109 - deterministic_sigsys-no-syscallbuf (Failed)
    110 - direct (Failed)
    111 - direct-no-syscallbuf (Failed)
    112 - dup (Failed)
    113 - dup-no-syscallbuf (Failed)
    114 - doublesegv (Failed)
    115 - doublesegv-no-syscallbuf (Failed)
    116 - epoll_create (Failed)
    117 - epoll_create-no-syscallbuf (Failed)
    118 - epoll_create1 (Failed)
    119 - epoll_create1-no-syscallbuf (Failed)
    120 - epoll_pwait_eintr_sigmask (Failed)
    121 - epoll_pwait_eintr_sigmask-no-syscallbuf (Failed)
    122 - eventfd (Failed)
    123 - eventfd-no-syscallbuf (Failed)
    124 - exec_flags (Failed)
    125 - exec_flags-no-syscallbuf (Failed)
    126 - exec_no_env (Failed)
    127 - exec_no_env-no-syscallbuf (Failed)
    128 - exec_self (Failed)
    129 - exec_self-no-syscallbuf (Failed)
    130 - exec_from_main_thread (Failed)
    131 - exec_from_main_thread-no-syscallbuf (Failed)
    132 - exec_from_other_thread (Failed)
    133 - exec_from_other_thread-no-syscallbuf (Failed)
    134 - exec_stopsig (Failed)
    135 - exec_stopsig-no-syscallbuf (Failed)
    136 - exit_with_syscallbuf_signal (Failed)
    137 - exit_with_syscallbuf_signal-no-syscallbuf (Failed)
    138 - fadvise (Failed)
    139 - fadvise-no-syscallbuf (Failed)
    140 - fanotify (Failed)
    141 - fanotify-no-syscallbuf (Failed)
    142 - fatal_sigsegv_thread (Failed)
    143 - fatal_sigsegv_thread-no-syscallbuf (Failed)
    144 - x86/fault_in_code_page (Failed)
    145 - x86/fault_in_code_page-no-syscallbuf (Failed)
    146 - fcntl_dupfd (Failed)
    147 - fcntl_dupfd-no-syscallbuf (Failed)
    148 - fcntl_misc (Failed)
    149 - fcntl_misc-no-syscallbuf (Failed)
    150 - fcntl_notify (Failed)
    151 - fcntl_notify-no-syscallbuf (Failed)
    152 - fcntl_owner_ex (Failed)
    153 - fcntl_owner_ex-no-syscallbuf (Failed)
    154 - fcntl_rw_hints (Failed)
    155 - fcntl_rw_hints-no-syscallbuf (Failed)
    156 - fcntl_seals (Failed)
    157 - fcntl_seals-no-syscallbuf (Failed)
    158 - fcntl_sig (Failed)
    159 - fcntl_sig-no-syscallbuf (Failed)
    160 - fd_cleanup (Failed)
    161 - fd_cleanup-no-syscallbuf (Failed)
    162 - fd_tracking_across_threads (Failed)
    163 - fd_tracking_across_threads-no-syscallbuf (Failed)
    164 - fds_clean (Failed)
    165 - fds_clean-no-syscallbuf (Failed)
    166 - flock (Failed)
    167 - flock-no-syscallbuf (Failed)
    168 - flock_ofd (Failed)
    169 - flock_ofd-no-syscallbuf (Failed)
    170 - flock2 (Failed)
    171 - flock2-no-syscallbuf (Failed)
    172 - fork_brk (Failed)
    173 - fork_brk-no-syscallbuf (Failed)
    174 - fork_child_crash (Failed)
    175 - fork_child_crash-no-syscallbuf (Failed)
    176 - fork_many (Failed)
    177 - fork_many-no-syscallbuf (Failed)
    178 - futex_exit_race (Failed)
    179 - futex_exit_race-no-syscallbuf (Failed)
    180 - futex_exit_race_sigsegv (Failed)
    181 - futex_exit_race_sigsegv-no-syscallbuf (Failed)
    182 - futex_pi (Failed)
    183 - futex_pi-no-syscallbuf (Failed)
    184 - futex_priorities (Failed)
    185 - futex_priorities-no-syscallbuf (Failed)
    186 - futex_requeue (Failed)
    187 - futex_requeue-no-syscallbuf (Failed)
    188 - gcrypt_rdrand (Failed)
    189 - gcrypt_rdrand-no-syscallbuf (Failed)
    190 - getcpu (Failed)
    191 - getcpu-no-syscallbuf (Failed)
    192 - getgroups (Failed)
    193 - getgroups-no-syscallbuf (Failed)
    194 - getpwnam (Failed)
    195 - getpwnam-no-syscallbuf (Failed)
    196 - getrandom (Failed)
    197 - getrandom-no-syscallbuf (Failed)
    198 - setitimer (Failed)
    199 - setitimer-no-syscallbuf (Failed)
    200 - getsid (Failed)
    201 - getsid-no-syscallbuf (Failed)
    202 - gettimeofday (Failed)
    203 - gettimeofday-no-syscallbuf (Failed)
    204 - grandchild_threads (Failed)
    205 - grandchild_threads-no-syscallbuf (Failed)
    206 - grandchild_threads_main_running (Failed)
    207 - grandchild_threads_main_running-no-syscallbuf (Failed)
    208 - grandchild_threads_thread_running (Failed)
    209 - grandchild_threads_thread_running-no-syscallbuf (Failed)
    210 - grandchild_threads_parent_alive (Failed)
    211 - grandchild_threads_parent_alive-no-syscallbuf (Failed)
    212 - x86/hle (Failed)
    213 - x86/hle-no-syscallbuf (Failed)
    214 - inotify (Failed)
    215 - inotify-no-syscallbuf (Failed)
    216 - int3 (Failed)
    217 - int3-no-syscallbuf (Failed)
    218 - intr_futex_wait_restart (Failed)
    219 - intr_futex_wait_restart-no-syscallbuf (Failed)
    220 - intr_poll (Failed)
    221 - intr_poll-no-syscallbuf (Failed)
    222 - intr_ppoll (Failed)
    223 - intr_ppoll-no-syscallbuf (Failed)
    224 - intr_pselect (Failed)
    225 - intr_pselect-no-syscallbuf (Failed)
    226 - intr_read_no_restart (Failed)
    227 - intr_read_no_restart-no-syscallbuf (Failed)
    228 - intr_read_restart (Failed)
    229 - intr_read_restart-no-syscallbuf (Failed)
    230 - intr_sleep (Failed)
    231 - intr_sleep-no-syscallbuf (Failed)
    232 - intr_sleep_no_restart (Failed)
    233 - intr_sleep_no_restart-no-syscallbuf (Failed)
    234 - invalid_exec (Failed)
    235 - invalid_exec-no-syscallbuf (Failed)
    236 - invalid_fcntl (Failed)
    237 - invalid_fcntl-no-syscallbuf (Failed)
    238 - invalid_ioctl (Failed)
    239 - invalid_ioctl-no-syscallbuf (Failed)
    240 - io (Failed)
    241 - io-no-syscallbuf (Failed)
    242 - ioctl (Failed)
    243 - ioctl-no-syscallbuf (Failed)
    244 - ioctl_fs (Failed)
    245 - ioctl_fs-no-syscallbuf (Failed)
    246 - ioctl_pty (Failed)
    247 - ioctl_pty-no-syscallbuf (Failed)
    248 - ioctl_sg (Failed)
    249 - ioctl_sg-no-syscallbuf (Failed)
    250 - ioctl_tty (Failed)
    251 - ioctl_tty-no-syscallbuf (Failed)
    252 - join_threads (Failed)
    253 - join_threads-no-syscallbuf (Failed)
    254 - joystick (Failed)
    255 - joystick-no-syscallbuf (Failed)
    256 - kcmp (Failed)
    257 - kcmp-no-syscallbuf (Failed)
    258 - keyctl (Failed)
    259 - keyctl-no-syscallbuf (Failed)
    260 - kill_newborn (Failed)
    261 - kill_newborn-no-syscallbuf (Failed)
    262 - kill_ptracee (Failed)
    263 - kill_ptracee-no-syscallbuf (Failed)
    264 - large_hole (Failed)
    265 - large_hole-no-syscallbuf (Failed)
    266 - large_write_deadlock (Failed)
    267 - large_write_deadlock-no-syscallbuf (Failed)
    268 - legacy_ugid (Failed)
    269 - legacy_ugid-no-syscallbuf (Failed)
    270 - x86/lsl (Failed)
    271 - x86/lsl-no-syscallbuf (Failed)
    272 - madvise (Failed)
    273 - madvise-no-syscallbuf (Failed)
    274 - madvise_free (Failed)
    275 - madvise_free-no-syscallbuf (Failed)
    276 - madvise_wipeonfork (Failed)
    277 - madvise_wipeonfork-no-syscallbuf (Failed)
    278 - map_fixed (Failed)
    279 - map_fixed-no-syscallbuf (Failed)
    280 - map_shared_syscall (Failed)
    281 - map_shared_syscall-no-syscallbuf (Failed)
    282 - membarrier (Failed)
    283 - membarrier-no-syscallbuf (Failed)
    284 - memfd_create (Failed)
    285 - memfd_create-no-syscallbuf (Failed)
    286 - memfd_create_shared (Failed)
    287 - memfd_create_shared-no-syscallbuf (Failed)
    288 - memfd_create_shared_huge (Failed)
    289 - memfd_create_shared_huge-no-syscallbuf (Failed)
    290 - mincore (Failed)
    291 - mincore-no-syscallbuf (Failed)
    292 - mknod (Failed)
    293 - mknod-no-syscallbuf (Failed)
    294 - mlock (Failed)
    295 - mlock-no-syscallbuf (Failed)
    296 - mmap_adjacent_to_rr_usage (Failed)
    297 - mmap_adjacent_to_rr_usage-no-syscallbuf (Failed)
    298 - mmap_private (Failed)
    299 - mmap_private-no-syscallbuf (Failed)
    300 - mmap_private_grow_under_map (Failed)
    301 - mmap_private_grow_under_map-no-syscallbuf (Failed)
    302 - mmap_ro (Failed)
    303 - mmap_ro-no-syscallbuf (Failed)
    304 - mmap_self_maps_shared (Failed)
    305 - mmap_self_maps_shared-no-syscallbuf (Failed)
    306 - mmap_shared (Failed)
    307 - mmap_shared-no-syscallbuf (Failed)
    308 - mmap_shared_dev_zero (Failed)
    309 - mmap_shared_dev_zero-no-syscallbuf (Failed)
    310 - mmap_shared_grow (Failed)
    311 - mmap_shared_grow-no-syscallbuf (Failed)
    312 - mmap_shared_grow_under_map (Failed)
    313 - mmap_shared_grow_under_map-no-syscallbuf (Failed)
    314 - mmap_shared_multiple (Failed)
    315 - mmap_shared_multiple-no-syscallbuf (Failed)
    316 - mmap_shared_subpage (Failed)
    317 - mmap_shared_subpage-no-syscallbuf (Failed)
    318 - mmap_shared_write (Failed)
    319 - mmap_shared_write-no-syscallbuf (Failed)
    320 - mmap_shared_write_fork (Failed)
    321 - mmap_shared_write_fork-no-syscallbuf (Failed)
    322 - mmap_short_file (Failed)
    323 - mmap_short_file-no-syscallbuf (Failed)
    324 - mmap_write_complex (Failed)
    325 - mmap_write_complex-no-syscallbuf (Failed)
    326 - mmap_zero_size_fd (Failed)
    327 - mmap_zero_size_fd-no-syscallbuf (Failed)
    328 - x86/modify_ldt (Failed)
    329 - x86/modify_ldt-no-syscallbuf (Failed)
    330 - mount_ns_exec (Failed)
    331 - mount_ns_exec-no-syscallbuf (Failed)
    332 - mount_ns_exec2 (Failed)
    333 - mount_ns_exec2-no-syscallbuf (Failed)
    334 - mprotect (Failed)
    335 - mprotect-no-syscallbuf (Failed)
    336 - mprotect_heterogenous (Failed)
    337 - mprotect_heterogenous-no-syscallbuf (Failed)
    338 - mprotect_none (Failed)
    339 - mprotect_none-no-syscallbuf (Failed)
    340 - mprotect_stack (Failed)
    341 - mprotect_stack-no-syscallbuf (Failed)
    342 - mq (Failed)
    343 - mq-no-syscallbuf (Failed)
    344 - mremap (Failed)
    345 - mremap-no-syscallbuf (Failed)
    346 - mremap_after_coalesce (Failed)
    347 - mremap_after_coalesce-no-syscallbuf (Failed)
    348 - mremap_grow (Failed)
    349 - mremap_grow-no-syscallbuf (Failed)
    350 - mremap_grow_shared (Failed)
    351 - mremap_grow_shared-no-syscallbuf (Failed)
    352 - mremap_non_page_size (Failed)
    353 - mremap_non_page_size-no-syscallbuf (Failed)
    354 - mremap_overwrite (Failed)
    355 - mremap_overwrite-no-syscallbuf (Failed)
    356 - mremap_private_grow_under_map (Failed)
    357 - mremap_private_grow_under_map-no-syscallbuf (Failed)
    358 - mremap_shrink (Failed)
    359 - mremap_shrink-no-syscallbuf (Failed)
    360 - msg (Failed)
    361 - msg-no-syscallbuf (Failed)
    362 - msg_trunc (Failed)
    363 - msg_trunc-no-syscallbuf (Failed)
    364 - msync (Failed)
    365 - msync-no-syscallbuf (Failed)
    366 - mtio (Failed)
    367 - mtio-no-syscallbuf (Failed)
    368 - multiple_pending_signals (Failed)
    369 - multiple_pending_signals-no-syscallbuf (Failed)
    370 - multiple_pending_signals_sequential (Failed)
    371 - multiple_pending_signals_sequential-no-syscallbuf (Failed)
    372 - munmap_segv (Failed)
    373 - munmap_segv-no-syscallbuf (Failed)
    374 - munmap_discontinuous (Failed)
    375 - munmap_discontinuous-no-syscallbuf (Failed)
    376 - nanosleep (Failed)
    377 - nanosleep-no-syscallbuf (Failed)
    378 - netfilter (Failed)
    379 - netfilter-no-syscallbuf (Failed)
    380 - netlink_mmap_disable (Failed)
    381 - netlink_mmap_disable-no-syscallbuf (Failed)
    382 - no_mask_timeslice (Failed)
    383 - no_mask_timeslice-no-syscallbuf (Failed)
    384 - nscd (Failed)
    385 - nscd-no-syscallbuf (Failed)
    386 - numa (Failed)
    387 - numa-no-syscallbuf (Failed)
    388 - x86/old_fork (Failed)
    389 - x86/old_fork-no-syscallbuf (Failed)
    390 - orphan_process (Failed)
    391 - orphan_process-no-syscallbuf (Failed)
    392 - packet_mmap_disable (Failed)
    393 - packet_mmap_disable-no-syscallbuf (Failed)
    394 - pause (Failed)
    395 - pause-no-syscallbuf (Failed)
    396 - perf_event (Failed)
    397 - perf_event-no-syscallbuf (Failed)
    398 - personality (Failed)
    399 - personality-no-syscallbuf (Failed)
    400 - pid_ns_reap (Failed)
    401 - pid_ns_reap-no-syscallbuf (Failed)
    402 - pid_ns_segv (Failed)
    403 - pid_ns_segv-no-syscallbuf (Failed)
    404 - pidfd (Failed)
    405 - pidfd-no-syscallbuf (Failed)
    406 - poll_sig_race (Failed)
    407 - poll_sig_race-no-syscallbuf (Failed)
    408 - ppoll (Failed)
    409 - ppoll-no-syscallbuf (Failed)
    410 - prctl (Failed)
    411 - prctl-no-syscallbuf (Failed)
    412 - prctl_caps (Failed)
    413 - prctl_caps-no-syscallbuf (Failed)
    414 - prctl_deathsig (Failed)
    415 - prctl_deathsig-no-syscallbuf (Failed)
    416 - prctl_name (Failed)
    417 - prctl_name-no-syscallbuf (Failed)
    418 - prctl_short_name (Failed)
    419 - prctl_short_name-no-syscallbuf (Failed)
    420 - prctl_speculation_ctrl (Failed)
    421 - prctl_speculation_ctrl-no-syscallbuf (Failed)
    422 - x86/prctl_tsc (Failed)
    423 - x86/prctl_tsc-no-syscallbuf (Failed)
    424 - privileged_net_ioctl (Failed)
    425 - privileged_net_ioctl-no-syscallbuf (Failed)
    426 - proc_fds (Failed)
    427 - proc_fds-no-syscallbuf (Failed)
    428 - proc_mem (Failed)
    429 - proc_mem-no-syscallbuf (Failed)
    430 - protect_rr_fds (Failed)
    431 - protect_rr_fds-no-syscallbuf (Failed)
    432 - prw (Failed)
    433 - prw-no-syscallbuf (Failed)
    434 - pthread_condvar_locking (Failed)
    435 - pthread_condvar_locking-no-syscallbuf (Failed)
    436 - pthread_mutex_timedlock (Failed)
    437 - pthread_mutex_timedlock-no-syscallbuf (Failed)
    438 - pthread_pi_mutex (Failed)
    439 - pthread_pi_mutex-no-syscallbuf (Failed)
    440 - pthread_rwlocks (Failed)
    441 - pthread_rwlocks-no-syscallbuf (Failed)
    442 - x86/ptrace (Failed)
    443 - x86/ptrace-no-syscallbuf (Failed)
    444 - ptrace_attach_null_status (Failed)
    445 - ptrace_attach_null_status-no-syscallbuf (Failed)
    446 - ptrace_attach_running (Failed)
    447 - ptrace_attach_running-no-syscallbuf (Failed)
    448 - ptrace_attach_sleeping (Failed)
    449 - ptrace_attach_sleeping-no-syscallbuf (Failed)
    450 - ptrace_attach_stopped (Failed)
    451 - ptrace_attach_stopped-no-syscallbuf (Failed)
    452 - ptrace_attach_thread_running (Failed)
    453 - ptrace_attach_thread_running-no-syscallbuf (Failed)
    454 - ptrace_breakpoint (Failed)
    455 - ptrace_breakpoint-no-syscallbuf (Failed)
    456 - ptrace_change_patched_syscall (Failed)
    457 - ptrace_change_patched_syscall-no-syscallbuf (Failed)
    458 - x86/ptrace_debug_regs (Failed)
    459 - x86/ptrace_debug_regs-no-syscallbuf (Failed)
    460 - ptrace_exec (Failed)
    461 - ptrace_exec-no-syscallbuf (Failed)
    462 - x86/ptrace_exec32 (Failed)
    463 - x86/ptrace_exec32-no-syscallbuf (Failed)
    464 - ptrace_kill_grandtracee (Failed)
    465 - ptrace_kill_grandtracee-no-syscallbuf (Failed)
    466 - x86/ptrace_tls (Failed)
    467 - x86/ptrace_tls-no-syscallbuf (Failed)
    468 - ptrace_seize (Failed)
    469 - ptrace_seize-no-syscallbuf (Failed)
    470 - ptrace_sigchld_blocked (Failed)
    471 - ptrace_sigchld_blocked-no-syscallbuf (Failed)
    472 - ptrace_signals (Failed)
    473 - ptrace_signals-no-syscallbuf (Failed)
    474 - ptrace_singlestep (Failed)
    475 - ptrace_singlestep-no-syscallbuf (Failed)
    476 - ptrace_syscall (Failed)
    477 - ptrace_syscall-no-syscallbuf (Failed)
    478 - ptrace_syscall_clone_untraced (Failed)
    479 - ptrace_syscall_clone_untraced-no-syscallbuf (Failed)
    480 - ptrace_sysemu (Failed)
    481 - ptrace_sysemu-no-syscallbuf (Failed)
    482 - ptrace_sysemu_syscall (Failed)
    483 - ptrace_sysemu_syscall-no-syscallbuf (Failed)
    484 - ptrace_trace_clone (Failed)
    485 - ptrace_trace_clone-no-syscallbuf (Failed)
    486 - ptrace_trace_exit (Failed)
    487 - ptrace_trace_exit-no-syscallbuf (Failed)
    488 - ptrace_traceme (Failed)
    489 - ptrace_traceme-no-syscallbuf (Failed)
    490 - ptracer_death (Failed)
    491 - ptracer_death-no-syscallbuf (Failed)
    492 - ptracer_death_multithread (Failed)
    493 - ptracer_death_multithread-no-syscallbuf (Failed)
    494 - ptracer_death_multithread_peer (Failed)
    495 - ptracer_death_multithread_peer-no-syscallbuf (Failed)
    496 - quotactl (Failed)
    497 - quotactl-no-syscallbuf (Failed)
    498 - x86/rdtsc (Failed)
    499 - x86/rdtsc-no-syscallbuf (Failed)
    500 - read_nothing (Failed)
    501 - read_nothing-no-syscallbuf (Failed)
    502 - readdir (Failed)
    503 - readdir-no-syscallbuf (Failed)
    504 - read_large (Failed)
    505 - read_large-no-syscallbuf (Failed)
    506 - read_oversize (Failed)
    507 - read_oversize-no-syscallbuf (Failed)
    508 - readlink (Failed)
    509 - readlink-no-syscallbuf (Failed)
    510 - readlinkat (Failed)
    511 - readlinkat-no-syscallbuf (Failed)
    512 - readv (Failed)
    513 - readv-no-syscallbuf (Failed)
    514 - record_replay_subject (Failed)
    515 - record_replay_subject-no-syscallbuf (Failed)
    516 - recvfrom (Failed)
    517 - recvfrom-no-syscallbuf (Failed)
    518 - redzone_integrity (Failed)
    519 - redzone_integrity-no-syscallbuf (Failed)
    520 - rename (Failed)
    521 - rename-no-syscallbuf (Failed)
    522 - rlimit (Failed)
    523 - rlimit-no-syscallbuf (Failed)
    524 - robust_futex (Failed)
    525 - robust_futex-no-syscallbuf (Failed)
    526 - rusage (Failed)
    527 - rusage-no-syscallbuf (Failed)
    528 - samask (Failed)
    529 - samask-no-syscallbuf (Failed)
    530 - save_data_fd (Failed)
    531 - save_data_fd-no-syscallbuf (Failed)
    532 - sched_attr (Failed)
    533 - sched_attr-no-syscallbuf (Failed)
    534 - sched_setaffinity (Failed)
    535 - sched_setaffinity-no-syscallbuf (Failed)
    536 - sched_setparam (Failed)
    537 - sched_setparam-no-syscallbuf (Failed)
    538 - sched_yield (Failed)
    539 - sched_yield-no-syscallbuf (Failed)
    540 - sched_yield_to_lower_priority (Failed)
    541 - sched_yield_to_lower_priority-no-syscallbuf (Failed)
    542 - scm_rights (Failed)
    543 - scm_rights-no-syscallbuf (Failed)
    544 - scratch_read (Failed)
    545 - scratch_read-no-syscallbuf (Failed)
    546 - seccomp (Failed)
    547 - seccomp-no-syscallbuf (Failed)
    548 - seccomp_clone_fail (Failed)
    549 - seccomp_clone_fail-no-syscallbuf (Failed)
    550 - seccomp_desched (Failed)
    551 - seccomp_desched-no-syscallbuf (Failed)
    552 - seccomp_kill_exit (Failed)
    553 - seccomp_kill_exit-no-syscallbuf (Failed)
    554 - seccomp_null (Failed)
    555 - seccomp_null-no-syscallbuf (Failed)
    556 - seccomp_sigsys_args (Failed)
    557 - seccomp_sigsys_args-no-syscallbuf (Failed)
    558 - seccomp_sigsys_sigtrap (Failed)
    559 - seccomp_sigsys_sigtrap-no-syscallbuf (Failed)
    560 - seccomp_sigsys_syscallbuf (Failed)
    561 - seccomp_sigsys_syscallbuf-no-syscallbuf (Failed)
    562 - seccomp_tsync (Failed)
    563 - seccomp_tsync-no-syscallbuf (Failed)
    564 - seccomp_veto_exec (Failed)
    565 - seccomp_veto_exec-no-syscallbuf (Failed)
    566 - self_shebang (Failed)
    567 - self_shebang-no-syscallbuf (Failed)
    568 - self_sigint (Failed)
    569 - self_sigint-no-syscallbuf (Failed)
    570 - sem (Failed)
    571 - sem-no-syscallbuf (Failed)
    572 - send_block (Failed)
    573 - send_block-no-syscallbuf (Failed)
    574 - sendfile (Failed)
    575 - sendfile-no-syscallbuf (Failed)
    576 - set_ptracer (Failed)
    577 - set_ptracer-no-syscallbuf (Failed)
    578 - set_tid_address (Failed)
    579 - set_tid_address-no-syscallbuf (Failed)
    580 - setgid (Failed)
    581 - setgid-no-syscallbuf (Failed)
    582 - setgroups (Failed)
    583 - setgroups-no-syscallbuf (Failed)
    584 - setsid (Failed)
    585 - setsid-no-syscallbuf (Failed)
    586 - setuid (Failed)
    587 - setuid-no-syscallbuf (Failed)
    588 - shared_exec (Failed)
    589 - shared_exec-no-syscallbuf (Failed)
    590 - shared_monitor (Failed)
    591 - shared_monitor-no-syscallbuf (Failed)
    592 - shared_write (Failed)
    593 - shared_write-no-syscallbuf (Failed)
    594 - shm (Failed)
    595 - shm-no-syscallbuf (Failed)
    596 - shm_unmap (Failed)
    597 - shm_unmap-no-syscallbuf (Failed)
    598 - sigaction_old (Failed)
    599 - sigaction_old-no-syscallbuf (Failed)
    600 - sigaltstack (Failed)
    601 - sigaltstack-no-syscallbuf (Failed)
    602 - sigchld_interrupt_signal (Failed)
    603 - sigchld_interrupt_signal-no-syscallbuf (Failed)
    604 - sigcont (Failed)
    605 - sigcont-no-syscallbuf (Failed)
    606 - sighandler_bad_rsp_sigsegv (Failed)
    607 - sighandler_bad_rsp_sigsegv-no-syscallbuf (Failed)
    608 - sighandler_fork (Failed)
    609 - sighandler_fork-no-syscallbuf (Failed)
    610 - sighandler_mask (Failed)
    611 - sighandler_mask-no-syscallbuf (Failed)
    612 - sigill (Failed)
    613 - sigill-no-syscallbuf (Failed)
    614 - signal_deferred (Failed)
    615 - signal_deferred-no-syscallbuf (Failed)
    616 - signal_during_preload_init (Failed)
    617 - signal_during_preload_init-no-syscallbuf (Failed)
    618 - signal_frame (Timeout)
    619 - signal_frame-no-syscallbuf (Failed)
    620 - signal_unstoppable (Failed)
    621 - signal_unstoppable-no-syscallbuf (Failed)
    622 - signalfd (Failed)
    623 - signalfd-no-syscallbuf (Failed)
    624 - sigprocmask (Failed)
    625 - sigprocmask-no-syscallbuf (Failed)
    626 - sigprocmask_ensure_delivery (Failed)
    627 - sigprocmask_ensure_delivery-no-syscallbuf (Failed)
    628 - sigprocmask_exec (Failed)
    629 - sigprocmask_exec-no-syscallbuf (Failed)
    630 - sigprocmask_evil (Failed)
    631 - sigprocmask_evil-no-syscallbuf (Failed)
    632 - sigprocmask_in_syscallbuf_sighandler (Failed)
    633 - sigprocmask_in_syscallbuf_sighandler-no-syscallbuf (Failed)
    634 - sigprocmask_rr_sigs (Failed)
    635 - sigprocmask_rr_sigs-no-syscallbuf (Failed)
    636 - sigprocmask_syscallbuf (Failed)
    637 - sigprocmask_syscallbuf-no-syscallbuf (Failed)
    638 - sigqueueinfo (Failed)
    639 - sigqueueinfo-no-syscallbuf (Failed)
    640 - x86/sigreturn (Failed)
    641 - x86/sigreturn-no-syscallbuf (Failed)
    642 - sigreturn_reg (Failed)
    643 - sigreturn_reg-no-syscallbuf (Failed)
    644 - sigreturnmask (Failed)
    645 - sigreturnmask-no-syscallbuf (Failed)
    646 - sigrt (Failed)
    647 - sigrt-no-syscallbuf (Failed)
    648 - sigstop (Failed)
    649 - sigstop-no-syscallbuf (Failed)
    650 - sigstop2 (Failed)
    651 - sigstop2-no-syscallbuf (Failed)
    652 - sigsuspend (Failed)
    653 - sigsuspend-no-syscallbuf (Failed)
    654 - sigtrap (Failed)
    655 - sigtrap-no-syscallbuf (Failed)
    656 - simple_threads_stress (Failed)
    657 - simple_threads_stress-no-syscallbuf (Failed)
    658 - sioc (Failed)
    659 - sioc-no-syscallbuf (Failed)
    660 - small_holes (Failed)
    661 - small_holes-no-syscallbuf (Failed)
    662 - sock_names_opts (Failed)
    663 - sock_names_opts-no-syscallbuf (Failed)
    664 - spinlock_priorities (Failed)
    665 - spinlock_priorities-no-syscallbuf (Failed)
    666 - splice (Failed)
    667 - splice-no-syscallbuf (Failed)
    668 - stack_growth_after_syscallbuf (Failed)
    669 - stack_growth_after_syscallbuf-no-syscallbuf (Failed)
    670 - stack_growth_syscallbuf (Failed)
    671 - stack_growth_syscallbuf-no-syscallbuf (Failed)
    672 - stack_growth_with_guard (Failed)
    673 - stack_growth_with_guard-no-syscallbuf (Failed)
    674 - stack_invalid (Failed)
    675 - stack_invalid-no-syscallbuf (Failed)
    676 - stack_overflow (Failed)
    677 - stack_overflow-no-syscallbuf (Failed)
    678 - stack_overflow_altstack (Failed)
    679 - stack_overflow_altstack-no-syscallbuf (Failed)
    680 - stack_overflow_with_guard (Failed)
    681 - stack_overflow_with_guard-no-syscallbuf (Failed)
    682 - statfs (Failed)
    683 - statfs-no-syscallbuf (Failed)
    684 - statx (Failed)
    685 - statx-no-syscallbuf (Failed)
    686 - stdout_child (Failed)
    687 - stdout_child-no-syscallbuf (Failed)
    688 - stdout_cloexec (Failed)
    689 - stdout_cloexec-no-syscallbuf (Failed)
    690 - stdout_dup (Failed)
    691 - stdout_dup-no-syscallbuf (Failed)
    692 - stdout_redirect (Failed)
    693 - stdout_redirect-no-syscallbuf (Failed)
    694 - switch_read (Failed)
    695 - switch_read-no-syscallbuf (Failed)
    696 - symlink (Failed)
    697 - symlink-no-syscallbuf (Failed)
    698 - sync (Failed)
    699 - sync-no-syscallbuf (Failed)
    700 - sync_file_range (Failed)
    701 - sync_file_range-no-syscallbuf (Failed)
    702 - syscall_bp (Failed)
    703 - syscall_bp-no-syscallbuf (Failed)
    704 - syscall_in_writable_mem (Failed)
    705 - syscall_in_writable_mem-no-syscallbuf (Failed)
    706 - syscallbuf_signal_reset (Failed)
    707 - syscallbuf_signal_reset-no-syscallbuf (Failed)
    708 - syscallbuf_signal_blocking (Failed)
    709 - syscallbuf_signal_blocking-no-syscallbuf (Failed)
    710 - syscallbuf_sigstop (Failed)
    711 - syscallbuf_sigstop-no-syscallbuf (Failed)
    712 - syscallbuf_timeslice (Failed)
    713 - syscallbuf_timeslice-no-syscallbuf (Failed)
    714 - syscallbuf_timeslice2 (Failed)
    715 - syscallbuf_timeslice2-no-syscallbuf (Failed)
    716 - sysconf (Failed)
    717 - sysconf-no-syscallbuf (Failed)
    718 - sysctl (Failed)
    719 - sysctl-no-syscallbuf (Failed)
    720 - sysemu_singlestep (Failed)
    721 - sysemu_singlestep-no-syscallbuf (Failed)
    722 - x86/sysfs (Failed)
    723 - x86/sysfs-no-syscallbuf (Failed)
    724 - sysinfo (Failed)
    725 - sysinfo-no-syscallbuf (Failed)
    726 - tgkill (Failed)
    727 - tgkill-no-syscallbuf (Failed)
    728 - thread_yield (Failed)
    729 - thread_yield-no-syscallbuf (Failed)
    730 - timer (Failed)
    731 - timer-no-syscallbuf (Failed)
    732 - timerfd (Failed)
    733 - timerfd-no-syscallbuf (Failed)
    734 - times (Failed)
    735 - times-no-syscallbuf (Failed)
    736 - truncate_temp (Failed)
    737 - truncate_temp-no-syscallbuf (Failed)
    738 - tun (Failed)
    739 - tun-no-syscallbuf (Failed)
    740 - two_signals_with_mask (Failed)
    741 - two_signals_with_mask-no-syscallbuf (Failed)
    742 - ulimit_low (Failed)
    743 - ulimit_low-no-syscallbuf (Failed)
    744 - uname (Failed)
    745 - uname-no-syscallbuf (Failed)
    746 - unexpected_exit (Failed)
    747 - unexpected_exit-no-syscallbuf (Failed)
    748 - unexpected_exit_execve (Failed)
    749 - unexpected_exit_execve-no-syscallbuf (Failed)
    750 - unexpected_exit_execve_twice (Failed)
    751 - unexpected_exit_execve_twice-no-syscallbuf (Failed)
    752 - unexpected_exit_pid_ns (Failed)
    753 - unexpected_exit_pid_ns-no-syscallbuf (Failed)
    754 - unjoined_thread (Failed)
    755 - unjoined_thread-no-syscallbuf (Failed)
    756 - unshare (Failed)
    757 - unshare-no-syscallbuf (Failed)
    758 - userfaultfd (Failed)
    759 - userfaultfd-no-syscallbuf (Failed)
    760 - utimes (Failed)
    761 - utimes-no-syscallbuf (Failed)
    762 - vfork_flush (Failed)
    763 - vfork_flush-no-syscallbuf (Failed)
    764 - vfork_shared (Failed)
    765 - vfork_shared-no-syscallbuf (Failed)
    766 - video_capture (Failed)
    767 - video_capture-no-syscallbuf (Failed)
    768 - vm_readv_writev (Failed)
    769 - vm_readv_writev-no-syscallbuf (Failed)
    770 - vsyscall (Failed)
    771 - vsyscall-no-syscallbuf (Failed)
    772 - vsyscall_timeslice (Failed)
    773 - vsyscall_timeslice-no-syscallbuf (Failed)
    774 - x86/x87env (Failed)
    775 - x86/x87env-no-syscallbuf (Failed)
    776 - wait (Failed)
    777 - wait-no-syscallbuf (Failed)
    778 - wait_sigstop (Failed)
    779 - wait_sigstop-no-syscallbuf (Failed)
    780 - write_race (Failed)
    781 - write_race-no-syscallbuf (Failed)
    782 - writev (Failed)
    783 - writev-no-syscallbuf (Failed)
    784 - xattr (Failed)
    785 - xattr-no-syscallbuf (Failed)
    786 - zero_length_read (Failed)
    787 - zero_length_read-no-syscallbuf (Failed)
    788 - std_random (Failed)
    789 - std_random-no-syscallbuf (Failed)
    790 - unwind_rr_page (Failed)
    791 - unwind_rr_page-no-syscallbuf (Failed)
    792 - abort_nonmain (Failed)
    793 - abort_nonmain-no-syscallbuf (Failed)
    794 - alternate_thread_diversion (Failed)
    795 - alternate_thread_diversion-no-syscallbuf (Failed)
    796 - args (Failed)
    797 - args-no-syscallbuf (Failed)
    798 - async_kill_with_threads (Timeout)
    799 - async_kill_with_threads-no-syscallbuf (Timeout)
    800 - async_kill_with_threads_main_running (Failed)
    801 - async_kill_with_threads_main_running-no-syscallbuf (Failed)
    802 - async_kill_with_threads_thread_running (Failed)
    803 - async_kill_with_threads_thread_running-no-syscallbuf (Failed)
    804 - async_segv (Failed)
    805 - async_segv-no-syscallbuf (Failed)
    806 - async_signal_syscalls (Failed)
    808 - async_signal_syscalls2 (Failed)
    810 - async_signal_syscalls_siginfo (Failed)
    812 - async_usr1 (Failed)
    813 - async_usr1-no-syscallbuf (Failed)
    814 - blacklist (Failed)
    815 - blacklist-no-syscallbuf (Failed)
    818 - block_clone_interrupted (Failed)
    819 - block_clone_interrupted-no-syscallbuf (Failed)
    820 - block_clone_syscallbuf_overflow (Failed)
    821 - block_clone_syscallbuf_overflow-no-syscallbuf (Failed)
    822 - block_intr_sigchld (Failed)
    823 - block_intr_sigchld-no-syscallbuf (Failed)
    824 - blocked_bad_ip (Failed)
    825 - blocked_bad_ip-no-syscallbuf (Failed)
    826 - blocked_sigill (Failed)
    827 - blocked_sigill-no-syscallbuf (Failed)
    828 - x86/blocked_sigsegv (Failed)
    829 - x86/blocked_sigsegv-no-syscallbuf (Failed)
    834 - breakpoint_overlap (Failed)
    835 - breakpoint_overlap-no-syscallbuf (Failed)
    838 - call_gettid (Failed)
    839 - call_gettid-no-syscallbuf (Failed)
    840 - checkpoint_dying_threads (Failed)
    841 - checkpoint_dying_threads-no-syscallbuf (Failed)
    844 - checksum_sanity (Failed)
    845 - checksum_sanity-no-syscallbuf (Failed)
    846 - check_lost_interrupts (Failed)
    848 - clone_interruption (Failed)
    849 - clone_interruption-no-syscallbuf (Failed)
    850 - clone_vfork (Failed)
    851 - clone_vfork-no-syscallbuf (Failed)
    856 - condvar_stress (Failed)
    857 - condvar_stress-no-syscallbuf (Failed)
    864 - daemon_read (Failed)
    865 - daemon_read-no-syscallbuf (Failed)
    866 - dconf_mock (Failed)
    867 - dconf_mock-no-syscallbuf (Failed)
    876 - early_error (Failed)
    880 - exclusion_region (Failed)
    881 - exclusion_region-no-syscallbuf (Failed)
    884 - exec_many (Failed)
    885 - exec_many-no-syscallbuf (Failed)
    886 - execve_loop (Failed)
    887 - execve_loop-no-syscallbuf (Failed)
    888 - exit_codes (Failed)
    889 - exit_codes-no-syscallbuf (Failed)
    890 - exit_group (Failed)
    891 - exit_group-no-syscallbuf (Failed)
    892 - exit_race (Failed)
    893 - exit_race-no-syscallbuf (Failed)
    894 - exit_status (Failed)
    895 - exit_status-no-syscallbuf (Failed)
    896 - explicit_checkpoints (Failed)
    897 - explicit_checkpoints-no-syscallbuf (Failed)
    898 - fork_stress (Failed)
    899 - fork_stress-no-syscallbuf (Failed)
    900 - fork_syscalls (Failed)
    901 - fork_syscalls-no-syscallbuf (Failed)
    906 - getcwd (Failed)
    907 - getcwd-no-syscallbuf (Failed)
    910 - goto_event (Failed)
    911 - goto_event-no-syscallbuf (Failed)
    912 - hello (Failed)
    913 - hello-no-syscallbuf (Failed)
    914 - hooks (Failed)
    916 - ignored_async_usr1 (Failed)
    917 - ignored_async_usr1-no-syscallbuf (Failed)
    918 - ignored_sigsegv (Failed)
    919 - ignored_sigsegv-no-syscallbuf (Failed)
    920 - ignore_nested (Failed)
    921 - ignore_nested-no-syscallbuf (Failed)
    922 - immediate_restart (Failed)
    923 - immediate_restart-no-syscallbuf (Failed)
    924 - x86/int3_ok (Failed)
    925 - x86/int3_ok-no-syscallbuf (Failed)
    928 - intr_ptrace_decline (Failed)
    929 - intr_ptrace_decline-no-syscallbuf (Failed)
    936 - link (Failed)
    937 - link-no-syscallbuf (Failed)
    940 - main_thread_exit (Failed)
    941 - main_thread_exit-no-syscallbuf (Failed)
    942 - mmap_fd_reuse_checkpoint (Failed)
    943 - mmap_fd_reuse_checkpoint-no-syscallbuf (Failed)
    948 - mmap_shared_write_exec_race (Failed)
    949 - mmap_shared_write_exec_race-no-syscallbuf (Failed)
    950 - mmap_tmpfs (Failed)
    951 - mmap_tmpfs-no-syscallbuf (Failed)
    954 - mmap_write_private (Failed)
    955 - mmap_write_private-no-syscallbuf (Failed)
    958 - mprotect_growsdown (Failed)
    959 - mprotect_growsdown-no-syscallbuf (Failed)
    960 - mprotect_syscallbuf_overflow (Failed)
    961 - mprotect_syscallbuf_overflow-no-syscallbuf (Failed)
    962 - mutex_pi_stress (Failed)
    963 - mutex_pi_stress-no-syscallbuf (Failed)
    964 - nested_detach_wait (Failed)
    965 - nested_detach_wait-no-syscallbuf (Failed)
    966 - overflow_branch_counter (Failed)
    967 - overflow_branch_counter-no-syscallbuf (Failed)
    968 - patch_page_end (Failed)
    970 - x86/patch_40_80_f6_81 (Failed)
    972 - priority (Failed)
    973 - priority-no-syscallbuf (Failed)
    976 - remove_latest_trace (Failed)
    977 - remove_latest_trace-no-syscallbuf (Failed)
    978 - rr_ps (Failed)
    979 - rr_ps-no-syscallbuf (Failed)
    980 - rr_ps_ns (Failed)
    981 - rr_ps_ns-no-syscallbuf (Failed)
    994 - reverse_step_long (Failed)
    995 - reverse_step_long-no-syscallbuf (Failed)
    996 - reverse_step_threads (Failed)
    997 - reverse_step_threads-no-syscallbuf (Failed)
    998 - reverse_step_threads_break (Failed)
    999 - reverse_step_threads_break-no-syscallbuf (Failed)
    1003 - seccomp_blocks_rr-no-syscallbuf (Failed)
    1004 - seccomp_signals (Failed)
    1005 - seccomp_signals-no-syscallbuf (Failed)
    1006 - segfault (Failed)
    1007 - segfault-no-syscallbuf (Failed)
    1008 - shared_map (Failed)
    1009 - shared_map-no-syscallbuf (Failed)
    1014 - sigprocmask_race (Failed)
    1018 - simple (Failed)
    1019 - simple-no-syscallbuf (Failed)
    1024 - step_thread (Failed)
    1025 - step_thread-no-syscallbuf (Failed)
    1026 - strict_priorities (Failed)
    1027 - strict_priorities-no-syscallbuf (Failed)
    1028 - x86/string_instructions (Failed)
    1029 - x86/string_instructions-no-syscallbuf (Failed)
    1030 - x86/string_instructions_async_signals (Failed)
    1031 - x86/string_instructions_async_signals-no-syscallbuf (Failed)
    1032 - x86/string_instructions_async_signals_shared (Failed)
    1033 - x86/string_instructions_async_signals_shared-no-syscallbuf
(Failed)
    1036 - x86/string_instructions_replay (Failed)
    1037 - x86/string_instructions_replay-no-syscallbuf (Failed)
    1042 - syscallbuf_fd_disabling (Failed)
    1043 - syscallbuf_fd_disabling-no-syscallbuf (Failed)
    1044 - syscallbuf_signal_blocking_read (Failed)
    1045 - syscallbuf_signal_blocking_read-no-syscallbuf (Failed)
    1046 - sysconf_onln (Failed)
    1047 - sysconf_onln-no-syscallbuf (Failed)
    1050 - target_process (Failed)
    1051 - target_process-no-syscallbuf (Failed)
    1052 - term_nonmain (Failed)
    1053 - term_nonmain-no-syscallbuf (Failed)
    1060 - thread_exit_signal (Failed)
    1061 - thread_exit_signal-no-syscallbuf (Failed)
    1062 - thread_open_race (Failed)
    1064 - thread_stress (Failed)
    1065 - thread_stress-no-syscallbuf (Failed)
    1068 - threads (Failed)
    1069 - threads-no-syscallbuf (Failed)
    1070 - tls (Failed)
    1071 - tls-no-syscallbuf (Failed)
    1076 - user_ignore_sig (Failed)
    1077 - user_ignore_sig-no-syscallbuf (Failed)
    1084 - vfork (Failed)
    1085 - vfork-no-syscallbuf (Failed)
    1086 - vfork_read_clone_stress (Failed)
    1087 - vfork_read_clone_stress-no-syscallbuf (Failed)
    1090 - wait_for_all (Failed)
    1091 - wait_for_all-no-syscallbuf (Failed)
    1092 - watchpoint (Failed)
    1093 - watchpoint-no-syscallbuf (Failed)
    1094 - watchpoint_at_sched (Failed)
    1095 - watchpoint_at_sched-no-syscallbuf (Failed)
    1096 - watchpoint_before_signal (Failed)
    1097 - watchpoint_before_signal-no-syscallbuf (Failed)
    1106 - async_signal_syscalls_100 (Failed)
    1107 - async_signal_syscalls_100-no-syscallbuf (Failed)
    1108 - async_signal_syscalls_1000 (Failed)
    1109 - async_signal_syscalls_1000-no-syscallbuf (Failed)
    1110 - bad_breakpoint (Timeout)
    1111 - bad_breakpoint-no-syscallbuf (Timeout)
    1112 - break_block (Failed)
    1113 - break_block-no-syscallbuf (Failed)
    1116 - break_clone (Failed)
    1117 - break_clone-no-syscallbuf (Failed)
    1120 - break_int3 (Failed)
    1121 - break_int3-no-syscallbuf (Failed)
    1132 - break_thread (Failed)
    1133 - break_thread-no-syscallbuf (Failed)
    1138 - call_exit (Failed)
    1139 - call_exit-no-syscallbuf (Failed)
    1142 - checkpoint_async_signal_syscalls_1000 (Failed)
    1144 - checkpoint_mmap_shared (Failed)
    1145 - checkpoint_mmap_shared-no-syscallbuf (Failed)
    1146 - checkpoint_prctl_name (Failed)
    1147 - checkpoint_prctl_name-no-syscallbuf (Failed)
    1148 - checkpoint_simple (Failed)
    1149 - checkpoint_simple-no-syscallbuf (Failed)
    1150 - checksum_sanity_noclone (Failed)
    1151 - checksum_sanity_noclone-no-syscallbuf (Failed)
    1154 - x86/cpuid (Failed)
    1155 - x86/cpuid-no-syscallbuf (Failed)
    1156 - dead_thread_target (Failed)
    1157 - dead_thread_target-no-syscallbuf (Failed)
    1160 - deliver_async_signal_during_syscalls (Timeout)
    1162 - env_newline (Failed)
    1163 - env_newline-no-syscallbuf (Failed)
    1164 - exec_deleted (Failed)
    1165 - exec_deleted-no-syscallbuf (Failed)
    1168 - execp (Failed)
    1169 - execp-no-syscallbuf (Failed)
    1170 - explicit_checkpoint_clone (Failed)
    1171 - explicit_checkpoint_clone-no-syscallbuf (Failed)
    1172 - file_name_newline (Failed)
    1173 - file_name_newline-no-syscallbuf (Failed)
    1178 - fork_exec_info_thr (Failed)
    1179 - fork_exec_info_thr-no-syscallbuf (Failed)
    1180 - get_thread_list (Failed)
    1181 - get_thread_list-no-syscallbuf (Failed)
    1182 - hardlink_mmapped_files (Failed)
    1183 - hardlink_mmapped_files-no-syscallbuf (Failed)
    1188 - nested_detach (Timeout)
    1189 - nested_detach-no-syscallbuf (Timeout)
    1202 - record_replay (Timeout)
    1203 - record_replay-no-syscallbuf (Timeout)
    1204 - remove_watchpoint (Failed)
    1205 - remove_watchpoint-no-syscallbuf (Failed)
    1212 - restart_unstable (Failed)
    1213 - restart_unstable-no-syscallbuf (Failed)
    1214 - restart_diversion (Failed)
    1215 - restart_diversion-no-syscallbuf (Failed)
    1232 - reverse_step_threads2 (Failed)
    1233 - reverse_step_threads2-no-syscallbuf (Failed)
    1234 - reverse_watchpoint (Failed)
    1235 - reverse_watchpoint-no-syscallbuf (Failed)
    1240 - run_in_function (Failed)
    1241 - run_in_function-no-syscallbuf (Failed)
    1244 - seekticks (Failed)
    1245 - seekticks-no-syscallbuf (Failed)
    1250 - x86/sigreturn_checksum (Failed)
    1251 - x86/sigreturn_checksum-no-syscallbuf (Failed)
    1256 - simple_script (Failed)
    1257 - simple_script-no-syscallbuf (Failed)
    1276 - switch_processes (Failed)
    1277 - switch_processes-no-syscallbuf (Failed)
    1278 - syscallbuf_timeslice_250 (Failed)
    1279 - syscallbuf_timeslice_250-no-syscallbuf (Failed)
    1280 - trace_version (Failed)
    1281 - trace_version-no-syscallbuf (Failed)
    1288 - vfork_exec (Failed)
    1289 - vfork_exec-no-syscallbuf (Failed)
    1290 - vfork_break_parent (Failed)
    1291 - vfork_break_parent-no-syscallbuf (Failed)
    1296 - watchpoint_cond (Failed)
    1297 - watchpoint_cond-no-syscallbuf (Failed)
Errors while running CTest

[Kyle Huey]

If you `./bin/rr record ./bin/alarm` and then `./bin/rr replay -a`
(adjust paths appropriately) what happens?

- Kyle

[Ben Stanley]

On 21/5/21 12:33 am, Kyle Huey wrote:
>
> If you `./bin/rr record ./bin/alarm` and then `./bin/rr replay -a`
> (adjust paths appropriately) what happens?
>
> - Kyle

================ Test 1 ====================

alarm program by itself

[ben@rr obj]$ ./bin/alarm
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Signal 14 caught, Counter is 155085609
EXIT-SUCCESS
[ben@rr obj]$ echo $?
0
[ben@rr obj]$

So the alarm program by itself works and exits properly.

================ Test 2 ====================

alarm program under rr record

[ben@rr obj]$ ./bin/rr record ./bin/alarm
rr: Saving execution to trace directory `/home/ben/.local/share/rr/alarm-3'.
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Signal 14 caught, Counter is 65800001
EXIT-SUCCESS
<doesn't exit - waited several hours>

^C^C^C^C^Z
[1]+  Stopped                 ./bin/rr record ./bin/alarm
[ben@rr obj]$ bg
[1]+ ./bin/rr record ./bin/alarm &
[ben@rr obj]$ kill -9 %1
[ben@rr obj]$ jobs
[1]+  Killed                  ./bin/rr record ./bin/alarm
[ben@rr obj]$ ./bin/rr replay -a

rr: Trace file `/home/ben/.local/share/rr/alarm-3/incomplete' found.
rr recording terminated abnormally and the trace is incomplete.

[ben@rr obj]$

Can't examine recording because I had to interrupt rr when rr failed to
finish the recording session properly.

================ Test 3 ====================

[ben@rr obj]$ ./bin/rr record ./bin/alarm
rr: Saving execution to trace directory `/home/ben/.local/share/rr/alarm-4'.
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Signal 14 caught, Counter is 73700001
EXIT-SUCCESS

<doesn't exit>

<in a new shell>

[ben@rr obj]$ ps -ale | grep rr
0 S  1000  3616  3528  2  80   0 - 139141 do_wai pts/4   00:00:00 rr
[ben@rr obj]$ gdb -p 3616
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 3616
Reading symbols from /home/ben/rr/obj/bin/rr...done.
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libcapnp-0.5.3.so...Reading symbols from
/lib64/libcapnp-0.5.3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcapnp-0.5.3.so
Reading symbols from /lib64/libkj-0.5.3.so...Reading symbols from
/lib64/libkj-0.5.3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkj-0.5.3.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[New LWP 3623]
[New LWP 3622]
[New LWP 3621]
[New LWP 3620]
[New LWP 3619]
[New LWP 3618]
[New LWP 3617]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libstdc++.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f6999ef0640 in waitid () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
capnproto-libs-0.5.3.1-1.el7.x86_64 glibc-2.17-324.el7_9.x86_64
libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64
(gdb) where
#0  0x00007f6999ef0640 in waitid () from /lib64/libc.so.6
#1  0x0000000000894e6d in rr::Task::wait_exit (this=0x1df2800)
    at /home/ben/rr/rr/src/Task.cc:167
#2  0x00000000008952b0 in rr::Task::proceed_to_exit (this=0x1df2800,
wait=true)
    at /home/ben/rr/rr/src/Task.cc:197
#3  0x0000000000799a14 in rr::handle_ptrace_exit_event (t=0x1df2800)
    at /home/ben/rr/rr/src/RecordSession.cc:235
#4  0x00000000007a29ae in rr::RecordSession::record_step (this=0x1df0450)
    at /home/ben/rr/rr/src/RecordSession.cc:2349
#5  0x0000000000796547 in rr::record (
    args=std::vector of length 1, capacity 2 = {...}, flags=...)
    at /home/ben/rr/rr/src/RecordCommand.cc:656
#6  0x0000000000797017 in rr::RecordCommand::run (
    this=0xcdcc70 <rr::RecordCommand::singleton>,
    args=std::vector of length 1, capacity 2 = {...})
    at /home/ben/rr/rr/src/RecordCommand.cc:791
#7  0x00000000008efb9e in main (argc=3, argv=0x7ffdd51d6468)
    at /home/ben/rr/rr/src/main.cc:249
(gdb)

I'm not sure where to go from here. I can try more stuff if you provide
instructions, but I suspect it might be more efficient if you set up a
CentOS-7 VM for yourself so that you can test it directly.

Ben.