Illegal instruction (core dumped) when the cpu Architecture: armv7l,Byte Order:Little Endian

zhenhuan li

unread,

Nov 11, 2021, 5:01:59 AM11/11/21

to DynamoRIO Users

when i run the drrun, it crash like the following:

./drrun -debug /usr/bin/sftp

<Paste into GDB to debug DynamoRIO clients:

set confirm off

add-symbol-file 'DynamoRIO-ARM-Linux-EABIHF-8.0.18929/lib32/debug/libdynamorio.so' 0x54b685a0

>

<(1+x) Handling our fault in a TRY at 0x54d31024>

<Application /usr/bin/sftp (10975). Application exception at PC 0x76f62f52.

Signal 4 delivered to application as default action.

Callstack:

0x76f62f52 </usr/lib/libcrypto.so.1.0.0+0x119f52>

0xf8af04b0

>

Illegal instruction (core dumped)

The following is the cpu information :

Architecture: armv7l

Byte Order: Little Endian

CPU(s): 2

On-line CPU(s) list: 0,1

Thread(s) per core: 1

Core(s) per socket: 2

Socket(s): 1

Model name: ARMv7 Processor rev 10 (v7l)

CPU max MHz: 996.0000

CPU min MHz: 396.0000

Has anyone encountered the same problem？

Does DynamoRIO supports armv7l Little Endian?

zhenhuan li

unread,

Nov 11, 2021, 5:57:55 AM11/11/21

to DynamoRIO Users

Partial information in the logfile when -loglevel 4

223370 d_r_dispatch: target = 0x76f2ff52

223371

223372 interp: start_pc = 0x76f2ff52

223373 check_thread_vm_area: pc = 0x76f2ff52

223374 prepend_entry_to_fraglist: putting fragment @0x76f2ff52 (shared) on vmarea 0x76e16000-0x76f815c0

223375 check_thread_vm_area: check_stop = 0x76f815c0

223376 SYSLOG_WARNING: Invalid opcode encountered

223377 Invalid Thumb opcode @0x76f2ff52: 0xebfc8002

223378 decode: invalid instr at 0x76f2ff52

223379 Invalid Thumb opcode @0x76f2ff52: 0xebfc8002

223380 decode: invalid instr at 0x76f2ff52

223381 0x76f2ff52 ebfc 8002 <INVALID>

223382 interp: invalid instr at 0x76f2ff52

223383 vm_area_remove_fragment: entry 0x485e03e8

223384 record_pending_signal(4 at pc 0x76f2ff52): signal is currently blocked

223385 action is not SIG_IGN

223386 blocked fatal signal 4 cannot be delayed: terminating

223387 execute_default_action for signal 4

223388 SYSLOG_WARNING: Application /usr/bin/sftp (9084). Application exception at PC 0x76f2ff52.

223389 Signal 4 delivered to application as default action.

223390 Callstack:

223391 0x76f2ff52 </usr/lib/libcrypto.so.1.0.0+0x119f52>

223392 0xf8af04b0

223393

223394 Terminating via kill

assad.hashm...@gmail.com

unread,

Nov 18, 2021, 12:34:20 PM11/18/21

to DynamoRIO Users

The faulty instruction encoding is a Thumb one: ebfc 8002

53561 SYSLOG_WARNING: Invalid opcode encountered

53562 Invalid Thumb opcode @0x76f51f52: 0xebfc8002

53563 decode: invalid instr at 0x76f51f52

53564 Invalid Thumb opcode @0x76f51f52: 0xebfc8002

53565 decode: invalid instr at 0x76f51f52

53566 0x76f51f52 ebfc 8002

The LLVM13 Thumb disassembler gives:

0x8002

strh r2, [r0]

ebfc

<stdin>:1:1: warning: invalid instruction encoding

0x00

^

.text

The error message:

Signal 4 delivered to application as default action.

means the hardware has tried to execute an illegal instruction, i.e. an instruction not supported by the machine.

This can happen if:

1) The executable including linked libraries have been compiled for target hardware other than the one you are running on.

2) DynamoRIO has generated an instruction which is invalid.

In order to narrow down the cause, can you run the following and let us know what happens:

1) Run the binary natively on your hardware, i.e. do not use DynamorRIO's drrun, just run /usr/bin/sftp.

2) Run the binary with just drrun, no clients.

3) Run the binary with the opcodes client: drrun -client libopcodes.so -- /usr/bin/sftp

We need to establish if DynamoRIO is generating an illegal instruction or /usr/bin/sftp has been built for different target hardware.

zhenhuan li

unread,

Nov 24, 2021, 3:20:50 AM11/24/21

to DynamoRIO Users

Thanks for your response!

1) Run the binary natively on your hardware, i.e. do not use DynamorRIO's drrun, just run /usr/bin/sftp.

Answer:

i can run /bin/sftp success on the same machine with DynamorRIO.

/bin/sftp

usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]

[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]

[-o ssh_option] [-P port] [-R num_requests] [-S program]

[-s subsystem | sftp_server] host

sftp [user@]host[:file ...]

sftp [user@]host[:dir[/]]

sftp -b batchfile [user@]host

2) Run the binary with just drrun, no clients.

Answer:

when i use drrun run /bin/sftp with no clients, output like following:

./drrun /bin/sftp

Illegal instruction (core dumped)

3) Run the binary with the opcodes client: drrun -client libopcodes.so -- /usr/bin/sftp

Answer:

run the /bin/sftp with client libopcodes.so , the output like following:

./drrun -c ../samples/bin32/libopcodes.so -- /bin/sftp

Client opcodes is running

Top 15 opcode execution counts in 32-bit ARM mode:

4088 : clz

4208 : blx

5338 : stmdb

5655 : lsr

5660 : ldm

6753 : bl

9348 : tst

14531 : and

14685 : ldrb

22690 : adc

28934 : sub

40143 : add

43758 : mov

49473 : str

83097 : b

93892 : cmp

117997 : ldr

Top 15 opcode execution counts in 32-bit Thumb mode:

2933 : cbnz

3194 : and

3462 : lsl

3470 : lsr

3518 : bic

3788 : sel

3788 : uadd8

3805 : ldr

3836 : strd

3982 : eor

4208 : bx

4223 : it

4740 : sub

5257 : ldrd

9034 : cmp

9048 : ldrb

13684 : b

Illegal instruction (core dumped)

By the way, there may be some information I overlooked last time

Partial information in the logfile when -loglevel 3

2392

2393 Entry into F81(0x76f71548).0x6af9cb78 (A32)(shared)

2394

2395 Exit from F81(0x76f71548).0x6af9cb98 (A32)(shared)

2396 (target 0x76f72748 not in cache)

2397

2398 d_r_dispatch: target = 0x76f72748

2399

2400 interp: start_pc = 0x76f72748

2401 0x76f72748 e52d7004 str %r7 $0xfffffffc %sp -> -0x04(%sp)[4byte] %sp

2402 0x76f7274c e1a02000 mov %r0 -> %r2

2403 0x76f72750 e3a0702d mov $0x0000002d -> %r7

2404 0x76f72754 ef000000 svc $0x00000000

2405 make_writable: pc 0x6af94000 -> 0x6af94000-0x6af96000 0

2406 Just updated syscall routine:

2407 decode: invalid instr at 0x6af95480

2408 0x6af95480 0c00f84a <INVALID>

2409 decode: invalid instr at 0x6af95484

2410 0x6af95484 f84adf00 <INVALID>

2411 ...

2412 make_unwritable: pc 0x6af94000 -> 0x6af94000-0x6af96000

2413 find_syscall_num: found syscall number write: 45

2414 syscall # is 45

2415 found non-ignorable system call 0x002d

2416 ending bb at syscall & removing the interrupt itself

2417 end_pc = 0x76f72758

2418

2419 exit_branch_type=0x600 bb->exit_target=0x76f72758

2420 exit_branch_type=0x600 target=0x76f72758 l->flags=0x1e01

2421 Exit cti 0x6af9cbc4 is targeting 0x6af9cbc8 + 0x0 => 0x6af9cbc8

2422 Fragment 82, tag 0x76f72748, flags 0x1801030, shared, size 40, must end trace:

53633 Entry into F1415(0x76f05ee4).0x6b033204 (T32)(shared)

53634

53635 Exit from F1415(0x76f05ee4).0x6b033206 (T32)(shared)

53636 (target 0x76f05f52 not in cache)

53637

53638 d_r_dispatch: target = 0x76f05f52

53639

53640 interp: start_pc = 0x76f05f52

53641 SYSLOG_WARNING: Invalid opcode encountered

53642 Invalid Thumb opcode @0x76f05f52: 0xebfc8002

53643 decode: invalid instr at 0x76f05f52

53644 Invalid Thumb opcode @0x76f05f52: 0xebfc8002

53645 decode: invalid instr at 0x76f05f52

53646 0x76f05f52 ebfc 8002 <INVALID>

53647 interp: invalid instr at 0x76f05f52

53648 record_pending_signal(4 at pc 0x76f05f52): signal is currently blocked

53649 action is not SIG_IGN

53650 blocked fatal signal 4 cannot be delayed: terminating

53651 execute_default_action for signal 4

53652 SYSLOG_WARNING: Application /usr/bin/sftp (19343). Application exception at PC 0x76f05f52.

53653 Signal 4 delivered to application as default action.

53654 Callstack:

53655 0x76f05f52 </usr/lib/libcrypto.so.1.0.0+0x119f52>

53656 0xf8af04b0

53657

53658 Terminating via kill

53659 SYSLOG_INFORMATION: Stopping application /usr/bin/sftp (19343)

53660 synch with all threads my id = 19343 Giving 4 permission and seeking 3 state

Looking forward to your reply, thank you very much！

zhenhuan li

unread,

Nov 24, 2021, 5:44:06 AM11/24/21

to DynamoRIO Users

the following is i use the ida disassembly the libcrypto.so.1.0.0 , i can see the EBFC opcode is in the middle of the following red instruction.

.text:00119F0C sub_119F0C ; CODE XREF: BN_div+470↑p

.text:00119F0C ; BN_mod_word+60↑p ...

.text:00119F0C

.text:00119F0C var_C = -0xC

.text:00119F0C

.text:00119F0C 00 00 53 E3 CMP R3, #0

.text:00119F10 00 00 52 03 CMPEQ R2, #0

.text:00119F14 04 00 00 1A BNE loc_119F2C

.text:00119F18 00 00 51 E3 CMP R1, #0

.text:00119F1C 00 00 50 03 CMPEQ R0, #0

.text:00119F20 00 10 E0 13 MOVNE R1, #0xFFFFFFFF

.text:00119F24 00 00 E0 13 MOVNE R0, #0xFFFFFFFF

.text:00119F28 06 00 00 EA B loc_119F48

.text:00119F2C ; ---------------------------------------------------------------------------

.text:00119F2C

.text:00119F2C loc_119F2C ; CODE XREF: sub_119F0C+8↑j

.text:00119F2C 08 D0 4D E2 SUB SP, SP, #8

.text:00119F30 00 60 2D E9 STMFD SP!, {SP,LR}

.text:00119F34 07 00 00 EB BL sub_119F58

.text:00119F38 04 E0 9D E5 LDR LR, [SP,#0x10+var_C]

.text:00119F3C 08 D0 8D E2 ADD SP, SP, #8

.text:00119F40 0C 00 BD E8 LDMFD SP!, {R2,R3}

.text:00119F44 1E FF 2F E1 BX LR

.text:00119F48 ; ---------------------------------------------------------------------------

.text:00119F48

.text:00119F48 loc_119F48 ; CODE XREF: sub_119AC0+1E8↑j

.text:00119F48 ; sub_119CCC+21C↑j ...

.text:00119F48 02 40 2D E9 STMFD SP!, {R1,LR}

.text:00119F4C 08 00 A0 E3 MOV R0, #8 ; sig

.text:00119F50 74 94 FC EB BL raise

.text:00119F54 02 80 BD E8 LDMFD SP!, {R1,PC}

.text:00119F54 ; End of function sub_119F0C

the following is the outpout of the dynamorio, i can see the following red instruction jump to the 0x76f05f52, which lead the crash.

interp: start_pc = 0x76f05ee4

0x76f05ee4 0102 lsls %r0 $0x00000004 -> %r2

0x76f05ee6 b3a0 cbz $0x76f05f52 %r0

end_pc = 0x76f05ee8

but in the ida disassembly ,we can see ida use four bytes to encode the instruction, the DynamoRIO just use two bytes to encode the instruction.

.text:00119EDC 00 00 50 E3 CMP R0, #0

.text:00119EE0 02 01 E0 C3 MOVGT R0, #0x7FFFFFFF

.text:00119EE4 02 01 A0 B3 MOVLT R0, #0x80000000

.text:00119EE8 16 00 00 EA B loc_119F48

the root cause of this issue may be the like following:

DynamoRIO output:

53413

53414 interp: start_pc = 0x76f05a08

53415 0x76f05a08 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53416 0x76f05a0c e12fff1e bx %lr

53417 end_pc = 0x76f05a10

53418

53419 setting cur_pc (for fall-through) to 0x76f05a10

53420 exit_branch_type=0x6 bb->exit_target=0x6af95180

53421

53422 done building bb instrlist *********************

53423

53424 recreate_app : pc is in F1412(0x76f05a08)

53425 ilist for recreation:

53426 TAG 0x76f05a08

53427 +0 L3 @0x6afdffe8 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53428 +4 m4 @0x6afdfea8 e58a2008 str %r2 -> +0x08(%r10)[4byte]

53429 +8 m4 @0x6afdebe0 e58a2008 <label>

53430 +8 m4 @0x6afdf708 e1a0200e mov %lr -> %r2

53431 +12 L4 @0x6afdfe08 eafe82fc b $0x6af95180 <shared_bb_ibl_ret>

53432 END 0x76f05a08

53433

the ida disassemble like following:

.text:00119A08 EXPORT _armv7_tick

.text:00119A08 _armv7_tick ; CODE XREF: OPENSSL_cpuid_setup+2CC↑p

.text:00119A08 ; OPENSSL_rdtsc+1C↑j

.text:00119A08 ; DATA XREF: ...

.text:00119A08 1E 0F 51 EC MRRC p15, 1, R0,R1,c14

.text:00119A0C 1E FF 2F E1 BX LR

.text:00119A0C ; End of function _armv7_tick

there are different decode of "1E 0F 51 EC" between ida and DynamoRIO:

IDA:

.text:00119A08 1E 0F 51 EC MRRC p15, 1, R0,R1,c14

DynamoRIO：

0x76f05a08 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

Derek Bruening

unread,

Nov 24, 2021, 1:13:01 PM11/24/21

to dynamor...@googlegroups.com

I think the MRRC decodings are in fact identical and it's just differences in disassembly style.

The issue looks like DR thinks the app transitioned to Thumb mode when it really didn't, so it's decoding in the wrong mode.

Look back in the log to find where it first incorrectly goes to Thumb: is it on an application indirect branch, or is it on a signal return, or something else?

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/8a1acc05-4910-47a9-98fd-1a6443b06f54n%40googlegroups.com.

zhenhuan li

unread,

Nov 24, 2021, 8:38:07 PM11/24/21

to DynamoRIO Users

Thanks for response.

The following is the output of DR with "loglevel 3" , in 53560 line ,DR switch to ARM mode, but at 53589 line , the DR use thumb decode the opcode.I think this is the first incorrectly place.

The IDA disassembly :

.text:0011993C ; void __fastcall __noreturn sub_11993C(int val)

.text:0011993C sub_11993C ; DATA XREF: OPENSSL_cpuid_setup+F0↑o

.text:0011993C ; .text:off_3F9A8↑o

.text:0011993C 00 10 A0 E1 MOV R1, R0 ; val

.text:00119940 08 00 9F E5 LDR R0, =(unk_16B3A8 - 0x119950)

.text:00119944 10 40 2D E9 STMFD SP!, {R4,LR}

.text:00119948 00 00 8F E0 ADD R0, PC, R0 ; unk_16B3A8 ; env

.text:0011994C 24 97 FC EB BL siglongjmp

.text:0011994C ; End of function sub_11993C

The DR output :

53434 recreate_app : looking for 0x6b033198 in frag @ 0x6b033198 (tag 0x76f05a08)

53435 recreate_app -- found valid state pc 0x76f05a08

53436 translation 0x76f05a08 is post-walk 0x00000000 so not fixing xsp

53437 recreate_app -- found ok pc 0x76f05a08

53438 restoring stolen register to 0x76f8a000

53439 recreate_app_state -- translation is:

53440 priv_mcontext_t @0x6aff49ac

53441 r0 = 0x00000000

53442 r1 = 0x00000000

53443 r2 = 0x76f574ac

53444 r3 = 0x00000008

53445 r4 = 0x76f57534

53446 r5 = 0x76f573a8

53447 r6 = 0x76f575b4

53448 r7 = 0x7ec48b18

53449 r8 = 0x00000001

53450 r9 = 0x7ec48c0c

53451 r10 = 0x76f8a000

53452 r11 = 0x00000000

53453 r12 = 0x76f57410

53454 r13 = 0x7ec48a00

53455 r14 = 0x76e2b934

53456 r15 = 0x76f05a08

53457 q0 = 0x76f575b4 7ec48b18 00000001 76f70254

53458 q1 = 0x00000000 00000000 00000000 00000000

53459 q2 = 0x00000000 00000000 00000000 00000000

53460 q3 = 0x00000000 00000000 00000000 00000000

53461 q4 = 0x00000000 00000000 00000000 00000000

53462 q5 = 0x00000000 00000000 00000000 00000000

53463 q6 = 0x00000000 00000000 00000000 00000000

53464 q7 = 0x00000000 00000000 00000000 00000000

53465 q8 = 0x00001000 00000000 0000000c 00000000

53466 q9 = 0x0000000d 00000000 00000000 00000000

53467 q10 = 0x00000000 00000000 00000000 00000000

53468 q11 = 0x00000000 00000000 00000000 00000000

53469 q12 = 0x00000000 00000000 00000000 00000000

53470 q13 = 0x00000000 00000000 00000000 00000000

53471 q14 = 0x00000000 00000000 00000000 00000000

53472 q15 = 0x00000000 00000000 00000000 00000000

53473 eflags = 0x600b0010

53474 pc = 0x76f05a08

53475 translate_sigcontext: just set frame's eip to 0x76f05a08

53476 Got signal at pc 0x6b033198 in this fragment:

53477 Fragment 1412, tag 0x76f05a08, flags 0x1000030, shared, size 36:

53478

53479 -------- prefix entry: --------

53480 0x6b033194 e59a0000 ldr (%r10)[4byte] -> %r0

53481 -------- normal entry: --------

53482 0x6b033198 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53483 0x6b03319c e58a2008 str %r2 -> +0x08(%r10)[4byte]

53484 0x6b0331a0 e1a0200e mov %lr -> %r2

53485 0x6b0331a4 eaffffff b $0x6b0331a8 <exit stub 0>

53486 -------- exit stub 0: -------- <target: 0x6af95180> type: ret

53487 0x6b0331a8 e58a1004 str %r1 -> +0x04(%r10)[4byte]

53488 0x6b0331ac e3041e10 movw $0x00004e10 -> %r1

53489 0x6b0331b0 e3461b02 movt $0x6b02 -> %r1[2byte]

53490 0x6b0331b4 e59af03c ldr +0x3c(%r10)[4byte] -> %pc

53491

53492 Going to receive signal now

53493 get_sigstack_frame_ptr: using frame's xsp 0x7ec48a00

53494 placing frame at 0x7ec48700

53495 execute_handler_from_cache for signal 4

53496 xsp is 0x7ec48700

53497 copy_frame_to_stack: rt=0, src=0x6aff4c90, sp=0x7ec48700

53498 fixup_siginfo: updating si_addr from 0x6b033198 to 0x76f05a08

53499 converted sig=4 rt frame to non-rt frame

53500 copied frame from 0x6aff4c90 to 0x7ec48700

53501 blocked signals are now:

53502 1 = blocked

53503 2 = blocked

53504 3 = blocked

53505 4 = blocked

53506 6 = blocked

53507 9 = blocked

53508 10 = blocked

53509 12 = blocked

53510 13 = blocked

53511 14 = blocked

53512 15 = blocked

53513 16 = blocked

53514 17 = blocked

53515 18 = blocked

53516 19 = blocked

53517 20 = blocked

53518 21 = blocked

53519 22 = blocked

53520 23 = blocked

53521 24 = blocked

53522 25 = blocked

53523 26 = blocked

53524 27 = blocked

53525 28 = blocked

53526 29 = blocked

53527 30 = blocked

53528 31 = blocked

53529 34 = blocked

53530 35 = blocked

53531 36 = blocked

53532 37 = blocked

53533 38 = blocked

53534 39 = blocked

53535 40 = blocked

53536 41 = blocked

53537 42 = blocked

53538 43 = blocked

53539 44 = blocked

53540 45 = blocked

53541 46 = blocked

53542 47 = blocked

53543 48 = blocked

53544 49 = blocked

53545 50 = blocked

53546 51 = blocked

53547 52 = blocked

53548 53 = blocked

53549 54 = blocked

53550 55 = blocked

53551 56 = blocked

53552 57 = blocked

53553 58 = blocked

53554 59 = blocked

53555 60 = blocked

53556 61 = blocked

53557 62 = blocked

53558 63 = blocked

53559 64 = blocked

53560 Switching to ARM mode @0x76f0593c

53561 saved xax 0x00000004

53562 set next_tag to 0x76f0593c, resuming in fcache_return

53563 transfer_from_sig_handler_to_fcache_return

53564 sigcontext @0x6aff4d24:

53565 r0 =0x54de219c

53566 r1 =0x00000000

53567 r2 =0x76f574ac

53568 r3 =0x00000008

53569 r4 =0x76f57534

53570 r5 =0x76f573a8

53571 r6 =0x76f575b4

53572 r7 =0x7ec48b18

53573 r8 =0x00000001

53574 r9 =0x7ec48c0c

53575 r10 =0x6afc5000

53576 r11 =0x00000000

53577 r12 =0x76f57410

53578 sp =0x7ec48700

53579 r14 =0x76c32800

53580 pc =0x6af94ee0

53581 cpsr=0x600b0030

53582 set next_tag to handler 0x76f0593c, xsp to 0x7ec48700

53583 master_signal_handler 4 returning now to 0x6af94ee0

53584

53585 Exit from asynch event

53586

53587 d_r_dispatch: target = 0x76f0593c

53588

53589 interp: start_pc = 0x76f0593c

53590 0x76f0593c 1000 asrs %r0 $0x00000000 -> %r0

53591 0x76f0593e e1a0 b $0x76f05c82

53592 end_pc = 0x76f05940

53593

53594 exit_branch_type=0x0 target=0x76f05c82 l->flags=0x1801

53595 Exit cti 0x6b0331c2 is targeting 0x6b0331c8 + 0x0 => 0x6b0331c8

53596 Fragment 1413, tag 0x76f0593c, flags 0x1400030, shared, size 32:

53597

53598 Entry into F1413(0x76f0593c).0x6b0331c0 (T32)(shared)

53599

53600 Exit from F1413(0x76f0593c).0x6b0331c2 (T32)(shared)

53601 (target 0x76f05c82 not in cache)

53602

53603 d_r_dispatch: target = 0x76f05c82

53604

53605 interp: start_pc = 0x76f05c82

53606 0x76f05c82 e12f b $0x76f05ee4

53607 end_pc = 0x76f05c84

53608

53609 exit_branch_type=0x0 target=0x76f05ee4 l->flags=0x1801

53610 Exit cti 0x6b0331e4 is targeting 0x6b0331e8 + 0x0 => 0x6b0331e8

53611 Fragment 1414, tag 0x76f05c82, flags 0x1400030, shared, size 28:

53612

53613 Entry into F1414(0x76f05c82).0x6b0331e4 (T32)(shared)

53614

53615 Exit from F1414(0x76f05c82).0x6b0331e4 (T32)(shared)

53616 (target 0x76f05ee4 not in cache)

53617

53618 d_r_dispatch: target = 0x76f05ee4

53619

53620 interp: start_pc = 0x76f05ee4

53621 0x76f05ee4 0102 lsls %r0 $0x00000004 -> %r2

53622 0x76f05ee6 b3a0 cbz $0x76f05f52 %r0

53623 end_pc = 0x76f05ee8

53624

53625 exit_branch_type=0x0 bb->exit_target=0x76f05ee8

53626 convert_to_near_rel: cbz/cbnz opcode

53627 exit_branch_type=0x11 target=0x76f05f52 l->flags=0x11

53628 exit_branch_type=0x0 target=0x76f05ee8 l->flags=0x1001

53629 Exit cti 0x6b033206 is targeting 0x6b033210 + 0x0 => 0x6b033210

53630 Exit cti 0x6b03320c is targeting 0x6b033224 + 0x0 => 0x6b033224

53631 Fragment 1415, tag 0x76f05ee4, flags 0x9400030, shared, size 56:

53632

53633 Entry into F1415(0x76f05ee4).0x6b033204 (T32)(shared)

53634

53635 Exit from F1415(0x76f05ee4).0x6b033206 (T32)(shared)

53636 (target 0x76f05f52 not in cache)

53637

53638 d_r_dispatch: target = 0x76f05f52

53639

53640 interp: start_pc = 0x76f05f52

53641 SYSLOG_WARNING: Invalid opcode encountered

53642 Invalid Thumb opcode @0x76f05f52: 0xebfc8002

53643 decode: invalid instr at 0x76f05f52

53644 Invalid Thumb opcode @0x76f05f52: 0xebfc8002

53645 decode: invalid instr at 0x76f05f52

53646 0x76f05f52 ebfc 8002 <INVALID>

53647 interp: invalid instr at 0x76f05f52

53648 record_pending_signal(4 at pc 0x76f05f52): signal is currently blocked

53649 action is not SIG_IGN

53650 blocked fatal signal 4 cannot be delayed: terminating

53651 execute_default_action for signal 4

53652 SYSLOG_WARNING: Application /usr/bin/sftp (19343). Application exception at PC 0x76f05f52.

53653 Signal 4 delivered to application as default action.

53654 Callstack:

53657 0x76f05f52 </usr/lib/libcrypto.so.1.0.0+0x119f52>

53658 0xf8af04b0

53659

53660 Terminating via kill

53661 SYSLOG_INFORMATION: Stopping application /usr/bin/sftp (19343)

53662 synch with all threads my id = 19343 Giving 4 permission and seeking 3 state

53663 add_process_lock: 0 lock 0x54eb2a7c: name=all_threads_synch_lock(mutex)@/home/runner/work/dynamorio/dynamorio/core/synch .c:94

zhenhuan li

unread,

Nov 25, 2021, 12:46:02 AM11/25/21

to DynamoRIO Users

3643 /**** real work ***********************************************/

3644

3645 /* transfer control from signal handler to fcache return routine */

3646 static void

3647 transfer_from_sig_handler_to_fcache_return(dcontext_t *dcontext, kernel_ucontext_t *uc,

3648 sigcontext_t *sc_interrupted, int sig,

3649 app_pc next_pc, linkstub_t *last_exit,

3650 bool is_kernel_xfer)

3651 {

3652 sigcontext_t *sc = SIGCXT_FROM_UCXT(uc);

3653 if (is_kernel_xfer) {

3654 sig_full_cxt_t sc_interrupted_full = { sc_interrupted, NULL /*not provided*/ };

3655 sig_full_cxt_t sc_full;

3656 sig_full_initialize(&sc_full, uc);

3657 sc->SC_XIP = (ptr_uint_t)next_pc;

3658 /* i#4041: Provide the actually-interrupted mid-rseq PC to this event. */

3659 ptr_uint_t official_xl8 = sc_interrupted->SC_XIP;

3660 sc_interrupted->SC_XIP =

3661 (ptr_uint_t)translate_last_direct_translation(dcontext, (app_pc)official_xl8);

3662 if (instrument_kernel_xfer(dcontext, DR_XFER_SIGNAL_DELIVERY, sc_interrupted_full,

3663 NULL, NULL, next_pc, sc->SC_XSP, sc_full, NULL, sig))

3664 next_pc = (app_pc)sc->SC_XIP;

3665 sc_interrupted->SC_XIP = official_xl8;

3666 }

3667 dcontext->next_tag = canonicalize_pc_target(dcontext, next_pc);

3668 IF_ARM(dr_set_isa_mode(dcontext, get_pc_mode_from_cpsr(sc), NULL));

3669

3670 /* Set our sigreturn context to point to fcache_return!

3671 * Then we'll go back through kernel, appear in fcache_return,

3672 * and go through d_r_dispatch & interp, without messing up dynamo stack.

3673 * Note that even if this is a write in the shared cache, we

3674 * still go to the private fcache_return for simplicity.

3675 */

3676 sc->SC_XIP = (ptr_uint_t)fcache_return_routine(dcontext);

3677 #ifdef AARCHXX

3678 /* We do not have to set dr_reg_stolen in dcontext's mcontext here

3679 * because dcontext's mcontext is stale and we used the mcontext

3680 * created from recreate_app_state_internal with the original sigcontext.

3681 */

3682 /* We restore dr_reg_stolen's app value in recreate_app_state_internal,

3683 * so now we need set dr_reg_stolen to hold DR's TLS before sigreturn

3684 * from DR's handler.

3685 */

3686 ASSERT(get_sigcxt_stolen_reg(sc) != (reg_t)*get_dr_tls_base_addr());

3687 /* Preserve the translated value. */

3688 dcontext->local_state->spill_space.reg_stolen = get_sigcxt_stolen_reg(sc);

3689 /* Now put DR's base in the sigcontext. */

3690 set_sigcxt_stolen_reg(sc, (reg_t)*get_dr_tls_base_addr());

3691 # ifndef AARCH64

3692 /* We're going to our fcache_return gencode which uses DEFAULT_ISA_MODE */

3693 set_pc_mode_in_cpsr(sc, DEFAULT_ISA_MODE);

3694 # endif

3695 #endif

3696

3697 #if defined(X64) || defined(ARM)

3698 /* x64 always uses shared gencode */

3699 dcontext->local_state->spill_space.IF_X86_ELSE(xax, r0) =

3700 sc->IF_X86_ELSE(SC_XAX, SC_R0);

3701 # ifdef AARCH64

3702 /* X1 needs to be spilled because of br x1 in exit stubs. */

3703 dcontext->local_state->spill_space.r1 = sc->SC_R1;

3704 # endif

3705 #else

3706 get_mcontext(dcontext)->IF_X86_ELSE(xax, r0) = sc->IF_X86_ELSE(SC_XAX, SC_R0);

3707 #endif

3708 LOG(THREAD, LOG_ASYNCH, 2, "\tsaved xax " PFX "\n", sc->IF_X86_ELSE(SC_XAX, SC_R0));

3709

3710 sc->IF_X86_ELSE(SC_XAX, SC_R0) = (ptr_uint_t)last_exit;

3711 LOG(THREAD, LOG_ASYNCH, 2, "\tset next_tag to " PFX ", resuming in fcache_return\n",

3712 next_pc);

3713 LOG(THREAD, LOG_ASYNCH, 3, "transfer_from_sig_handler_to_fcache_return\n");

3714 DOLOG(3, LOG_ASYNCH, {

3715 LOG(THREAD, LOG_ASYNCH, 3, "sigcontext @" PFX ":\n", sc);

3716 dump_sigcontext(dcontext, sc);

3717 });

3718 }

transfer_from_sig_handler_to_fcache_return function Analysis:

according from the above function:

At line 3667, because next_pc is 0x76f0593c ,because the pc bit[0] is zero, so this line will set isa_mode DR_ISA_ARM_A32

3667 dcontext->next_tag = canonicalize_pc_target(dcontext, next_pc);

at line 3668, get_pc_mode_from_cpsr(sc) will return DR_ISA_ARM_THUMB in this case.so this line will set isa_mode DR_ISA_ARM_THUMB

3668 IF_ARM(dr_set_isa_mode(dcontext, get_pc_mode_from_cpsr(sc), NULL));

why get_pc_mode_from_cpsr return DR_ISA_ARM_THUMB in this case，we can see the function define in the following:

2713 # ifndef AARCH64

2714 static dr_isa_mode_t

2715 get_pc_mode_from_cpsr(sigcontext_t *sc)

2716 {

2717 return TEST(EFLAGS_T, sc->SC_XFLAGS) ? DR_ISA_ARM_THUMB : DR_ISA_ARM_A32;

2718 }

because EFLAGS_T = 0x00000020,sc->SC_XFLAGS =0x600b0030 according from the DR output, so get_pc_mode_from_cpsr return DR_ISA_ARM_THUMB

53563 transfer_from_sig_handler_to_fcache_return

53564 sigcontext @0x6aff4d24:

53565 r0 =0x54de219c

53566 r1 =0x00000000

53567 r2 =0x76f574ac

53568 r3 =0x00000008

53569 r4 =0x76f57534

53570 r5 =0x76f573a8

53571 r6 =0x76f575b4

53572 r7 =0x7ec48b18

53573 r8 =0x00000001

53574 r9 =0x7ec48c0c

53575 r10 =0x6afc5000

53576 r11 =0x00000000

53577 r12 =0x76f57410

53578 sp =0x7ec48700

53579 r14 =0x76c32800

53580 pc =0x6af94ee0

53581 cpsr=0x600b0030

53582 set next_tag to handler 0x76f0593c, xsp to 0x7ec48700

53583 master_signal_handler 4 returning now to 0x6af94ee0

because the opcode at 0x76f0593c must be decode at DR_ISA_ARM_A32 mode , but the isa_mode is set to be DR_ISA_ARM_THUMB, cause a conflict here.

I hope my analysis can help solve this problem，Looking forward to receiving your reply。

Derek Bruening

unread,

Nov 26, 2021, 3:09:21 PM11/26/21

to dynamor...@googlegroups.com

It might be better to file an issue at this point to store these details and keep the list traffic lighter.

Further info below:

On Wed, Nov 24, 2021 at 8:38 PM zhenhuan li <lizhen...@gmail.com> wrote:

Thanks for response.
The following is the output of DR with "loglevel 3" , in 53560 line ,DR switch to ARM mode, but at 53589 line , the DR use thumb decode the opcode.I think this is the first incorrectly place.

The IDA disassembly :
.text:0011993C ; void __fastcall __noreturn sub_11993C(int val)
.text:0011993C sub_11993C ; DATA XREF: OPENSSL_cpuid_setup+F0↑o
.text:0011993C ; .text:off_3F9A8↑o
.text:0011993C 00 10 A0 E1 MOV R1, R0 ; val
.text:00119940 08 00 9F E5 LDR R0, =(unk_16B3A8 - 0x119950)
.text:00119944 10 40 2D E9 STMFD SP!, {R4,LR}
.text:00119948 00 00 8F E0 ADD R0, PC, R0 ; unk_16B3A8 ; env
.text:0011994C 24 97 FC EB BL siglongjmp
.text:0011994C ; End of function sub_11993C

The DR output :
53434 recreate_app : looking for 0x6b033198 in frag @ 0x6b033198 (tag 0x76f05a08)

This "receare_app" is a translation for a fault. In fact we see below that this is for delivering a SIGILL to the application:

so that SIGILL is the first problem, before the bad thumb transition which seems to be as part of passing this SIGILL to the app.

So this MRRC instruction raised the SIGILL. Does it raise SIGILL natively (not under DR)?

Run under gdb to see.

As you pointed out the 0x0e does seem misleading when the sub-opcode is 1:

"mrrc p15, #1, r0, r1, c14".

This p15,c14,0x1 combo is CNTVCT it looks like.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/5950d64f-b066-40fb-8805-53cfbb5ed5dcn%40googlegroups.com.

zhenhuan li

unread,

Nov 29, 2021, 4:13:53 AM11/29/21

to DynamoRIO Users

Thanks for your response!

you are right! MRRC instruction raised the SIGILL natively(not under DR).

the following output is i run sftp natively under gdb.

./gdb-build/bin/gdb sftp

GNU gdb (GDB) 10.2

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "arm-cortexa9_neon-linux-gnueabihf".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from sftp...

(No debugging symbols found in sftp)

(gdb) r

Starting program: /usr/bin/sftp

Program received signal SIGILL, Illegal instruction.

0x76f7ca08 in _armv7_tick () from /lib/libcrypto.so.1.0.0

(gdb) bt

#0 0x76f7ca08 in _armv7_tick () from /lib/libcrypto.so.1.0.0

#1 0x76ea2934 in OPENSSL_cpuid_setup () from /lib/libcrypto.so.1.0.0

#2 0x76fdefa8 in call_init (l=<optimized out>, argc=argc@entry=1,

argv=argv@entry=0x7efffd34, env=env@entry=0x7efffd3c) at dl-init.c:72

#3 0x76fdf104 in call_init (env=<optimized out>, argv=<optimized out>,

argc=<optimized out>, l=<optimized out>) at dl-init.c:30

#4 _dl_init (main_map=0x76fff958, argc=1, argv=0x7efffd34, env=0x7efffd3c) at dl-init.c:120

#5 0x76fcfb04 in _dl_start_user () from /lib/ld-linux-armhf.so.3

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) disassemble 0x76f7ca08

Dump of assembler code for function _armv7_tick:

=> 0x76f7ca08 <+0>: mrrc 15, 1, r0, r1, cr14

0x76f7ca0c <+4>: bx lr

End of assembler dump.

(gdb) c

Continuing.

usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]

[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]

[-o ssh_option] [-P port] [-R num_requests] [-S program]

[-s subsystem | sftp_server] host

sftp [user@]host[:file ...]

sftp [user@]host[:dir[/]]

sftp -b batchfile [user@]host

[Inferior 1 (process 11536) exited with code 01]

(gdb)

The following may useful information to diag this issue which is output by DR with --loglevel 3.

53317 interp: start_pc = 0x76e2b930

53318 0x76e2b930 eb036834 bl $0x76f05a08 -> %lr

53319 NOT following direct call from 0x76e2b930 to 0x76f05a08

53320 end_pc = 0x76e2b934

53321

53322 exit_branch_type=0x9 bb->exit_target=0x76f05a08

53323 exit_branch_type=0x9 target=0x76f05a08 l->flags=0x1809

53324 Exit cti 0x6b033178 is targeting 0x6b03317c + 0x0 => 0x6b03317c

53325 Fragment 1411, tag 0x76e2b930, flags 0x1000030, shared, size 36:

53326

53327 Entry into F1411(0x76e2b930).0x6b033170 (A32)(shared)

53328

53329 Exit from F1411(0x76e2b930).0x6b033178 (A32)(shared)

53330 (target 0x76f05a08 not in cache)

53331

53332 d_r_dispatch: target = 0x76f05a08

53333

53334 interp: start_pc = 0x76f05a08

53335 0x76f05a08 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53336 0x76f05a0c e12fff1e bx %lr

53337 end_pc = 0x76f05a10

53338

53339 exit_branch_type=0x6 bb->exit_target=0x6af95180

53340 emit_fragment: bb use ibl <0x6af95180>

53341 exit_branch_type=0x6 target=0x6af95180 l->flags=0x1006

53342 Exit cti 0x6b0331a4 is targeting 0x6b0331a8 + 0x0 => 0x6b0331a8

53343 Fragment 1412, tag 0x76f05a08, flags 0x1000030, shared, size 36:

53344

53345 Entry into F1412(0x76f05a08).0x6b033198 (A32)(shared)

53346

53347

53348 master_signal_handler: thread=19343, sig=4, xsp=0x6aff4c90, retaddr=0x00000004

53349 siginfo: sig = 4, pid = 1795371416, status = -1414812757, errno = 0, si_code = 1

53350 r0 =0x00000000

53351 r1 =0x00000000

53352 r2 =0x76f574ac

53353 r3 =0x00000008

53354 r4 =0x76f57534

53355 r5 =0x76f573a8

53356 r6 =0x76f575b4

53357 r7 =0x7ec48b18

53358 r8 =0x00000001

53359 r9 =0x7ec48c0c

53360 r10 =0x6afc5000

53361 r11 =0x00000000

53362 r12 =0x76f57410

53363 sp =0x7ec48a00

53364 r14 =0x76e2b934

53365 pc =0x6b033198

53366 cpsr=0x600b0010

53367 handle_nudge_signal: sig=4 code=1 errno=0

53368 record_pending_signal(4) from cache pc 0x6b033198

53369 not certain can delay so handling now

53370 action is not SIG_IGN

53371 add_process_lock: 0 lock 0x6af66dc0: name=synch_lock(mutex)@/home/runner/work/dynamorio/dynamorio/core/synch.c:144

53372 rank=10 owner=19343 owning_dc=0x6afaf040 contended_event=0xffffffff prev=0x00000000

53373 lock count_times_acquired= 1 0 0 0 0+2 synch_lock(mutex)@/home/runner/work/dynamorio/dynamorio/core/synch.c:144

53374 translate context, thread 19343 at pc_recreatable spot translating

53375 recreate_app_state -- translating from:

53376 priv_mcontext_t @0x6aff49ac

53377 r0 = 0x00000000

53378 r1 = 0x00000000

53379 r2 = 0x76f574ac

53380 r3 = 0x00000008

53381 r4 = 0x76f57534

53382 r5 = 0x76f573a8

53383 r6 = 0x76f575b4

53384 r7 = 0x7ec48b18

53385 r8 = 0x00000001

53386 r9 = 0x7ec48c0c

53387 r10 = 0x6afc5000

53388 r11 = 0x00000000

53389 r12 = 0x76f57410

53390 r13 = 0x7ec48a00

53391 r14 = 0x76e2b934

53392 r15 = 0x6b033198

53393 q0 = 0x76f575b4 7ec48b18 00000001 76f70254

53394 q1 = 0x00000000 00000000 00000000 00000000

53395 q2 = 0x00000000 00000000 00000000 00000000

53396 q3 = 0x00000000 00000000 00000000 00000000

53397 q4 = 0x00000000 00000000 00000000 00000000

53398 q5 = 0x00000000 00000000 00000000 00000000

53399 q6 = 0x00000000 00000000 00000000 00000000

53400 q7 = 0x00000000 00000000 00000000 00000000

53401 q8 = 0x00001000 00000000 0000000c 00000000

53402 q9 = 0x0000000d 00000000 00000000 00000000

53403 q10 = 0x00000000 00000000 00000000 00000000

53404 q11 = 0x00000000 00000000 00000000 00000000

53405 q12 = 0x00000000 00000000 00000000 00000000

53406 q13 = 0x00000000 00000000 00000000 00000000

53407 q14 = 0x00000000 00000000 00000000 00000000

53408 q15 = 0x00000000 00000000 00000000 00000000

53409 eflags = 0x600b0010

53410 pc = 0x6b033198

53411

53412 building bb instrlist now *********************

53413

53414 interp: start_pc = 0x76f05a08

53415 0x76f05a08 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53416 0x76f05a0c e12fff1e bx %lr

53417 end_pc = 0x76f05a10

53418

53419 setting cur_pc (for fall-through) to 0x76f05a10

53420 exit_branch_type=0x6 bb->exit_target=0x6af95180

53421

53422 done building bb instrlist *********************

53423

53424 recreate_app : pc is in F1412(0x76f05a08)

53425 ilist for recreation:

53426 TAG 0x76f05a08

53427 +0 L3 @0x6afdffe8 ec510f1e mrrc $0x0f $0x0e %c14 -> %r0 %r1

53428 +4 m4 @0x6afdfea8 e58a2008 str %r2 -> +0x08(%r10)[4byte]

53429 +8 m4 @0x6afdebe0 e58a2008 <label>

53430 +8 m4 @0x6afdf708 e1a0200e mov %lr -> %r2

53431 +12 L4 @0x6afdfe08 eafe82fc b $0x6af95180 <shared_bb_ibl_ret>

53432 END 0x76f05a08

53433

53434 recreate_app : looking for 0x6b033198 in frag @ 0x6b033198 (tag 0x76f05a08)

53435 recreate_app -- found valid state pc 0x76f05a08

53483 0x6b03319c e58a2008 str %r2 -> +0x08(%r10)[4byte]

......

Derek Bruening

unread,

Nov 29, 2021, 3:43:34 PM11/29/21

to dynamor...@googlegroups.com

If you could file an issue in our tracker I would again suggest moving further discussion there for figuring out the incorrect mode switch.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/70fbd9b9-361f-44a0-be99-55a1d958befdn%40googlegroups.com.

zhenhuan li

unread,

Nov 29, 2021, 9:08:23 PM11/29/21

to DynamoRIO Users

Thanks for your suggestion, I have file an issue at this link https://github.com/DynamoRIO/dynamorio/issues/5233

Reply all

Reply to author

Forward