How to choose between "app" and "meta" for generated instructions?

[edmund.gri...@gmail.com]

I am working with a client that emulates certain instructions, so it replaces an app instruction with a sequence of instructions that may include branches and calls to libraries. It is often unclear to me which of my generated instructions should be marked as "meta". Sometimes it seems to work either way, and sometimes I see sporadic failures if I change the way an instruction is marked.

In particular, if I replace an original app instruction with a sequence of only meta instructions then, although I haven't found a simple example that fails, I tend to see failures with larger programs. Is there some optimisation or clean-up process that only applies to large programs and might be confused by a sequence of meta instructions replacing an original app instruction? (I understand that there is no opportunity to have a translation field referring to the original instruction if there are no app instructions in the emulation code but I don't think this problem has to do with signals because the source code of the app does not refer to signals and I don't see any signals when I run it under a debugger.)

For performance reasons, my generated instructions sometimes operate on the stolen register: those instructions must be marked as meta to avoid DynamoRIO mangling them, so that's one constraint. And the instructions generated by certain API calls are automatically meta, so that's another constraint. But for some instructions I just don't know whether I should be marking them as meta so I'd be grateful for any general guidance on that.

I suppose another question might be: under what circumstances do I need to set the translation field on a meta instruction and then have a restore_state callback? Is that only if my meta instruction might cause a precise signal?

Edmund

[Abhinav Sharma]

Hi,

> I understand that there is no opportunity to have a translation field referring to the original instruction if there are no app instructions in the emulation code

Even though the emulation code may not retain any original app instr, some emulation instr can still be marked as app with the translation field set. E.g., see the scatter gather expansion on x86 which removes the original scatter gather instr and emulates it with a sequence of scalar stores and loads and sets the pc of the original app instr in some of them: https://github.com/DynamoRIO/dynamorio/blob/0ab4d54744519e01854784612a83ddc57a16b0ad/ext/drx/scatter_gather_x86.c#L293. This helps DR in restarting execution correctly if there's a signal.

> but I don't think this problem has to do with signals because the source code of the app does not refer to signals and I don't see any signals when I run it under a debugger.

Even though the app doesn't have signals, DR may still send signals for its own operation and may want to reconstruct app state. So I think it's still important to have the translation field set appropriately.

> under what circumstances do I need to set the translation field on a meta instruction and then have a restore_state callback? Is that only if my meta instruction might cause a precise signal?

I believe it is important to set the translation field even if the instr may not cause a precise signal. The assumption about no signals in the app is not true anymore for the app running under DR.

What I'm not completely sure about: I know DR tries to delay signals until we're at a safe spot (e.g., end of fragment). So is there nothing in DR's signal handling code that would try to translate app state if we're not at a safe spot?

Also, we recently encountered some instances of missing read/write records in drmemtrace traces, which may indicate some mechanism/bug leading to a signal getting delivered of place when we don't expect it (https://github.com/DynamoRIO/dynamorio/issues/7050#issuecomment-2443133453). https://github.com/DynamoRIO/dynamorio/issues/5063 mentions some cases this may happen. Also, https://github.com/DynamoRIO/dynamorio/issues/5790 mentions DR may look for a "safe spot" within a fragment too.

> Sometimes it seems to work either way, and sometimes I see sporadic failures if I change the way an instruction is marked.

For the crashes you see, do you have more debug info? Where does the crash happen (in the code cache, in the middle of an emulated instr, or core DR, or your client?). Does the crash reproduce with the DR debug build, with logs enabled?

Abhinav

[Abhinav Sharma]

> I believe it is important to set the translation field even if the instr may not cause a precise signal.

I should clarify: I mean that there should be some instr in the instrumented fragment with its translation field set to the pc of the emulated app instr (even if no meta instr is expected to cause a precise signal).

[edmund.gri...@gmail.com]

> Sometimes it seems to work either way, and sometimes I see sporadic failures if I change the way an instruction is marked.

For the crashes you see, do you have more debug info? Where does the crash happen (in the code cache, in the middle of an emulated instr, or core DR, or your client?). Does the crash reproduce with the DR debug build, with logs enabled?

The failures I saw weren't crashes, unfortunately; there were wrong numbers coming out at the end of the run. So it's a bit more difficult to investigate.

Since the instructions we're emulating do not often interact with control flow, omitting an emulated instruction or running it twice could have an effect like that. I'll post again here if I get some more information.

Thanks,

Edmund

--

[Derek Bruening]

> What I'm not completely sure about: I know DR tries to delay signals until we're at a safe spot (e.g., end of fragment). So is there nothing in DR's signal handling code that would try to translate app state if we're not at a safe spot?

DR only delays the application's own asynchronous signals.  DR's own signals used for synchronous flushes, cache resets, thread synchronization, and detach are *not* delayed: wherever they land, DR tries to translate that spot, and if the translation fails, it has to let the target thread keep running and try again.  So if a client adds a whole bunch of instrumentation without translations, it can cause delays in these cross-thread synchronization operations where DR keeps trying to find spots with valid state but fails as its signals keep landing on instrumentation without translation.

I believe app vs meta is more about mangling: if the instruction uses a PC-relative load or uses the stolen register or TLS register or is a branch.  Safest to mark all the emulating instructions as app, esp those that use from-original-app-instruction operands, to avoid failures from omitted mangling.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/dadb921f-b63c-49bc-81b6-98bd8dc22f32n%40googlegroups.com.

[Abhinav Sharma]

> So it's a bit more difficult to investigate.

Also wanted to note the drstatecmp extension (https://dynamorio.org/group__drstatecmp.html) which may help in detecting such instrumentation-induced bugs in app state. In drmemtrace it is enabled using the -enable_drstatecmp option (https://dynamorio.org/sec_drcachesim_ops.html); you may similarly want to try this in your client too.

To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/CAO1ikSYwN_yjz9xcd6HrCbD6k1OWGL7_ZtFaq%3Dr1BsXAE%2BVpGg%40mail.gmail.com.