Detected Nondeterminism in Dr. Memory between two runs of the same executable under same configurations

[Alen Jo]

I had originally created an GitHub Issue, but was told to redirect here so here it is: 

Hi Dr. Memory Maintainers and Team!

I have been using Dr. Memory as a tool to check for nondeterministic behaviors in dynamic code analysis tools. Using the tool, I saw that out of our sample of 52 repositories, we found 20 repositories with nondeterministic reports across each of the ten runs we've conducted.

To Reproduce
1. Run on GitHub Actions utilizing a runner with Ubuntu 22.04 and Dr. Memory 2.6.0 installed
2. Make sure you have ten runs of this GitHub Action with your desired executable (in our case, we've used the project kcp on GitHub and compiled as an executable)
3. Obtain 10 of the Dr. Memory global logs for the kcp executable

Expected behavior
Consistent number of errors reported, the total/unique count of errors, consistency in the duplicate error count sections, and the fields such as the byte counts for "still-reachable allocations."

Screenshots or Pasted Text
The attached PDF is a file difference of the project kcp's (found on GitHub) executable, where Run 1 is in the left column and Run 2 is in the right column:

Observed results
In Run 1, we noticed the following:

• There are only four errors reported (two possible memory leaks and two memory leaks)
• Errors found show two unique and 85 errors for possible memory leaks + the size of possible memory leak bytes & actual memory leak bytes
• Errors ignored show eight unique and 8 total errors for "still-reachable allocations" + byte sizes.

In Run 2, we noticed the following:

• There is an additional error reported (two possible memory leaks and three memory leaks)
• Error found shows three unique and three total errors for possible memory leaks + the size of possible memory leak bytes & actual memory leak bytes
• Errors ignored shows 9 unique and 15 total errors for "still-reachable allocations" + byte sizes

Versions

• We are using Dr. Memory 2.6.0, which is the official latest version from the Dr. Memory webpage
• We are using Ubuntu 22.04 on GitHub Actions
• kcp should be a 64-bit ELF binary application

Would only the Dr. Memory team and maintainers look into this insight and offer remedies or reasons for this behavior? Thank you so much!!

[Alen Jo]

I am writing this message from the original GitHub Issue I posted for #6956 as I was redirected here:

Hi Dr. Memory Maintainers and Team!

I have been using Dr. Memory as a tool to check for nondeterministic behaviors in dynamic code analysis tools. Using the tool, I saw that out of our sample of 52 repositories, we found 20 repositories with nondeterministic reports across each of the ten runs we've conducted.

To Reproduce
1. Run on GitHub Actions utilizing a runner with Ubuntu 22.04 and Dr. Memory 2.6.0 installed
2. Make sure you have ten runs of this GitHub Action with your desired executable (in our case, we've used the project kcp on GitHub and compiled as an executable)
3. Obtain 10 of the Dr. Memory global logs for the kcp executable

Expected behavior
Consistent number of errors reported, the total/unique count of errors, consistency in the duplicate error count sections, and the fields such as the byte counts for "still-reachable allocations."

Screenshots or Pasted Text

The attached screenshot is a file difference of the project kcp's executable, where Run 1 is in the left column and Run 2 is in the right column (attached below this message):

Observed results

In Run 1, I noticed the following:

• There are only four errors reported (two possible memory leaks and two memory leaks)
• Errors found show two unique and 85 errors for possible memory leaks + the size of possible memory leak bytes & actual memory leak bytes
• Errors ignored show eight unique and 8 total errors for "still-reachable allocations" + byte sizes.

In Run 2, I noticed the following:

• There is an additional error reported (two possible memory leaks and three memory leaks)
• Error found shows three unique and three total errors for possible memory leaks + the size of possible memory leak bytes & actual memory leak bytes
• Errors ignored shows 9 unique and 15 total errors for "still-reachable allocations" + byte sizes

Versions

• We are using Dr. Memory 2.6.0, which is the official latest version from the Dr. Memory webpage
• We are using Ubuntu 22.04

• kcp should be a 64-bit ELF binary application

Would only the Dr. Memory maintainers and team look into this insight and offer remedies or reasons for this behavior? Thank you so much!!

[Derek Bruening]

Please see my reply to a similar issue that you raised here: https://github.com/DynamoRIO/dynamorio/issues/6958#issuecomment-2321731587

The same comments apply here: there is non-determinism in most applications, some coming from the system libraries used by all dynamically linked applications.

Furthermore, many applications themselves contain varying behavior: did you first study your target application(s) by themselves to see how they vary?  You can't assume a large application plus dynamic libraries plus kernel plus hardware will behave exactly the same in every run.  As mentioned in the aforementioned issue, even "hello,world" if dynamically linked often has varying instruction counts across runs on the same machine.

For the "possible leaks": some of the leak categorization in Dr. Memory and similar tools involves some heuristics in identifying pointers in memory scans, especially for "possible leaks".  If you look at the description https://drmemory.org/page_leaks.html you can see these involve a possible pointer to the middle of a heap block, which can result from an incorrectly identified non-pointer (this is why it's "possible" and not for-sure).  Multiple heap blocks are connected to avoid too many separate reports for connected objects, affecting the bytes reported. Thus, variations in data values can cause slight differences in "possible leaks" and bytes.

Most likely there are in fact slight variations in values across runs leading to these diffferences.  There doesn't seem to be anything here indicating a problem with the Dr. Memory tool.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/4e9693c6-1bdc-4405-8474-3f8c274383ban%40googlegroups.com.

[Alen Jo]

I see, thank you for the "possible leaks" explanation. Could you also explain the jump from 85 total definite memory leak errors for Run #1 of the screenshot attached compared to only three total definite memory leak errors for run #2?