Error: dll starting at ... not found. Program exited with code 037777777777

[JimB]

When I run one of my apps under drmemory.exe, it ends before my
program does any real work, giving no error message.

So I run drmemory.exe under gdb, and gdb tells me:
Error: dll starting at 0x160000 not found.
Program exited with code 037777777777.

All my dlls are in the same directory as my .exe (and depends.exe is
fine with them). Same is true for drmemory itself (typical install).
I tried it with a couple other apps and get similar results -- all
these apps run to completion on their own.

As a point of reference, I run drmemory under gdb as such:
gdb --args drmemory.exe %ComSpec% /C "echo hello"

It seems to run to completion, giving the summary ("Details: ..."),
but still exits with:
Error: dll starting at 0x170000 not found.
Program exited with code 01.

So what's that tell us?

Platform: Win32. (Windows 7 x64)
DrMemory: 1.4.5, installed.

[JimB]

On Feb 8, 11:48 am, JimB <j...@jc-bell.com> wrote:
> So I run drmemory.exe under gdb, and gdb tells me:
>     Error: dll starting at 0x160000 not found.
>     Program exited with code 037777777777.

Oops: more research shows that the "Error: ..." is a gdb artifact,
64/32-bit <http://stackoverflow.com/questions/3982194/why-do-i-get-
missing-dll-errors-debugging-my-fortran-code-using-gdb-on-windows-7>.

But the program still mysteriously exits on its own.

Also, "-verbose 30" and -dr_ops "-dumpcore_mask 0x8bff" seem to have
no effect.

[Derek Bruening]

On Wed, Feb 8, 2012 at 12:48 PM, JimB <j...@jc-bell.com> wrote:

When I run one of my apps under drmemory.exe, it ends before my
program does any real work, giving no error message.

What kind of app is this?  Given that you're using gdb, is this app built with the Cygwin toolchain?  Dr. Memory does not currently officially support Cygwin applications.

- Derek

[JimB]

On Feb 8, 12:08 pm, Derek Bruening <bruen...@google.com> wrote:
> ...

>
> What kind of app is this?  Given that you're using gdb, is this app built
> with the Cygwin toolchain?  Dr. Memory does not currently officially
> support Cygwin applications.
>

It's built with MinGW-32/gcc-4.4. But a binary is a binary, right? (I
realize I'll have to do address translations myself.)

[Derek Bruening]

There are a number of problems with cygwin1.dll relating to early threads and heap behavior.  Eventually these will be supported, but it's not our highest priority.

Other than symbols (which should be supported in the near future), apps built with MinGW-32 gcc should work in principle, though we have not tested them thoroughly.  Simple apps built with gcc 4.6 work fine in our tests on win7x64 and winXPx64.

Pass "-debug" to drmemory.exe and see if it provides further information.  If not, if you're willing to make your app (or a simplified version that exhibits the problem) available, please file an Issue in our tracker and attach the app code.

- Derek

[JimB]

On Feb 8, 10:14 pm, Derek Bruening <bruen...@google.com> wrote:
> On Wed, Feb 8, 2012 at 10:01 PM, JimB <j...@jc-bell.com> wrote:
>

> ...

>
> > It's built with MinGW-32/gcc-4.4.  But a binary is a binary, right? (I
> > realize I'll have to do address translations myself.)
>

> ...

>
> Other than symbols (which should be supported in the near future), apps
> built with MinGW-32 gcc should work in principle, though we have not tested
> them thoroughly.  Simple apps built with gcc 4.6 work fine in our tests on
> win7x64 and winXPx64.
>
> Pass "-debug" to drmemory.exe and see if it provides further information.
>  If not, if you're willing to make your app (or a simplified version that
> exhibits the problem) available, please file an Issue in our tracker and
> attach the app code.

I added the -debug flag but don't see a difference in behavior or
reporting. (There's too much code to attach to a tracker, and it's
proprietary.)

Is the global.<pid>.log file of interest? It looks pretty complete,
and has a variety of potentially-interesting things in it.

E.g.:
WARNING: requested max frames truncates suppression callstacks
NOT using redzones for _dbg routines in msvcrt.dll 0x76c30000
WARNING: unable to disable crtdbg checks
WARNING: new == new[] => disabling mismatch detection for msvcrt.dll
WARNING: unable to load symbols for E:\Project\X\NonSrc\Test.exe
WARNING: ignore_unaddr_table entry came to slowpath (x 141)
WARNING: unhandled system calls found

It makes it to the end of the SECOND "**** per-500K-slowpath stats
dump:", but I see no further progress after that. It happens whether
or not my -suppress is enabled, too.

Is there a debug version or a way to instrument it differently?

[Derek Bruening]

On Thu, Feb 9, 2012 at 5:12 PM, JimB <j...@jc-bell.com> wrote:

I added the -debug flag but don't see a difference in behavior or
reporting. (There's too much code to attach to a tracker, and it's
proprietary.)

Is the global.<pid>.log file of interest? It looks pretty complete,
and has a variety of potentially-interesting things in it.

Yes, this is the main source of debugging information.  -verbose N controls how much (it did nothing before because you were running the release build).

I would suggest:

1) Run under drmemory.exe with the debug build of DynamoRIO via "-dr_debug" and see if DynamoRIO reports any errors

2) Does it work under any of the following options to drmemory.exe:

  A) -leaks_only -no_zero_stack

  B) -no_check_uninitialized -no_count_leaks

  C) -no_replace_libc -no_replace_realloc -no_repstr_to_loop

3) If it doesn't work under -leaks_only, does it work under plain DynamoRIO?  (http://code.google.com/p/dynamorio/downloads/list)

4) Run it under windbg (not sure I trust gdb on windows)

- Derek

[Derek Bruening]

On Thu, Feb 9, 2012 at 11:53 PM, Derek Bruening <brue...@google.com> wrote:

4) Run it under windbg (not sure I trust gdb on windows)

On Windows, note that drmemory.exe creates a child process where the app is run, so you need to either follow children in the debugger or attach after startup (use -dr_ops "-msgbox_mask 15" for an attach point).

- Derek

 

[JimB]

On Feb 9, 10:53 pm, Derek Bruening <bruen...@google.com> wrote:
> ...

> I would suggest:
>
> 1) Run under drmemory.exe with the debug build of DynamoRIO via "-dr_debug"
> and see if DynamoRIO reports any errors

EUREKA!

"Applications <my-app> <my-PID>.
Internal Error DynamoRIO Error: Not implemented @..\..\..\trunk
\dynamorio\core\win32\ntdll.c:990 (0) && "i#437: no ymm CONTEXT
support yet"
(Error occurred @ 21197 frags)
version 3.0.1152, custom build
-client_lib
'E:\Software\Open\app\DrMemory/bin/release/drmemorylib.dll;0;`-
callstack_style` `2` -suppress ....
...

(Please forgive typos, as I couldn't copy & paste the above from the
pop-up window.)

This message does seem to stop processing, but seems also to happen
sooner than it did before.

> 2) Does it work under any of the following options to drmemory.exe:
>   A) -leaks_only -no_zero_stack

These flags give a different error under -dr_debug:
Internal Error Internal DynamoRIO Error: ....\core\win32\ntdll.c:954
TESTALL(CONTEXT_DR_STATE, cxt->ContextFlags)
(Error occurred @42639 frags)

>   B) -no_check_uninitialized -no_count_leaks

Back to the first error:
...\ntdll.c:990 (0) && "i#437: no ymm CONTEXT support yet"

>   C) -no_replace_libc -no_replace_realloc -no_repstr_to_loop

Same as previous. (But note that "@xxxx frags" is always different.)

> 3) If it doesn't work under -leaks_only, does it work under plain
> DynamoRIO?  (http://code.google.com/p/dynamorio/downloads/list)
> 4) Run it under windbg (not sure I trust gdb on windows)

re gdb: agreed. But the above probably render it unnecessary.

So does the above ultimately mean that MinGW really isn't supported?

I was hoping we'd run drmemory under drmemory itself, in true
Ouroboros fashon. ;-)

[Derek Bruening]

On Sat, Feb 11, 2012 at 3:03 PM, JimB <j...@jc-bell.com> wrote:

"Applications <my-app> <my-PID>.
Internal Error DynamoRIO Error: Not implemented @..\..\..\trunk
\dynamorio\core\win32\ntdll.c:990 (0) && "i#437: no ymm CONTEXT
support yet"

Ah, ok, this is a known issue.  You have an AVX-enabled processor and Win7 SP1, I take it?

This issue is fixed in the latest DynamoRIO, but we have not put out a new Dr. Memory package containing the fix.  We were planning to put out a new release relatively soon, but if you want the fix sooner you could get the 3.1 DynamoRIO binaries (http://code.google.com/p/dynamorio/downloads/list) and replace Dr. Memory's dynamorio/lib32/release/dynamorio.dll.

- Derek

[JimB]

On Feb 11, 3:34 pm, Derek Bruening <bruen...@google.com> wrote:
> On Sat, Feb 11, 2012 at 3:03 PM, JimB <j...@jc-bell.com> wrote:
> > "Applications <my-app> <my-PID>.
> > Internal Error DynamoRIO Error: Not implemented @..\..\..\trunk
> > \dynamorio\core\win32\ntdll.c:990 (0) && "i#437: no ymm CONTEXT
> > support yet"
>
> Ah, ok, this is a known issue.  You have an AVX-enabled processor and Win7
> SP1, I take it?

Yeah, it's an x64 AMD & Win7, pretty new. (I'll just nod like I know
what AVX-enabled means...)

>
> This issue is fixed in the latest DynamoRIO, but we have not put out a new
> Dr. Memory package containing the fix.  We were planning to put out a new
> release relatively soon, but if you want the fix sooner you could get the
> 3.1 DynamoRIO binaries (http://code.google.com/p/dynamorio/downloads/list)
> and replace Dr. Memory's dynamorio/lib32/release/dynamorio.dll.

So swapped out the DLL as instructed. It ran very fast (50% of full-
speed!) and to completion, but seems lost with respect to the file
system: I can't tell that it's picking up my suppress, and at the end
I get...

WARNING: unable to locate results file since can't open E:\Project\X
\zDrMem\Raw/resfile.5612: 2

And best I can tell, it's not doing a thing (except running the app):
catching no false positives or anything, even with my suppress turned
off.

When I run with -dr_debug (swapping out dynamorio/lib32/debug/
dynamorio.dll with the latest), it's unhappy right out of the gate:

DynamoRIO usage error : client library is incompatible with this
version of DR.

I'm resigning myself to waiting for the next release, but I'm glad to
try anything else you suggest.

[Derek Bruening]

On Mon, Feb 13, 2012 at 10:26 AM, JimB <j...@jc-bell.com> wrote:

DynamoRIO usage error : client library is incompatible with this
version of DR.

Sorry, my bad, binary compatibility change means the library swap I suggested won't work for this particular revision.

I'm resigning myself to waiting for the next release, but I'm glad to
try anything else you suggest.

If you don't mind trying this release candidate which has experimental MinGW symbol support as well: http://drmemory.org/DrMemory-Windows-1.4.6-1.zip

- Derek

[JimB]

On Feb 14, 1:21 pm, Derek Bruening <bruen...@google.com> wrote:
...

>
> If you don't mind trying this release candidate which has experimental
> MinGW symbol support as well:http://drmemory.org/DrMemory-Windows-1.4.6-1.zip
>

First, THANK YOU for putting in the MinGW symbol look-up! It works
great and is AWESOME! You ROCK!

Here's what -dr_debug is telling me:
DynamoRIO usage error : instr_encode error: no encoding found.

Then:
(..\..\..\..\withwiki\trunk\dynamorio\core\x86\encode.c, line 2285)
version 3.1.1251, custom build
...

And quits there. This is progress!

[Derek Bruening]

On Sat, Feb 18, 2012 at 3:45 PM, JimB <j...@jc-bell.com> wrote:

First, THANK YOU for putting in the MinGW symbol look-up!  It works
great and is AWESOME! You ROCK!

So DrMem works fine on at least some of your applications, then, and the error below is only on a subset of your applications?  Or the error is well after startup and after some reported callstacks where you see symbols?

 

Here's what -dr_debug is telling me:
DynamoRIO usage error : instr_encode error: no encoding found.

Can you add '-dr_ops "-loglevel 1"', locate the logfile in the dynamorio/logs/ directory, and paste the contents of the line "ERROR: Could not find encoding for"?

- Derek

[JimB]

On Feb 20, 11:11 am, Derek Bruening <bruen...@google.com> wrote:

> So DrMem works fine on at least some of your applications, then, and the
> error below is only on a subset of your applications?  Or the error is well
> after startup and after some reported callstacks where you see symbols?

The latter. Of my two "interesting" apps, neither runs to completion
under drmemory, but I see callstacks and much more progress. I'd say
most of the way through the smaller one, not sure about the bigger
one.

>
> > Here's what -dr_debug is telling me:
> > DynamoRIO usage error : instr_encode error: no encoding found.
>
> Can you add '-dr_ops "-loglevel 1"', locate the logfile in the
> dynamorio/logs/ directory, and paste the contents of the line "ERROR: Could
> not find encoding for"?

Current indcall_bb table bb ibl indcall lookup coll%=0.0000,
dyn.avgcoll=1.3284
ERROR: Could not find encoding for: mov %cl ->
SYSLOG_ERROR: Application E:\Project\X\X.exe (6044) DynamoRIO usage

error : instr_encode error: no encoding found

SYSLOG_ERROR: Usage error: instr_encode error: no encoding found (..\..

\..\..\withwiki\trunk\dynamorio\core\x86\encode.c, line 2285)
version 3.1.1251, custom build

-loglevel 1 -client_lib 'E:\Software\Open\app\DrMemory/bin/release/
drmemorylib.dll;0;`-callstack_style` `2` -suppress `E:\Project\X\zDrMem
\Raw\suppress.txt` -logdir `E:\Project\X
0x2090e640 0x5e9f3007
0x2090e668 0x5eac0ae2
0x2090e798 0x5eac317c
0x2090e7b8 0x5eadc49a
0x2090e7f0 0x5eadd06c
0x2090e824 0x5e9eb617
0x2090edc0 0x5e9f2cc8
0x2090ede8 0x5eaf328d
0x2090ef40 0x5e9dd2c3
0x2090eff4 0x208b210e
DynamoRIO call stack:
frame ptr 0x2090e5e4 => parent 0x2090e640, ret = 0x5e9f75e3
[dynamorio.dll]
frame ptr 0x2090e640 => parent 0x2090e668, ret = 0x5e9f3007
[dynamorio.dll]
frame ptr 0x2090e668 => parent 0x2090e798, ret = 0x5eac0ae2
[dynamorio.dll]
frame ptr 0x2090e798 => parent 0x2090e7b8, ret = 0x5eac317c
[dynamorio.dll]
frame ptr 0x2090e7b8 => parent 0x2090e7f0, ret = 0x5eadc49a
[dynamorio.dll]
frame ptr 0x2090e7f0 => parent 0x2090e824, ret = 0x5eadd06c
[dynamorio.dll]
frame ptr 0x2090e824 => parent 0x2090edc0, ret = 0x5e9eb617
[dynamorio.dll]
frame ptr 0x2090edc0 => parent 0x2090ede8, ret = 0x5e9f2cc8
[dynamorio.dll]
frame ptr 0x2090ede8 => parent 0x2090ef40, ret = 0x5eaf328d
[dynamorio.dll]
frame ptr 0x2090ef40 => parent 0x2090eff4, ret = 0x5e9dd2c3
[dynamorio.dll]
frame ptr 0x2090eff4 => parent 0x0028f808, ret = 0x208b210e []
frame ptr 0x0028f808 => parent 0x0028fbb8, ret = 0x0047b2b7 [X.exe]
frame ptr 0x0028fbb8 => parent 0x0028fcb8, ret = 0x00647d40 [X.exe]
...


(Note this log file is labeled log.0.<pid>.html, but it's not HTML
format.)

[Derek Bruening]

On Tue, Feb 21, 2012 at 5:25 PM, JimB <j...@jc-bell.com> wrote:

> > Here's what -dr_debug is telling me:

> > DynamoRIO usage error : instr_encode error: no encoding found.
>
> Can you add '-dr_ops "-loglevel 1"', locate the logfile in the
> dynamorio/logs/ directory, and paste the contents of the line "ERROR: Could
> not find encoding for"?

Current indcall_bb table bb ibl indcall lookup  coll%=0.0000,
dyn.avgcoll=1.3284
ERROR: Could not find encoding for: mov    %cl ->
SYSLOG_ERROR: Application E:\Project\X\X.exe (6044) DynamoRIO usage

error : instr_encode error: no encoding found

This would be much easier w/ a reproducible run or a core dump though it sounds like neither is possible for your app.

I'm going to need the app code for the block being instrumented.  Can you run with "-debug -verbose 3" and paste the final block instruction list at the end of the Dr. Memory log file (logs/DrMemory-<app>.<pid><#>.global.<tid>.log).  It will be something like this:

in event_basic_block(tag=0x77e801b4)

TAG  0x77e801b4

 +0    L3              89 44 24 04          mov    %eax -> 0x04(%esp) 

 +4    L3              89 5c 24 08          mov    %ebx -> 0x08(%esp) 

 +8    L3              e9 e9 9c 02 00       jmp    $0x77ea9eaa 

END 0x77e801b4

- Derek