DLL import binding
Cesar Eduardo Barros
BindImage/BindImageEx on the installer)? It should in theory make
process startup a little faster (which for git is a good thing) [1].
From what I could find on Google (searching for the fixme from Wine's
BindImageEx), it seems the first parameter to BindImage is the DLL or
EXE filename and the second parameter is either NULL or a
semicolon-separated list of directories, like what you would put in
MSI's BindImage table [2]. The documentation for BindImage/BindImageEx
does not seem to explain it that well, and it is unimplemented by Wine.
You can check if it worked by looking at the EXE or DLL with objdump.
The import table has a Bound-To column which seems to be only filled if
you did the DLL binding.
[1] What is DLL import binding?
<http://blogs.msdn.com/b/oldnewthing/archive/2010/03/18/9980802.aspx>
[2] BindImage Table
<http://msdn.microsoft.com/en-us/library/aa367828%28v=vs.85%29.aspx>
--
Cesar Eduardo Barros
ces...@cesarb.net
cesar....@gmail.com
Johannes Schindelin
On Fri, 15 Apr 2011, Cesar Eduardo Barros wrote:
> Have you considered binding the EXEs and DLLs of msysgit (by calling
> BindImage/BindImageEx on the installer)? It should in theory make
> process startup a little faster (which for git is a good thing) [1].
Do you have a patch for us?
Ciao,
Dscho
Cesar Eduardo Barros
Unfortunately not yet. I would have first to study how to do things in
InnoSetup, which will take some time. Besides, I cannot easily test if
it worked (much less measure how much difference it made), since as I
said on the previous message, Wine does not implement BindImageEx.
I will take a look at InnoSetup later this week.
Cesar Eduardo Barros
Now I have one, which I pushed to mob as 5946786.
It calls BindImage as expected on Wine:
[...]
fixme:imagehlp:BindImageEx (0, "bin/git-upload-pack.exe", "C:\\Arquivos
de programas (x86)\\Git\\bin;C:\\Arquivos de programas
(x86)\\Git\\mingw\\bin;C:\\windows\\system32", (null), (nil)): stub
fixme:imagehlp:BindImageEx (0, "bin/git.exe", "C:\\Arquivos de programas
(x86)\\Git\\bin;C:\\Arquivos de programas
(x86)\\Git\\mingw\\bin;C:\\windows\\system32", (null), (nil)): stub
[...]
But, since Wine does not implement BindImageEx (BindImage is a one-line
wrapper around BindImageEx), I cannot tell if it does what I expect (the
MSDN documentation for BindImage/BindImageEx is not very clear). If it
works correctly, the output of objdump -x for the executable and dynamic
library files it was called on will have the "Bound-To" column filled on
its import tables, and in theory the executables will start a little faster.
Sebastian Schuberth
>>> Have you considered binding the EXEs and DLLs of msysgit (by calling
>>> BindImage/BindImageEx on the installer)? It should in theory make
>>> process startup a little faster (which for git is a good thing) [1].
>>
>> Do you have a patch for us?
>
> Now I have one, which I pushed to mob as 5946786.
Thanks for the patch, it looks good to me. A few minor issues / questions, though:
+ DllPath:=ExpandConstant('{app}\bin;{app}\mingw\bin;{sys}');
There is no "{app}\mingw\bin" in the directory Git for Windows installs to, I guess we should just drop it.
+ BindImage(ImageNames[i],DllPath,'');
Although we're not supporting per-Win2k OSes anymore, and thus the function pointer should initialize just fine, it would be good style to check BindImage for NIL before usage, or wrap the call to BindImage in a try ... except block. Here, I guess I'd prefer the first, e.g. like
+ if (BindImage<>NIL) and LoadStringsFromFile(FileName,ImageNames) then begin
or even wrap the whole code block in an if-statement (so DllPath and FileName are not even initialized if BindImage is NIL).
> MSDN documentation for BindImage/BindImageEx is not very clear). If it
> works correctly, the output of objdump -x for the executable and dynamic
> library files it was called on will have the "Bound-To" column filled on
> its import tables, and in theory the executables will start a little
> faster.
I'll give it a try tomorrow. Thanks again!
--
Sebastian Schuberth
Cesar Eduardo Barros
> On 01.05.2011 00:03, Cesar Eduardo Barros wrote:
>>>> Have you considered binding the EXEs and DLLs of msysgit (by calling
>>>> BindImage/BindImageEx on the installer)? It should in theory make
>>>> process startup a little faster (which for git is a good thing) [1].
>>>
>>> Do you have a patch for us?
>>
>> Now I have one, which I pushed to mob as 5946786.
>
> Thanks for the patch, it looks good to me. A few minor issues / questions, though:
>
> + DllPath:=ExpandConstant('{app}\bin;{app}\mingw\bin;{sys}');
>
> There is no "{app}\mingw\bin" in the directory Git for Windows installs to, I guess we should just drop it.
I saw that, which is why the find command in copy-files.sh does not look
into mingw/bin.
I kept it here to reflect how PATH is set in git.cmd and gitk.cmd (even
if the installer is not currently putting anything in mingw/bin, Windows
is looking into it when loading DLLs). But I see no problem with
dropping it.
> + BindImage(ImageNames[i],DllPath,'');
>
> Although we're not supporting per-Win2k OSes anymore, and thus the function pointer should initialize just fine, it would be good style to check BindImage for NIL before usage, or wrap the call to BindImage in a try ... except block. Here, I guess I'd prefer the first, e.g. like
>
> + if (BindImage<>NIL) and LoadStringsFromFile(FileName,ImageNames) then begin
>
> or even wrap the whole code block in an if-statement (so DllPath and FileName are not even initialized if BindImage is NIL).
Sounds good. I did not bother because pre-XP Windows versions are
obsolete (MSDN says BindImage/BindImageEx are Win2k or later), but I see
no problem in adding a test. Also, AFAIK MSDN is unreliable when it
comes to non-NT Windows versions; imagehlp.dll existed on Win9x
according to a quick Google search, and it is quite possible that
BindImage/BindImageEx already existed back then.
>> MSDN documentation for BindImage/BindImageEx is not very clear). If it
>> works correctly, the output of objdump -x for the executable and dynamic
>> library files it was called on will have the "Bound-To" column filled on
>> its import tables, and in theory the executables will start a little
>> faster.
>
> I'll give it a try tomorrow. Thanks again!
--
Johannes Schindelin
On Sat, 30 Apr 2011, Cesar Eduardo Barros wrote:
> Em 30-04-2011 20:16, Sebastian Schuberth escreveu:
> > On 01.05.2011 00:03, Cesar Eduardo Barros wrote:
> > > > > Have you considered binding the EXEs and DLLs of msysgit (by
> > > > > calling BindImage/BindImageEx on the installer)? It should in
> > > > > theory make process startup a little faster (which for git is a
> > > > > good thing) [1].
> > > >
> > > > Do you have a patch for us?
> > >
> > > Now I have one, which I pushed to mob as 5946786.
> >
> > Thanks for the patch, it looks good to me. A few minor issues /
> > questions, though:
> >
> > + DllPath:=ExpandConstant('{app}\bin;{app}\mingw\bin;{sys}');
> >
> > There is no "{app}\mingw\bin" in the directory Git for Windows
> > installs to, I guess we should just drop it.
>
> I saw that, which is why the find command in copy-files.sh does not look
> into mingw/bin.
>
> I kept it here to reflect how PATH is set in git.cmd and gitk.cmd (even
> if the installer is not currently putting anything in mingw/bin, Windows
> is looking into it when loading DLLs). But I see no problem with
> dropping it.
git.cmd and gitk.cmd are supposed to work both in the installed Git as
well as in the development environment; that's the whole reason why there
is mingw\bin, too.
Ciao,
Johannes
Sebastian Schuberth
<Johannes....@gmx.de> wrote:
>> > > Now I have one, which I pushed to mob as 5946786.
>> >
>> > Thanks for the patch, it looks good to me. A few minor issues /
>> > questions, though:
>> >
>> > + DllPath:=ExpandConstant('{app}\bin;{app}\mingw\bin;{sys}');
>> >
>> > There is no "{app}\mingw\bin" in the directory Git for Windows
>> > installs to, I guess we should just drop it.
>>
>> I saw that, which is why the find command in copy-files.sh does not look
>> into mingw/bin.
>>
>> I kept it here to reflect how PATH is set in git.cmd and gitk.cmd (even
>> if the installer is not currently putting anything in mingw/bin, Windows
>> is looking into it when loading DLLs). But I see no problem with
>> dropping it.
>
> git.cmd and gitk.cmd are supposed to work both in the installed Git as
> well as in the development environment; that's the whole reason why there
> is mingw\bin, too.
So I removed "{app}\mingw\bin", wrapped the BindImage stuff into a
try ... except block and performed some minor beautifications. I also
had to prefix AppDir to the ImageNames to make BindImage find the
files and succeed. See the ss/bindimage branch.
It basically works now, verified by running objdump, but unfortunately
running git.exe crashes now with error 0xc0000005 "the application was
unable to start correctly". I guess this is some manifest stuff that
probably is required for bound executables?
--
Sebastian Schuberth
Johannes Schindelin
On Sun, 1 May 2011, Sebastian Schuberth wrote:
> It basically works now, verified by running objdump, but unfortunately
> running git.exe crashes now with error 0xc0000005 "the application was
> unable to start correctly".
Ouch. That's an access violation, meaning that most likely it tried to
call a function that was not yet initialized...
Ciao,
Dscho
Sebastian Schuberth
>> It basically works now, verified by running objdump, but unfortunately
>> running git.exe crashes now with error 0xc0000005 "the application was
>> unable to start correctly". I guess this is some manifest stuff that
>> probably is required for bound executables?
> Unfortunately not.
> There appears to be some kind of problem between gcc created bianries
> and Windows loader.
>
> I ran into issue with GCC-4.5 as well.(only static builds are working)
>
> Exact cause is unkown,but I heared that it might be due to exported static data.
> Note:First,there is also bind.exe in sdk which can do binding. And
> second it works on exe produced by VisualC++ compiler. Others might
> as well.
Thanks Daniel. But please always use "reply to all" to keep all
involved people in the loop.
I experimented with bind.exe. I causes the same access violation
unless I specifc -o to "disable new import descriptors". I'm not sure
whether this has any impact on the performance theoretically gained.
Anyway, that switch seem to correspond to the BIND_NO_BOUND_IMPORTS
flag in BindImageEx [1]. Using that in the installer work OK.
Preliminary commits are on ss/bindimage again. I'll do some further
tests, squash some commits and merge soon unless I get negative
feedback.
[1] http://msdn.microsoft.com/en-us/library/ms679279%28v=vs.85%29.aspx
--
Sebastian Schuberth
Johannes Schindelin
On Sun, 1 May 2011, Sebastian Schuberth wrote:
> I experimented with bind.exe. I causes the same access violation unless
> I specifc -o to "disable new import descriptors". I'm not sure whether
> this has any impact on the performance theoretically gained. Anyway,
> that switch seem to correspond to the BIND_NO_BOUND_IMPORTS flag in
> BindImageEx [1]. Using that in the installer work OK. Preliminary
> commits are on ss/bindimage again. I'll do some further tests, squash
> some commits and merge soon unless I get negative feedback.
I'd be very much interested in knowing about the performance impact...
Ciao,
Dscho
Sebastian Schuberth
Note that I was talking about the (possible) performance impact
between using BindImage (or BindImageEx without BIND_NO_BOUND_IMPORTS)
and using BindImageEx with BIND_NO_BOUND_IMPORTS. As the first method
makes git.exe crash, we have no reference to compare to. Only the
second method works. However, in a few *very* simple tests (just
running "time git.exe" a few times), I was not able to observe a
consistently reduced load / execution time when comparing "BindImageEx
with BIND_NO_BOUND_IMPORTS" to the executables which were not bound at
all.
--
Sebastian Schuberth
Cesar Eduardo Barros
Does it show anything on objdump -x on the "Bound-To" column when using
BIND_NO_BOUND_IMPORTS? It is possible that using BIND_NO_BOUND_IMPORTS
disables the optimization which was the whole point of this exercise in
the first place. I have seen some people use BindImageEx to get a list
of the imports without changing the file, so it can be used in a
"read-only" way.
It is possible that only some of the executables (or DLL files) cause
the crash. If it is the case, they could be excluded from the list. I
would not be surprised if, for instance, the msys DLL was the culprit.
What happens if *only* git.exe is bound, for instance? Or if everything
else *but* git.exe is bound? Do the other executables also crash?
Interestingly, when looking for BIND_NO_BOUND_IMPORTS on Google, I found
that [1] it should be used on Win9x and [2] BindImageEx exists on NT4 or
later and Win95 or later.
[1] http://rmathew.com/articles/bind.html
[2] http://www.123helpme.com/view.asp?id=55472
Cesar Eduardo Barros
> Preliminary commits are on ss/bindimage again. I'll do some further
> tests, squash some commits and merge soon unless I get negative
> feedback.
I looked at the commits, and I am not sure if BIND_ALL_IMAGES is a good
idea. Wouldn't that risk modifying operating system DLLs?
Daniel Klíma
I got interesting results:
bind -u -o ->git works
bind -u ->git crashes
bind ->git works //bit strange
bind -o ->git works
(bind from SDK 7.1)
Slightly modified script I sent earlier where was -u removed seemed to work.
Hope it helps.
Daniel Klíma
Generally I obsereved major improvements with dll heavy applications
like Openoffice/LibreOffice.(~30% improvement)
For small programms like git(only few imports) it won't make (mostly)
noticable difference in small runs.
Also one warning - ASLR will cause binding skip ASLRed dll. (Just in
case a dll has this enabled)
That's why hinting is recommended to use in conjuction with binding.
(AFAIK compiler must support it)
BTW:For testing binding I created two-line batchscript:
for /R %%i in (*.dll) do bind -u -v -y "%%i"
for /R %%i in (*.exe) do bind -u -v -y "%%i"
(in case somebody would like to do mass-binding)
Sebastian Schuberth
> I looked at the commits, and I am not sure if BIND_ALL_IMAGES is a good
> idea. Wouldn't that risk modifying operating system DLLs?
I wasn't initially reading the flag's purpose like you do, but I guess
you're right. I'll remove that flag to be on the safe side.
--
Sebastian Schuberth
Sebastian Schuberth
> Does it show anything on objdump -x on the "Bound-To" column when using
> BIND_NO_BOUND_IMPORTS? It is possible that using BIND_NO_BOUND_IMPORTS
Yes, the "Bound-To" column is populated even with BIND_NO_BOUND_IMPORTS.
> disables the optimization which was the whole point of this exercise in the
> first place. I have seen some people use BindImageEx to get a list of the
That was my exact question (to myself) in the mail to Dscho. It is
possible that using BIND_NO_BOUND_IMPORTS has a negative performance
impact compared to not using BIND_NO_BOUND_IMPORTS, although the
"Bound-To" column is populated? I do not fully understand what "Do not
generate a new import address table" means. Does it mean the existing
one is modified instead, or does it mean that none is created /
modified (if that is even possible)?
> imports without changing the file, so it can be used in a "read-only" way.
Speaking of read-only, BIND_NO_UPDATE seems to imply that an image can
be bound without modified it. How can that be possible? Is the
information of the "Bound-To" column then stored elsewhere?
> crash. If it is the case, they could be excluded from the list. I would not
> be surprised if, for instance, the msys DLL was the culprit. What happens if
> *only* git.exe is bound, for instance? Or if everything else *but* git.exe
Even binding only git.exe (using bind.exe *without* -o) renders it unusable.
> is bound? Do the other executables also crash?
I tried not binding git.exe, but all other executables. This seems to
work, i.e. the other executables I've tested run fine. So maybe only
our self-compiled executable(s) are not compatible with bind.exe
*without* -o (aka not using BIND_NO_BOUND_IMPORT), but the binaries we
just take from MSYS are.
> Interestingly, when looking for BIND_NO_BOUND_IMPORTS on Google, I found
> that [1] it should be used on Win9x and [2] BindImageEx exists on NT4 or
> later and Win95 or later.
Yes, I've seen that piece of code, too.
--
Sebastian Schuberth
Sebastian Schuberth
> bind ->git works //bit strange
> bind -o ->git works
That's not really surprising, as without explicitly specifying -u the
image file is not modified at all (making me wonder what sense it
makes to ever omit it).
--
Sebastian Schuberth
Cesar Eduardo Barros
> On Sun, May 1, 2011 at 15:22, Cesar Eduardo Barros<ces...@cesarb.net> wrote:
>
>> Does it show anything on objdump -x on the "Bound-To" column when using
>> BIND_NO_BOUND_IMPORTS? It is possible that using BIND_NO_BOUND_IMPORTS
>
> Yes, the "Bound-To" column is populated even with BIND_NO_BOUND_IMPORTS.
Hmm...
>> disables the optimization which was the whole point of this exercise in the
>> first place. I have seen some people use BindImageEx to get a list of the
>
> That was my exact question (to myself) in the mail to Dscho. It is
> possible that using BIND_NO_BOUND_IMPORTS has a negative performance
> impact compared to not using BIND_NO_BOUND_IMPORTS, although the
> "Bound-To" column is populated? I do not fully understand what "Do not
> generate a new import address table" means. Does it mean the existing
> one is modified instead, or does it mean that none is created /
> modified (if that is even possible)?
AFAIK the table always exists, the loader only fills it. I think the
purpose of this flag is beginning to become clear to me: reuse the
existing table (only pre-filling it) versus discard the existing table
and generate a new one.
If that is the case, the correct thing for us would be of course to use
this flag to keep mingw's table and avoid any incompatibilities.
>> imports without changing the file, so it can be used in a "read-only" way.
>
> Speaking of read-only, BIND_NO_UPDATE seems to imply that an image can
> be bound without modified it. How can that be possible? Is the
> information of the "Bound-To" column then stored elsewhere?
It is used to query the imports (without modifying the file). The
callback (last parameter to BindImageEx) will be called for each import.
At least py2exe and cx_Freeze have problems on Wine because this
function is unimplemented, see http://bugs.winehq.org/show_bug.cgi?id=3591.
To see how cx_Freeze uses it, take a look at
http://cx-freeze.svn.sourceforge.net/viewvc/cx-freeze/trunk/source/util.c?revision=193
(it uses BIND_NO_BOUND_IMPORTS | BIND_NO_UPDATE | BIND_ALL_IMAGES plus a
callback function).
>> crash. If it is the case, they could be excluded from the list. I would not
>> be surprised if, for instance, the msys DLL was the culprit. What happens if
>> *only* git.exe is bound, for instance? Or if everything else *but* git.exe
>
> Even binding only git.exe (using bind.exe *without* -o) renders it unusable.
>
>> is bound? Do the other executables also crash?
>
> I tried not binding git.exe, but all other executables. This seems to
> work, i.e. the other executables I've tested run fine. So maybe only
> our self-compiled executable(s) are not compatible with bind.exe
> *without* -o (aka not using BIND_NO_BOUND_IMPORT), but the binaries we
> just take from MSYS are.
Perhaps it is something git.exe does? I think I read somewhere (perhaps
even in this thread) that variable imports are problematic; doesn't
git.exe set some variable from msvcrt.dll? Perhaps gcc does variable
imports in a way subtly different from MSVC, or perhaps BindImageEx is
just buggy?
So, it seems the best thing would be to keep using BIND_NO_BOUND_IMPORTS
(with or without BIND_CACHE_IMPORT_DLLS, which should not make any
difference other than installer speed unless BindImageEx is buggy).
Unfortunately, as Daniel Klíma said elsewhere in the thread, git might
have too few imports for all this BindImage work to make a noticeable
speed difference.
Daniel Klíma
Forgot. As for what reason - maybe in conjuction with -v to get full
information what and were it would bind. Something like
troubleshooting.
Sebastian Schuberth
> So, it seems the best thing would be to keep using BIND_NO_BOUND_IMPORTS
> (with or without BIND_CACHE_IMPORT_DLLS, which should not make any
> difference other than installer speed unless BindImageEx is buggy).
>
> Unfortunately, as Daniel Klíma said elsewhere in the thread, git might have
> too few imports for all this BindImage work to make a noticeable speed
> difference.
I agree, this seems to be the outcome of our little experiment. So how
to we proceed? Cesar, would you be OK with me squashing our commits
and merging them to devel? Even if we do not get a noticeable speed
improvement I think the code is worth keeping (as long as we discover
no side effects).
--
Sebastian Schuberth
Cesar Eduardo Barros
I am OK with that.
Sebastian Schuberth
>> I agree, this seems to be the outcome of our little experiment. So how
>> to we proceed? Cesar, would you be OK with me squashing our commits
>> and merging them to devel? Even if we do not get a noticeable speed
>> improvement I think the code is worth keeping (as long as we discover
>> no side effects).
>>
>
> I am OK with that.
Squashed and pushed to devel.
--
Sebastian Schuberth