Problem with Mingw crosscompiled vim on Win64

[Andy Kittner]

Hi All,

I usually build my windows vims by cross compiling them on my linux box.
This generally works like a charm, however over the years quite a few
hacks and ad-hoc fixes have accumulated and I have now finally decided
to get rid of them ;)

One of the few things that might be interesting for others is a fix for
the following phenomenon: On 32bit Windows the gvim binary works fine,
while on a 64bit Windows nothing happens when it is executed.

A little debugging at that time revealed that when the window procedure
for the text area is called with the WM_NCCREATE it essentially just
defers handling of the message to DefWindowProc which in turn returns
FALSE on my 64bit windows machines. According to the MSDN-docs this
aborts the window creation and the corresponding CreateWindowEx function
will receive a NULL handle.

To add to the mystery a build compiled the express edition of Visual
Studio worked fine as far as I remember.

Since then I have used the attached patch and just blindly returned TRUE
for NCCREATE. It must have worked without I'll effects because until now
I completely forgot about it ;)

So can anyone else reproduce this particular problem, or better yet, can
any win32-api gurus make some sense out of whats going on here?

As for the build environment:
Currently I am using gcc-4.4.2 with the windows headers and crt of the
mingw64 project from their svn-trunk. However when this first occurred I
was using the official release from the original mingw32 project.

I just verified that all this still happens in a VirtualBox installation
of Vista 64, with a vanilla vim build from the current mercurial tip
with this command:
make -f Make_ming.mak GUI=yes \
CC=i686-w64-mingw32-gcc \
CXX=i686-w64-mingw32-g++ \
WINDRES=i686-w64-mingw32-windres \
GETTEXT= ICONV= \
gvim.exe vimrun.exe

Best regards,
Andy
--
Once you've seen one nuclear war, you've seen them all.

[Sergey Khorev]

Andy,

> One of the few things that might be interesting for others is a fix for the
> following phenomenon: On 32bit Windows the gvim binary works fine,
> while on a 64bit Windows nothing happens when it is executed.
>

> So can anyone else reproduce this particular problem

I have similar problem with win32 gcc _debug_ build (MinGW gcc-3.4.5 on W7 x64).
I.e. normal win32 build works fine but once I add DEBUG=yes, Vim does
not display its window. A little debugging showed that it is caused by
s_textArea = CreateWindowEx(...) returning NULL but I did not track it
down to WM_NCCREATE. I thought this might have something to do with
gcc debug info and optimisations flags and postponed it.

Anyway, you patch fixed my problem too.

--
Sergey Khorev
http://sites.google.com/site/khorser
Can anybody think of a good tagline I can steal?

[Bram Moolenaar]

Andy Kittner wrote:

Looks harmless. Let me include this.

--
A bad peace is better than a good war. - Yiddish Proverb

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ download, build and distribute -- http://www.A-A-P.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Sergey Khorev]

> Since then I have used the attached patch and just blindly returned TRUE for
> NCCREATE. It must have worked without I'll effects because until now I

Btw the patch has a seemingly harmless side-effect: it makes client
window title empty instead of "Vim text area".
It seems one of the default actions on WM_NCCREATE is to set window title...

[Andy Kittner]

On Wed, Feb 03, 2010 at 09:42:33PM +0300, Sergey Khorev wrote:
>> Since then I have used the attached patch and just blindly returned TRUE for
>> NCCREATE. It must have worked without I'll effects because until now I
>
>Btw the patch has a seemingly harmless side-effect: it makes client
>window title empty instead of "Vim text area".
>It seems one of the default actions on WM_NCCREATE is to set window title...

Most interesting, as DefWindowProc is still called. So I guess that
indicates that _something_ goes wrong inside it. It doesn't bother to
put anything but "Operation successfully completed" or something like
that into GetLastError() though...

Regards,
Andy
--
I'll turn over a new leaf.
-- Miguel de Cervantes

[Sergey Khorev]

Andy,

> Most interesting, as DefWindowProc is still called. So I guess that
> indicates that _something_ goes wrong inside it. It doesn't bother to put
> anything but "Operation successfully completed" or something like that into
> GetLastError() though...

It seems I found what actually is wrong. In gui_w32.c we create ASCII
window class for text area, but _TextAreaWndProc forwards messages to
vim_WindowProc which in turn might call wide window proc!
The solution is to create wide text area window class if parent window is wide.

Attached patch solved my problem. Can you please try it and check
whether this helps you too or not.

[Andy Kittner]

On Thu, Feb 04, 2010 at 09:11:41AM +0300, Sergey Khorev wrote:
>Andy,
>
>> Most interesting, as DefWindowProc is still called. So I guess that
>> indicates that _something_ goes wrong inside it. It doesn't bother to put
>> anything but "Operation successfully completed" or something like that into
>> GetLastError() though...
>
>It seems I found what actually is wrong. In gui_w32.c we create ASCII
>window class for text area, but _TextAreaWndProc forwards messages to
>vim_WindowProc which in turn might call wide window proc!
>The solution is to create wide text area window class if parent window is wide.

Nice thanks for looking into it some more.

>Attached patch solved my problem. Can you please try it and check
>whether this helps you too or not.

I only did a quick check and got an error at startup, I'll take a closer
look this evening when I get back from work.

Regards,
Andy
--
Remember that whatever misfortune may be your lot, it could only be
worse in Cleveland.
-- National Lampoon, "Deteriorata"

[Andy Kittner]

On Thu, Feb 04, 2010 at 08:15:06AM +0100, Andy Kittner wrote:
>>It seems I found what actually is wrong. In gui_w32.c we create ASCII
>>window class for text area, but _TextAreaWndProc forwards messages to
>>vim_WindowProc which in turn might call wide window proc!
>>The solution is to create wide text area window class if parent window is wide.
>
>Nice thanks for looking into it some more.
>
>>Attached patch solved my problem. Can you please try it and check
>>whether this helps you too or not.
>
>I only did a quick check and got an error at startup, I'll take a
>closer look this evening when I get back from work.

Oh boy, if there is an award for stupidest person 2010, then I just got
the lead by a wide margin. When I tried out your patch this morning I
made a 32bit build and put it in the directory with my last build, which
as the directory name would have told had I bothered to read it,
contained 64bit versions of the iconv and gettext dlls...
Note to self: Don't try patches _before_ the morning coffee.

Long story short, your patch works fine for me too.

Thanks,
Andy
--
Too often I find that the volume of paper expands to fill the available
briefcases.
-- Governor Jerry Brown

[Sergey Khorev]

> Long story short, your patch works fine for me too.

Cool, thank you!

[John Marriott]

Hello. I have been following this thread with some interest because I
have also experienced the problem of being unable to build vim on Win64
using mingw64.

I have tried both the official patch and the unofficial one provided by
Sergey with success (ie I can build 64-bit gvim on Win64 using mingw64).
However, the "File->Save As..." command no longer works. It displays "
:browse confirm saveas" at the bottom of the screen but does not display
the file/folder "save as" requester window.

My OS is Win7 x64. MinGW is 4.4.4 20100129 (prerelease) autobuild from
the MinGW 64 site.

Any ideas would be greatly appreciated.

Regards
John

[Andy Kittner]

On Sun, Feb 07, 2010 at 09:07:38AM +1100, John Marriott wrote:
>[...]

>mingw64). However, the "File->Save As..." command no longer works. It
>displays "
>:browse confirm saveas" at the bottom of the screen but does not
>display the file/folder "save as" requester window.
>
>My OS is Win7 x64. MinGW is 4.4.4 20100129 (prerelease) autobuild from
>the MinGW 64 site.
>
>Any ideas would be greatly appreciated.

I never noticed this, as I don't use the gui browse dialogs, but I could
reproduce it. I traced it to some compatibility code with NT 4 (
use of OPENFILENAME_SIZE_VERSION_400 instead of the actual size of the
OPENFILENAMEW struct). With the attached patch the compatibility code is
commented out, and the browse dialog works again for me.

According to gdb fileStruct.lStructSize ends up as 8 bytes if
OPENFILENAME_SIZE_VERSION_400 is used, which seems quite small to me.
I'll try to see what other compilers (should have a vc-express and the
original mingw32 lying around here somewhere), maybe it's just some
bogus define in the mingw64 headers.

Regards,
Andy
--
Go not to the elves for counsel, for they will say both yes and no.
-- J.R.R. Tolkien

[Andy Kittner]

On Sun, Feb 07, 2010 at 12:46:34PM +0100, Andy Kittner wrote:
>According to gdb fileStruct.lStructSize ends up as 8 bytes

On second reading I see that the code uses
sizeof(OPENFILENAME_SIZE_VERSION_400) which is most likely not intended
here, as OPENFILENAME_SIZE_VERSION_400 already contains the required size.

Attached patch should fix the issue properly.

Regards,
Andy
--
I WILL NOT CUT CORNERS
" " " " " "
" " " " " "
" " " " " "

Bart Simpson on chalkboard in episode 7F11

[Bram Moolenaar]

Andy Kittner wrote:

> On Sun, Feb 07, 2010 at 12:46:34PM +0100, Andy Kittner wrote:
> >According to gdb fileStruct.lStructSize ends up as 8 bytes
> On second reading I see that the code uses
> sizeof(OPENFILENAME_SIZE_VERSION_400) which is most likely not intended
> here, as OPENFILENAME_SIZE_VERSION_400 already contains the required size.
>
> Attached patch should fix the issue properly.

Thanks! Strange that this ever worked.

--
Spam seems to be something useful to novices. Later you realize that
it's a bunch of indigestable junk that only clogs your system.
Applies to both the food and the e-mail!

[John Marriott]

Hi Andy,

You're right. It's fixed now.

Thanks very much!
Regards
John

[Andy Kittner]

On Mon, Feb 08, 2010 at 07:57:18AM +1100, John Marriott wrote:
>Hi Andy,
>
>You're right. It's fixed now.
>
>Thanks very much!
>Regards
>John

Your welcome! I use vim almost daily, so a little bug hunting is the
least I can do to give something back. Let me take this opportunity to
say a big thank you to Bram, and all the countless others, who have
maintained and improved vim over the years. Especially since it's tools
like vim, which give us geeks the reputation of being able to to magic ;)

Regards,
Andy
--
<carpaski> 84C does not require better cooling... It requires the
heatsink be reattached.

[John Marriott]

Is there a chance that this patch will become official?

On 08-Feb-2010 07:03, Bram Moolenaar wrote:
>