Re: Building XS modules with mismatched compiler suites on Windows

[bulk88]

Steve Hay wrote:
> In a couple of my XS modules I've long hand some rudimentary checks in
> place to disallow building with a different compiler suite to that
> which perl was built with, e.g. to disallow building with VC12 when
> perl was built with VC6.
>
> It has been mentioned to me several times over the years that there
> should not be a problem with using a different compiler suite so the
> checks are unnecessary, and I've finally got round to having a look...
>
> It seems to be true in general that there is no problem, but there is
> at least one thing that can go wrong due to VC7 and higher using
> versioned C RTL DLLs (msvcr70.dll etc instead of msvcrt.dll) so that
> two different C RTLs may get loaded (one by perl and a different one
> by the XS module): If you use a mix of CRT I/O functions, some of
> which are redirected by perl and others not, then you can end up
> passing a file descriptor that was opened in one C RTL to the other C
> RTL, which knows nothing about it.
>
> This can happen, for example, if you obtain a file descriptor using
> _sopen() (a Microsoft-specific variant of open(), which PerlIO doesn't
> redefine) and pass it to close() (a standard C function, which PerlIO
> does redefine). In this case the XS module's C RTL opens the file
> descriptor, but because close() is redefined to a call within perl it
> is perl's C RTL that is asked to close it, which will fail if that's a
> different C RTL.
>

This is documented by MS.
http://msdn.microsoft.com/en-us/library/ms235460.aspx

> A similar thing can happen when mixing Windows API functions
> (obviously not redefined by perl) with standard C functions, e.g.
> opening a file descriptor with open() (using perl's C RTL) and then
> passing it to _get_osfhandle() (in the XS module's C RTL).
>
> I have found cases exactly like the above in my code and am in the
> process of fixing them (either by using exclusively standard C
> functions, all redefined by perl, or else by using exclusively
> Microsoft-specific functions, none redefined by perl), but I have run
> into another problem which causes certain combinations of VC++
> versions to fail.

Then dont mix up CRTs in the first place. There are also the win32_*
functions that are exported that redirect to Perl's CRT DLL if you want
the same CRT as Perl.

>
> One module crashes when using a member of the MY_CXT structure;
> another crashes when calling chsize(), which is redefined as
> (*my_perl->ILIO->pChsize)() when PERL_IMPLICIT_SYS is defined. In each
> case, the struct member being used (MY_CXT.err_str in the former case;
> my_perl->ILIO->pChsize in the latter case) is unexpectedly NULL
> (0x00000000) with some combinations of C RTL DLLs.

Show the entire MY_CXT struct you created. Remember a MY_CXT struct can
never leave the compiland it was defined in. They are effectivly C
static declared. There is also win32_chsize. And NO_XSLOCKS define. If
my_perl->ILIO->pChsize (a vtable entry) is NULL, the problem is
somewhere in Perl, or your embeddeding code of Perl interp. I dont
understand "MY_CXT.err_str" to comment on that. If you have an
unthreaded Win32 Perl, my_perl is still a valid C symbol, but if you use
it, you will be broiled ;)
http://perl5.git.perl.org/perl.git/blob/HEAD:/perl.h#l4894 .

>
> The crashes only happen when using a perl built with VC8 or higher and
> using VC6 or VC7 to build the XS modules. Fortunately, that's an
> unlikely combination: If you have a VC8 or higher perl then you
> probably built it yourself, so you would probably build the XS module
> with the same version of VC++.
>
> Mismatches the other way around do not seem to pose any problem, e.g.
> a VC6 build of perl with VC12 being used to build the XS module.
> (Which is good, since that's a more likely scenario, e.g. ActivePerl
> used to be built with VC6, and is now built with a gcc port that also
> uses msvcrt.dll like VC6 did.)
>
> It would still be nice to understand what the problem is though. I'm
> not sure if it's something I'm doing wrong, something wrong in perl, a
> compiler bug that was fixed in VC8, or just some other fundamental
> incompatibility between <=VC7 DLLs and >=VC8 DLLs?
>
> The attached module illustrates the problem.
>

I'll try running it with a VC 2008/9.0 Perl with VC 2003/7.0 XS DLL.

> Using a VC8 or higher build of perl but trying to build this module
> with VC6 or VC7 causes both tests to fail:
>
> C:\Dev\Temp\Foo-1.00>perl -Mblib t\test.t
> 1..2
> # pOpen() is 00000000
> not ok 1
> # Failed test at t\test.t line 7.
> # buf is 00000000
> not ok 2
> # Failed test at t\test.t line 8.
> # Looks like you failed 2 tests of 2.
>
> Using the same VC8 or higher build but now using VC8 or higher (but
> not necessarily the same as was used for perl) to build this module is
> fine:
>
> C:\Dev\Temp\Foo-1.00>perl -Mblib t\test.t
> 1..2
> # pOpen() is 61931230
> ok 1
> # buf is 003479A4
> ok 2
>
> Using a VC6 or VC7 build of perl is also fine, whatever version of
> VC++ is used to build this module.
>
> Can anyone explain what the problem here could be? Is it something
> that I or we (p5p) could fix, or is it just a compiler problem that we
> can't do anything about?

Can you attach your VC6/VC7 Foo.dll binary/DLL for me to look at?

[bulk88]

bulk88 wrote:

> Steve Hay wrote:
>> Can anyone explain what the problem here could be? Is it something
>> that I or we (p5p) could fix, or is it just a compiler problem that we
>> can't do anything about?
>
> Can you attach your VC6/VC7 Foo.dll binary/DLL for me to look at?

And its pdb file.

[bulk88]

NVM. Reproduced.

In VC2008, ILIO is at 0x8E4. In VC2003, ILIO is at 0x8CC.

VC2003
--------------------------------------------------------
38: dMY_CXT;
39: printf("# pOpen() is %p\n", my_perl->ILIO->pOpen);
100010A7 8D 9E CC 08 00 00 lea ebx,[esi+8CCh]
100010AD 8B 03 mov eax,dword ptr [ebx]
100010AF FF 70 3C push dword ptr [eax+3Ch]
100010B2 68 00 21 00 10 push offset string "# pOpen() is %p\n"
(10002100h)
100010B7 FF 15 20 20 00 10 call dword ptr [__imp__printf
(10002020h)]
--------------------------------------------------------
VC2008
--------------------------------------------------------
38: dMY_CXT;
39: printf("# pOpen() is %p\n", my_perl->ILIO->pOpen);
01FA10A7 8D 9E E4 08 00 00 lea ebx,[esi+8E4h]
01FA10AD 8B 03 mov eax,dword ptr [ebx]
01FA10AF FF 70 3C push dword ptr [eax+3Ch]
01FA10B2 68 EC 20 FA 01 push offset string "# pOpen() is %p\n"
(1FA20ECh)
01FA10B7 FF 15 7C 20 FA 01 call dword ptr [__imp__printf (1FA207Ch)]
--------------------------------------------------------

This is a bug. I will investigate further. I suggest you file a RT
ticket. You can also investigate it yourself now. The C structs that
make up the interp struct don't match. This is wrong.

[Steve Hay]

Thanks for this. That gives me some more to look into.

What about the MY_CXT problem -- have you reproducd that too? (The
reference to MY_CXT.err_str was talking about my real code; I renamed
it MY_CXT.buf in the Foo module that I attached, but it was otherwise
just the same. I also used Open rather than Chsize in the Foo module
since it is better known outside of Windows-land than Chsize, but
again they exhibit the same problem.)

I'm aware of the potential problems passing CRT objects across DLL
boundaries -- that's why I was previously checking that the same
compiler was in use. But with care it should be possible to avoid the
things that cause problems, and that's what I've been trying to do so
that I can remove the compiler version checks.

I'm also aware of the exported win32_* functions. They are often what
gets called at the end of the day when you simply write something like
open() in XS code. The point is that perl's header files are intended
to set up whatever redirections/redefinitions are necessary so that
simply calling open() Does The Right Thing for whatever build of perl
you have (e.g. threaded or not) -- and using perl's CRT too, so that
one does not have to worry about such things in XS code.

The trouble is that in this rather obscure case, things are not
working as intended.

[bulk88]

Steve Hay wrote

>
> Thanks for this. That gives me some more to look into.

See new RT ticket.

>
> What about the MY_CXT problem -- have you reproducd that too? (The
> reference to MY_CXT.err_str was talking about my real code; I renamed
> it MY_CXT.buf in the Foo module that I attached, but it was otherwise
> just the same. I also used Open rather than Chsize in the Foo module
> since it is better known outside of Windows-land than Chsize, but
> again they exhibit the same problem.)

PL_my_cxt_list used in dMY_CXT is after PL_statbuf in intrpvar.h. Any
interp member after PL_statbuf is corrupt from old VC XS on new VC Perl
bug. 0x9A8 and 0x9C0 are higher than PL_LIO (0x8CC/0x8E4). Same bug.


VC2003
----------------------------------------------------------------
216: if (items != 0)
1000110A F7 C1 FC FF FF FF test ecx,0FFFFFFFCh
10001110 74 0F je $L37199 (10001121h)
217: croak_xs_usage(cv, "");
10001112 68 11 21 00 10 push offset string "" (10002111h)
10001117 FF 74 24 14 push dword ptr [esp+14h]
1000111B FF 15 34 20 00 10 call dword ptr
[__imp__Perl_croak_xs_usage (10002034h)]
--- c:\sources\foo-1.00\foo.xs
-------------------------------------------------
44: }
45:
46: void
47: two()
48: PROTOTYPE:
49:
50: PPCODE:
51: {
52: dMY_CXT;
10001121 8B 86 A8 09 00 00 mov eax,dword ptr [esi+9A8h]
10001127 8B 0D 10 30 00 10 mov ecx,dword ptr [___xi_z+4
(10003010h)]
1000112D 53 push ebx
1000112E 8B 1C 88 mov ebx,dword ptr [eax+ecx*4]
53: printf("# buf is %p\n", MY_CXT.buf);
10001131 53 push ebx
10001132 68 F0 20 00 10 push offset string "# buf is %p\n"
(100020F0h)
10001137 FF 15 20 20 00 10 call dword ptr [__imp__printf
(10002020h)]
1000113D 59 pop ecx
1000113E 59 pop ecx
54: if (MY_CXT.buf)
1000113F 85 DB test ebx,ebx
10001141 5B pop ebx
10001142 74 1A je $L37199+3Dh (1000115Eh)
----------------------------------------------------------------
VC2008
----------------------------------------------------------------
216: if (items != 0)
01FA110A F7 C1 FC FF FF FF test ecx,0FFFFFFFCh
01FA1110 74 0F je $LN19 (1FA1121h)
217: croak_xs_usage(cv, "");
01FA1112 68 E8 20 FA 01 push offset string "" (1FA20E8h)
01FA1117 FF 74 24 14 push dword ptr [esp+14h]
01FA111B FF 15 8C 20 FA 01 call dword ptr
[__imp__Perl_croak_xs_usage (1FA208Ch)]
--- c:\sources\foo-1.00\foo.xs
-------------------------------------------------
44: }
45:
46: void
47: two()
48: PROTOTYPE:
49:
50: PPCODE:
51: {
52: dMY_CXT;
01FA1121 8B 86 C0 09 00 00 mov eax,dword ptr [esi+9C0h]
01FA1127 8B 0D 10 30 FA 01 mov ecx,dword ptr
[___native_vcclrit_reason+4 (1FA3010h)]
01FA112D 53 push ebx
01FA112E 8B 1C 88 mov ebx,dword ptr [eax+ecx*4]
53: printf("# buf is %p\n", MY_CXT.buf);
01FA1131 53 push ebx
01FA1132 68 00 21 FA 01 push offset string "# buf is %p\n"
(1FA2100h)
01FA1137 FF 15 7C 20 FA 01 call dword ptr [__imp__printf (1FA207Ch)]
01FA113D 59 pop ecx
01FA113E 59 pop ecx
54: if (MY_CXT.buf)
01FA113F 85 DB test ebx,ebx
01FA1141 5B pop ebx
01FA1142 74 1A je $LN19+3Dh (1FA115Eh)
----------------------------------------------------------------

[Dave Mitchell]

On Sun, Mar 16, 2014 at 01:15:58AM -0400, bulk88 wrote:
> PL_my_cxt_list used in dMY_CXT is after PL_statbuf in intrpvar.h.
> Any interp member after PL_statbuf is corrupt from old VC XS on new
> VC Perl bug. 0x9A8 and 0x9C0 are higher than PL_LIO (0x8CC/0x8E4).
> Same bug.

So presumably Stat_t is getting a different size or alignment under
different compilers?

--
Music lesson: a symbiotic relationship whereby a pupil's embellishments
concerning the amount of practice performed since the last lesson are
rewarded with embellishments from the teacher concerning the pupil's
progress over the corresponding period.

[bulk88]

Dave Mitchell wrote:
> On Sun, Mar 16, 2014 at 01:15:58AM -0400, bulk88 wrote:
>> PL_my_cxt_list used in dMY_CXT is after PL_statbuf in intrpvar.h.
>> Any interp member after PL_statbuf is corrupt from old VC XS on new
>> VC Perl bug. 0x9A8 and 0x9C0 are higher than PL_LIO (0x8CC/0x8E4).
>> Same bug.
>
> So presumably Stat_t is getting a different size or alignment under
> different compilers?
>

See #121448. Stat_t and Time_t resolve to 2 different sizes between 2003
and older, and 2005 and newer, related to MS define called
_USE_32BIT_TIME_T (Google it). Win32 Perl always definesStat_t and
Time_t to the same token regardless of CC version. But the different MS
CCs in the CC's headers then typedef or #define it to different tokens
which are different sizes. A quick summary of the problem is "32 bit
time_t vs 64 bit time_t".