Move DevKit towards mingw-w64 (4.7.1)

[Luis Lavena]

Hello,

A few months ago I started a similar conversation:

https://groups.google.com/d/topic/rubyinstaller/h32LCbWTGJg/discussion

And lot have changed since we discussed this.

Today, I just verified that using Ruben's mingw-w64 builds for GCC
4.7.1 (win32 threading) Ruby 2.0 results in:

ruby -v: ruby 2.0.0dev (2012-09-07 trunk 36920) [i386-mingw32]
11339 tests, 1939137 assertions, 1 failures, 0 errors, 82 skips

Being the only remaining failure related to RubyGems (which will be
merged once RubyGems 1.8.25 gets released)

Before running tests, I've also rebuild all the Knapsack packages with
GCC 4.7.1 and tested both, existing and newer compiled ones, all with
same results (good ones).

My objective is move both x86 and x64 builds to the same compiler
toolchain and headers: mingw-w64.

What do you think?
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[Jon]

> Hello,
>
> A few months ago I started a similar conversation:
>
> https://groups.google.com/d/topic/rubyinstaller/h32LCbWTGJg/discussion
>
> And lot have changed since we discussed this.
>
> Today, I just verified that using Ruben's mingw-w64 builds for GCC
> 4.7.1 (win32 threading) Ruby 2.0 results in:
>
> ruby -v: ruby 2.0.0dev (2012-09-07 trunk 36920) [i386-mingw32]
> 11339 tests, 1939137 assertions, 1 failures, 0 errors, 82 skips
>
> Being the only remaining failure related to RubyGems (which will be
> merged once RubyGems 1.8.25 gets released)
>
> Before running tests, I've also rebuild all the Knapsack packages with
> GCC 4.7.1 and tested both, existing and newer compiled ones, all with
> same results (good ones).
>
> My objective is move both x86 and x64 builds to the same compiler
> toolchain and headers: mingw-w64.
>
> What do you think?

I like it.

This weekend I'll look at moving things around for 4.7.1 as per
https://github.com/oneclick/rubyinstaller/issues/130 so a few of us
can double check your build/test results on different systems.

We should also double check for any non-thread related deps issues.
IIRC 4.6.2 had different runtime deps issues than our current default
4.5.2 (TDM) but I think they were all C++ related. I don't think we'll
find any runtime dep issues using Ruben's 4.7.1 builds, but I'm just
paranoid enough to want us to double check ;)


Jon

---
Fail fast. Fail often. Fail publicly. Learn. Adapt. Repeat.
http://thecodeshop.github.com | http://jonforums.github.com/
twitter: @jonforums

[Boško Ivanišević]

I like it too, except I will not be able to check it this weekend but I'll try to do it during next week.

--
Regards,
Boško Ivanišević

http://pragdevnotes.com

[Jon]

I've just refactored the 4.7.1 DevKit's in this commit

https://github.com/oneclick/rubyinstaller/commit/1e4a567db4ff667957fe096824d0ee0603709c54

and have tested the updated `mingw64-32-4.7.1` DevKit (rubenvb's
mingw-w64 build) on my Win7 Ultimate 32bit system.

My results on trunk and ruby_1_9_3 http://hastebin.com/vajadawagu.tex

While some of the failures aren't concerning, a couple of the 1.9.3
failures need investigation.

I've also tested the updated mingw64-32-4.7.1 on the following projects:

ninja (C++): build OK, test OK, run OK
mercurial (Python/C): build OK, run FAIL with `abort: DLL load
failed: Invalid access to memory location.!`
mruby: build OK, test OK

Before setting 4.7.1 as the new default DevKit build toolchain

https://github.com/oneclick/rubyinstaller/blob/master/config/devkit.rb#L4

how does it look on your systems? WinXP 32bit and 64bit?

Also, one of you with Win7 64bit please add/test a rubenvb-based
`mingw64-64-4.7.1` configuration to
https://github.com/oneclick/rubyinstaller/blob/master/config/compilers/mingw64.rb
I'm guessing Luis already has this stashed away and ready to push ;)

Jon

[Luis Lavena]

On Sat, Sep 8, 2012 at 5:58 PM, Jon <jon.f...@gmail.com> wrote:
>
> I've just refactored the 4.7.1 DevKit's in this commit
>
> https://github.com/oneclick/rubyinstaller/commit/1e4a567db4ff667957fe096824d0ee0603709c54
>

Thank you.

> and have tested the updated `mingw64-32-4.7.1` DevKit (rubenvb's
> mingw-w64 build) on my Win7 Ultimate 32bit system.
>
> My results on trunk and ruby_1_9_3 http://hastebin.com/vajadawagu.tex
>

Service is down, can you use gist.github.com or pastie.org instead?

> While some of the failures aren't concerning, a couple of the 1.9.3
> failures need investigation.
>

Ruby 1.9.3 in x64 does not pass own self-tests yet:

http://ci.rubyinstaller.org/job/ruby-1_9_3-x64-build/6/console

So no wonder you encountered some failures (some of fixes haven't been
backported from trunk yet)

> I've also tested the updated mingw64-32-4.7.1 on the following projects:
>
> ninja (C++): build OK, test OK, run OK
> mercurial (Python/C): build OK, run FAIL with `abort: DLL load
> failed: Invalid access to memory location.!`

Perhaps the problem with that is bad pointers? Was mercurial *and*
python compiled entirely with mingw-w64? Did the compilation succeed
with tdm-64-4.6.1 before?

> mruby: build OK, test OK
>
> Before setting 4.7.1 as the new default DevKit build toolchain
>
> https://github.com/oneclick/rubyinstaller/blob/master/config/devkit.rb#L4
>
> how does it look on your systems? WinXP 32bit and 64bit?
>

Don't have XP 64bits, only Win7 x86 and x64

> Also, one of you with Win7 64bit please add/test a rubenvb-based
> `mingw64-64-4.7.1` configuration to
> https://github.com/oneclick/rubyinstaller/blob/master/config/compilers/mingw64.rb
> I'm guessing Luis already has this stashed away and ready to push ;)
>

Yup, I had that in a local branch, was waiting for push that to master ;-)

Will test it out locally tomorrow and push my changes.

Thank you for looking into this.

[Jon]

> > and have tested the updated `mingw64-32-4.7.1` DevKit (rubenvb's
> > mingw-w64 build) on my Win7 Ultimate 32bit system.
> >
> > My results on trunk and ruby_1_9_3 http://hastebin.com/vajadawagu.tex
> >
>
> Service is down, can you use gist.github.com or pastie.org instead?

http://pastie.org/4688083

> > I've also tested the updated mingw64-32-4.7.1 on the following projects:
> >
> > ninja (C++): build OK, test OK, run OK
> > mercurial (Python/C): build OK, run FAIL with `abort: DLL load
> > failed: Invalid access to memory location.!`
>
> Perhaps the problem with that is bad pointers? Was mercurial *and*
> python compiled entirely with mingw-w64? Did the compilation succeed
> with tdm-64-4.6.1 before?

Python is the official download and Mercurial is built with either mingw-64 or mingw. Failure only occurs with mingw-w64 builds and has failed for awhile...haven't investigated as my lazy "workaround" is to use mingw-32-4.6.2 and move on. This isn't a showstopper for us using mingw-w64 4.7.1.

[Hiroshi Shirosaki]

On Sun, Sep 9, 2012 at 10:43 AM, Jon <jon.f...@gmail.com> wrote:
>> > and have tested the updated `mingw64-32-4.7.1` DevKit (rubenvb's
>> > mingw-w64 build) on my Win7 Ultimate 32bit system.
>> >
>> > My results on trunk and ruby_1_9_3 http://hastebin.com/vajadawagu.tex
>> >
>>
>> Service is down, can you use gist.github.com or pastie.org instead?
>
> http://pastie.org/4688083
>
>

I've tested ruby_1_9_3 with x64. It has a lot of test failures, but
with my backport patch, both make test and test-all seem fine on Win7
64bit.

https://gist.github.com/3691394

--
Hiroshi Shirosaki

[Luis Lavena]

Nice!

Seems you have identified all the revisions. Want me to open back port requests for those?

Sorry for top posting. Sent from mobile.

--
You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
To post to this group, send email to rubyin...@googlegroups.com.
To unsubscribe from this group, send email to rubyinstalle...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyinstaller?hl=en.

[Hiroshi Shirosaki]

On Tue, Sep 11, 2012 at 7:42 AM, Luis Lavena <luisl...@gmail.com> wrote:
> Nice!
>
> Seems you have identified all the revisions. Want me to open back port
> requests for those?
>

Yes. Any help will be appreciated!

And I think this issue is hidden bug with x64, though I didn't have
this error on local box.
https://bugs.ruby-lang.org/issues/6990

Can you confirm this fix on CI?

--
Hiroshi Shirosaki

[Luis Lavena]

On Tue, Sep 11, 2012 at 8:12 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
> On Tue, Sep 11, 2012 at 7:42 AM, Luis Lavena <luisl...@gmail.com> wrote:
>> Nice!
>>
>> Seems you have identified all the revisions. Want me to open back port
>> requests for those?
>>
>
> Yes. Any help will be appreciated!
>

Will start opening backport requests later today.

> And I think this issue is hidden bug with x64, though I didn't have
> this error on local box.
> https://bugs.ruby-lang.org/issues/6990
>
> Can you confirm this fix on CI?

I'm running the test script you put there on CI and once I've a
failure will try a patched version.

[Hiroshi Shirosaki]

On Wed, Sep 12, 2012 at 12:19 AM, Luis Lavena <luisl...@gmail.com> wrote:
>
> Will start opening backport requests later today.
>
>> And I think this issue is hidden bug with x64, though I didn't have
>> this error on local box.
>> https://bugs.ruby-lang.org/issues/6990
>>
>> Can you confirm this fix on CI?
>
> I'm running the test script you put there on CI and once I've a
> failure will try a patched version.
>

Luis, thank you.

I tested ruby trunk on WinXP 32bit with rubenvb-4.7.1-2-release.

test-all
11285 tests, 1938140 assertions, 2 failures, 1 errors, 83 skips

Test result seems fine.

I saw strange issue that ln.exe crash when linking ruby dll.
Removing `--out-implib=libmsvcrt-ruby200.dll.a` option solved the crash.
--out-implib option with rubenvb-4.7.1-2 seems not to work on my XP box.

Test result is in this gist.
https://gist.github.com/3705031


--
Hiroshi Shirosaki

[Luis Lavena]

On Wed, Sep 12, 2012 at 6:17 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
>
> I tested ruby trunk on WinXP 32bit with rubenvb-4.7.1-2-release.
>
> test-all
> 11285 tests, 1938140 assertions, 2 failures, 1 errors, 83 skips
>
> Test result seems fine.
>
> I saw strange issue that ln.exe crash when linking ruby dll.
> Removing `--out-implib=libmsvcrt-ruby200.dll.a` option solved the crash.
> --out-implib option with rubenvb-4.7.1-2 seems not to work on my XP box.
>

Interesting that rubenvb builds fails with this but mingwbuilds not
(at least you don't mention it)

I'm testing it locally and will report back to mingw-w64 if needed.

Thank you.

[Hiroshi Shirosaki]

On Thu, Sep 13, 2012 at 5:00 AM, Luis Lavena <luisl...@gmail.com> wrote:
> On Wed, Sep 12, 2012 at 6:17 AM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
>>
>> I tested ruby trunk on WinXP 32bit with rubenvb-4.7.1-2-release.
>>
>> test-all
>> 11285 tests, 1938140 assertions, 2 failures, 1 errors, 83 skips
>>
>> Test result seems fine.
>>
>> I saw strange issue that ln.exe crash when linking ruby dll.
>> Removing `--out-implib=libmsvcrt-ruby200.dll.a` option solved the crash.
>> --out-implib option with rubenvb-4.7.1-2 seems not to work on my XP box.
>>
>
> Interesting that rubenvb builds fails with this but mingwbuilds not
> (at least you don't mention it)
>
> I'm testing it locally and will report back to mingw-w64 if needed.
>

Sorry for typo. `ld.exe` crashed. rubenvb-4.6.3 and mingwbuilds did not crash.
I don't know what is the cause so far.

--
Hiroshi Shirosaki

[Hiroshi Shirosaki]

On Wed, Sep 12, 2012 at 6:17 PM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
>
> I tested ruby trunk on WinXP 32bit with rubenvb-4.7.1-2-release.
>
> test-all
> 11285 tests, 1938140 assertions, 2 failures, 1 errors, 83 skips
>
> Test result seems fine.
>

> I saw strange issue that ld.exe crash when linking ruby dll.

> Removing `--out-implib=libmsvcrt-ruby200.dll.a` option solved the crash.
> --out-implib option with rubenvb-4.7.1-2 seems not to work on my XP box.
>
> Test result is in this gist.
> https://gist.github.com/3705031
>

I saw same ld.exe crash on another machine (XP mode on Win7).

Using rubenvb's build scripts, I've built cross compiled gcc 4.7.1.
https://github.com/rubenvb/MinGW-w64-build-scripts

My gcc build did not crash on WinXP.
https://gist.github.com/3721048

rubenvb-4.7.1-2 seems to have something weird.

I fixed remaining test failure. test_queue.rb failure was due to timeout.
The test is slow on XP. Over 20s.

--
Hiroshi Shirosaki

[Luis Lavena]

On Fri, Sep 14, 2012 at 8:42 PM, Hiroshi Shirosaki

<h.shi...@gmail.com> wrote:
> On Wed, Sep 12, 2012 at 6:17 PM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
>>
>> I tested ruby trunk on WinXP 32bit with rubenvb-4.7.1-2-release.
>>
>> test-all
>> 11285 tests, 1938140 assertions, 2 failures, 1 errors, 83 skips
>>
>> Test result seems fine.
>>
>> I saw strange issue that ld.exe crash when linking ruby dll.
>> Removing `--out-implib=libmsvcrt-ruby200.dll.a` option solved the crash.
>> --out-implib option with rubenvb-4.7.1-2 seems not to work on my XP box.
>>
>> Test result is in this gist.
>> https://gist.github.com/3705031
>>
>
> I saw same ld.exe crash on another machine (XP mode on Win7).
>
> Using rubenvb's build scripts, I've built cross compiled gcc 4.7.1.
> https://github.com/rubenvb/MinGW-w64-build-scripts
>
> My gcc build did not crash on WinXP.
> https://gist.github.com/3721048
>
> rubenvb-4.7.1-2 seems to have something weird.
>

This is interesting. I no longer have a XP around to test this out, so
will download XP mode for my Win7 and test it again.

If the crash doesn't happen on your local build using Ruben's build,
then is something we need to report back.

Downloading XP mode right now and will report back later today.

Thank you so much for not just testing this out but for testing
different combinations to discard other issues.

[Luis Lavena]

Also, is worth to mention: tdm just released 4.7.1 versions, not sure
from the announcement at mingw-w64 that 32bits compiler uses mingw or
mingw-w64 as foundation, but might be worth checking it out as we keep
investigating.

Jon, it mentions GDB 7.5 with python support built in, so it might be
your lucky day ;-)

[Luis Lavena]

On Sun, Sep 16, 2012 at 11:51 AM, Luis Lavena <luisl...@gmail.com> wrote:
>>
>> I saw same ld.exe crash on another machine (XP mode on Win7).
>>
>> Using rubenvb's build scripts, I've built cross compiled gcc 4.7.1.
>> https://github.com/rubenvb/MinGW-w64-build-scripts
>>
>> My gcc build did not crash on WinXP.
>> https://gist.github.com/3721048
>>
>> rubenvb-4.7.1-2 seems to have something weird.
>>
>
> This is interesting. I no longer have a XP around to test this out, so
> will download XP mode for my Win7 and test it again.
>
> If the crash doesn't happen on your local build using Ruben's build,
> then is something we need to report back.
>

I can confirm the crash on my XP mode too, but I seem unable to use
GDB to debug LD failure and provide a concrete backtrace to Ruben.

Will cross-post this to mingw-w64 list now.

Regards,

[Jon]

 

Also, is worth to mention: tdm just released 4.7.1 versions, not sure
from the announcement at mingw-w64 that 32bits compiler uses mingw or
mingw-w64 as foundation, but might be worth checking it out as we keep
investigating.

Jon, it mentions GDB 7.5 with python support built in, so it might be
your lucky day ;-)

While the tdm 4.7.1 ANN could have been clearer, it looks like a good build. On my Win7 32bit, I've successfully built ruby trunk, ffi, mruby, libuv, and lua so far.

So we've now got tdm-32-4.7.1 and tdm-64-4.7.1 recipes :)

  https://github.com/oneclick/rubyinstaller/commit/5b3b06a21b8547a6f08a4c897e49c1d79927e5fd

The tdm-32-4.7.1 uses mingw's internals (GDB is non-python enabled) while the tdm-64-4.7.1 uses mingw-w64 internals.  As I'll be updating to 64bit with Win8, you guys should verify that tdm-64-4.7.1 actually works ;)

Also, whoever is feeding and otherwise keeping Jenkins happy, please add a build config for these new tdm toolchains. Hopefully we'll find we can switch to tdm-32-4.7.1 and tdm-64-4.7.1 as the defaults for the upcoming 1.9.3 patch release. FWIW, I'll be using tdm-32-4.7.1 for the upcoming tcs-ruby release if all continues to look good for tdm-32-4.7.1.

Jon

[Hiroshi Shirosaki]

On Tue, Sep 18, 2012 at 2:05 AM, Jon <jon.f...@gmail.com> wrote:
>
> While the tdm 4.7.1 ANN could have been clearer, it looks like a good build.
> On my Win7 32bit, I've successfully built ruby trunk, ffi, mruby, libuv, and
> lua so far.
>
> So we've now got tdm-32-4.7.1 and tdm-64-4.7.1 recipes :)
>
>
> https://github.com/oneclick/rubyinstaller/commit/5b3b06a21b8547a6f08a4c897e49c1d79927e5fd
>

Thank you. :)
I've successfully built ruby trunk, but got one test failure.

make test-all TESTS="-q bigdecimal"

[104/123] TestBigDecimal#test_to_f = 0.00 s
1) Failure:
test_to_f(TestBigDecimal) [d:/work/ruby/test/bigdecimal/test_bigdecimal.rb:551]:

1e-322.
Exception raised:
<#<FloatDomainError: BigDecimal to Float conversion>>.


ruby -v: ruby 2.0.0dev (2012-09-16 trunk 36983) [i386-mingw32]
gcc version 4.7.1 (tdm-1)
WinXP 32bit

--
Hiroshi Shirosaki

[Luis Lavena]

On Tue, Sep 18, 2012 at 12:36 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
> On Tue, Sep 18, 2012 at 2:05 AM, Jon <jon.f...@gmail.com> wrote:
>>
>> While the tdm 4.7.1 ANN could have been clearer, it looks like a good build.
>> On my Win7 32bit, I've successfully built ruby trunk, ffi, mruby, libuv, and
>> lua so far.
>>
>> So we've now got tdm-32-4.7.1 and tdm-64-4.7.1 recipes :)
>>
>>
>> https://github.com/oneclick/rubyinstaller/commit/5b3b06a21b8547a6f08a4c897e49c1d79927e5fd
>>
>
> Thank you. :)
> I've successfully built ruby trunk, but got one test failure.
>
> make test-all TESTS="-q bigdecimal"
>
> [104/123] TestBigDecimal#test_to_f = 0.00 s
> 1) Failure:
> test_to_f(TestBigDecimal) [d:/work/ruby/test/bigdecimal/test_bigdecimal.rb:551]:
>
> 1e-322.
> Exception raised:
> <#<FloatDomainError: BigDecimal to Float conversion>>.
>

Saw you tested strtod discrepancies:

https://gist.github.com/3741901

And looks like tdm 4.7.1 might not be the best candidate for compiler default.

Re: Ruben's mingw-w64 4.7.1 issue, I've emailed mingw-w64 list but
haven't heard back, so dunno how to further report or resolve the
issue with ld.

[Jon]

> On Tue, Sep 18, 2012 at 12:36 AM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
> > On Tue, Sep 18, 2012 at 2:05 AM, Jon <jon.f...@gmail.com> wrote:
> >>
> >> While the tdm 4.7.1 ANN could have been clearer, it looks like a good build.
> >> On my Win7 32bit, I've successfully built ruby trunk, ffi, mruby, libuv, and
> >> lua so far.
> >>
> >> So we've now got tdm-32-4.7.1 and tdm-64-4.7.1 recipes :)
> >>
> >>
> >> https://github.com/oneclick/rubyinstaller/commit/5b3b06a21b8547a6f08a4c897e49c1d79927e5fd
> >>
> >
> > Thank you. :)
> > I've successfully built ruby trunk, but got one test failure.
> >
> > make test-all TESTS="-q bigdecimal"
> >
> > [104/123] TestBigDecimal#test_to_f = 0.00 s
> > 1) Failure:
> > test_to_f(TestBigDecimal) [d:/work/ruby/test/bigdecimal/test_bigdecimal.rb:551]:
> >
> > 1e-322.
> > Exception raised:
> > <#<FloatDomainError: BigDecimal to Float conversion>>.
> >
>
> Saw you tested strtod discrepancies:
>
> https://gist.github.com/3741901

Just added my results to gist for mingw-32-4.6.2 and tdm-32-4.7.1 on Win7 32bit...I can't repro.

I will rebuild trunk with tdm-32-4.7.1 and reply with my Win7 32bit results for make test-all TESTS="-q bigdecimal"

[Hiroshi Shirosaki]

On Tue, Sep 18, 2012 at 10:05 PM, Luis Lavena <luisl...@gmail.com> wrote:
> On Tue, Sep 18, 2012 at 12:36 AM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
>> On Tue, Sep 18, 2012 at 2:05 AM, Jon <jon.f...@gmail.com> wrote:
>>>
>>> While the tdm 4.7.1 ANN could have been clearer, it looks like a good build.
>>> On my Win7 32bit, I've successfully built ruby trunk, ffi, mruby, libuv, and
>>> lua so far.
>>>
>>> So we've now got tdm-32-4.7.1 and tdm-64-4.7.1 recipes :)
>>>
>>>
>>> https://github.com/oneclick/rubyinstaller/commit/5b3b06a21b8547a6f08a4c897e49c1d79927e5fd
>>>
>>
>> Thank you. :)
>> I've successfully built ruby trunk, but got one test failure.
>>
>> make test-all TESTS="-q bigdecimal"
>>
>> [104/123] TestBigDecimal#test_to_f = 0.00 s
>> 1) Failure:
>> test_to_f(TestBigDecimal) [d:/work/ruby/test/bigdecimal/test_bigdecimal.rb:551]:
>>
>> 1e-322.
>> Exception raised:
>> <#<FloatDomainError: BigDecimal to Float conversion>>.
>>
>
> Saw you tested strtod discrepancies:
>
> https://gist.github.com/3741901
>
> And looks like tdm 4.7.1 might not be the best candidate for compiler default.
>

I found strtod() result is different. This causes above failure.
With tdm gcc, float value is not returned properly on XP, but it works
fine on Win7.

http://mingw.5.n7.nabble.com/strtod-is-no-longer-C99-compliant-td15097.html

According to above thread, mingw gcc uses msvcrt implementation while
mingw-w64 gcc (rubenvb) uses mingw implementation of strtod().
stdlib.h is different between mingw and mingw-w64.

This would be XP's msvcrt strdod issue.
Possible workaround is the following path. It uses mingw implementation always.


diff --git a/include/ruby/win32.h b/include/ruby/win32.h
index 718da13..7567318 100644
--- a/include/ruby/win32.h
+++ b/include/ruby/win32.h
@@ -206,6 +206,8 @@ struct timezone;
#ifdef __MINGW32__
#undef isascii
#define isascii __isascii
+#undef strtod
+#define strtod __strtod
#endif

struct iovec {


--
Hiroshi Shirosaki

[Hiroshi Shirosaki]

> This would be XP's msvcrt strdod issue.
> Possible workaround is the following path. It uses mingw implementation always.

Sorry for typo.

This would be XP's msvcrt strtod issue.
Possible workaround is the following patch. It uses mingw implementation always.


You would not repro the bigdecimal failure on Win7.

---------------------------------------------

And I got the following failure on Win7 x64 with tdm.

gcc version 4.7.1 (tdm64-1)
ruby -v: ruby 2.0.0dev (2012-09-18 trunk 36986) [x64-mingw32]


make test-all TESTS="-q gdbm"

[27/48] TestGDBM#test_s_open_error = 0.12 s
1) Failure:
test_s_open_error(TestGDBM)
[c:/Users/hiroshi/work/ruby/test/gdbm/test_gdbm.rb:227]:
[Errno::EACCES, Errno::EWOULDBLOCK] exception expected, not
Class: <Errno::E10035>
Message: <"A non-blocking socket operation could not be completed
immediately. - C:/Users/hiroshi/Ap
pData/Local/Temp/tmptest_gdbm20120918-6724-1huzua4/tmptest_gdbm_6724">
---Backtrace---
c:/Users/hiroshi/work/ruby/test/gdbm/test_gdbm.rb:228:in `open'
c:/Users/hiroshi/work/ruby/test/gdbm/test_gdbm.rb:228:in `block in
test_s_open_error'
---------------


----------------------------------------------------------------------

About Ruben's mingw-w64 4.7.1 issue, I got backtrace of gdb. Backtrace
might not useful.

https://gist.github.com/3743443


--
Hiroshi Shirosaki

[Hiroshi Shirosaki]

On Wed, Sep 19, 2012 at 12:00 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
>
> About Ruben's mingw-w64 4.7.1 issue, I got backtrace of gdb. Backtrace
> might not useful.
>
> https://gist.github.com/3743443
>

The version of ld is different with my build. This might be related.

Ruben's build:
$ ld -V
GNU ld (rubenvb-4.7.1-2-release) 2.23.51.20120816
Supported emulations:
i386pe


My build (not crash) uses latest release (2.22) Binutils.
$ ld -V
GNU ld (rubenvb-4.7.1) 2.22
Supported emulations:
i386pe



--
Hiroshi Shirosaki

[Luis Lavena]

On Tue, Sep 18, 2012 at 12:37 PM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
> On Wed, Sep 19, 2012 at 12:00 AM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
>>
>> About Ruben's mingw-w64 4.7.1 issue, I got backtrace of gdb. Backtrace
>> might not useful.
>>
>> https://gist.github.com/3743443
>>
>
> The version of ld is different with my build. This might be related.
>
> Ruben's build:
> $ ld -V
> GNU ld (rubenvb-4.7.1-2-release) 2.23.51.20120816
>

> My build (not crash) uses latest release (2.22) Binutils.
> $ ld -V
> GNU ld (rubenvb-4.7.1) 2.22

So 2.22 works but 2.23.x doesn't

I don't have my Windows machine close to test this out, but can you
build newer binutils (same as Ruben's) do discard a build issue?

Thank you.

[Hiroshi Shirosaki]

On Wed, Sep 19, 2012 at 12:39 AM, Luis Lavena <luisl...@gmail.com> wrote:
> On Tue, Sep 18, 2012 at 12:37 PM, Hiroshi Shirosaki
> <h.shi...@gmail.com> wrote:
>> On Wed, Sep 19, 2012 at 12:00 AM, Hiroshi Shirosaki
>> <h.shi...@gmail.com> wrote:
>>>
>>> About Ruben's mingw-w64 4.7.1 issue, I got backtrace of gdb. Backtrace
>>> might not useful.
>>>
>>> https://gist.github.com/3743443
>>>
>>
>> The version of ld is different with my build. This might be related.
>>
>> Ruben's build:
>> $ ld -V
>> GNU ld (rubenvb-4.7.1-2-release) 2.23.51.20120816
>>
>> My build (not crash) uses latest release (2.22) Binutils.
>> $ ld -V
>> GNU ld (rubenvb-4.7.1) 2.22
>
> So 2.22 works but 2.23.x doesn't
>
> I don't have my Windows machine close to test this out, but can you
> build newer binutils (same as Ruben's) do discard a build issue?
>

I got ld crash using same binutils source as Ruben's. So this seems a
binutils problem with WinXP.

Result is in this gist.
https://gist.github.com/3748627

Thank you.
--
Hiroshi Shirosaki

[Luis Lavena]

On Wed, Sep 19, 2012 at 6:24 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
>
> I got ld crash using same binutils source as Ruben's. So this seems a
> binutils problem with WinXP.
>
> Result is in this gist.
> https://gist.github.com/3748627
>

Ruben's 4.7.2 release just went out, updated binutils to
2.23.51.20120920 and the crash is gone (tested with XP mode)

Perhaps we should push forward for 4.7.2 instead of 4.7.1... so I'm
updating DevKit recipes so we can further tests it.

[Jon]

On Sunday, September 23, 2012 12:31:23 PM UTC-4, Luis Lavena wrote:

On Wed, Sep 19, 2012 at 6:24 AM, Hiroshi Shirosaki
<h.shi...@gmail.com> wrote:
>
> I got ld crash using same binutils source as Ruben's. So this seems a
> binutils problem with WinXP.
>
> Result is in this gist.
> https://gist.github.com/3748627
>

Ruben's 4.7.2 release just went out, updated binutils to
2.23.51.20120920 and the crash is gone (tested with XP mode)

Perhaps we should push forward for 4.7.2 instead of 4.7.1... so I'm
updating DevKit recipes so we can further tests it.

I've updated mingw64-{32,64}-4.7.2 with Ruben's MinGW-w64 v2.0.7 headers/CRT overlay update. So far works fine for me on Win7 32bit with the projects I care about, but I'm not done testing.

https://github.com/oneclick/rubyinstaller/commit/839217fea3617947fac2d3f4e8ae61e4738ee7b3

Note that I get the same 3 test-all failures on trunk@37032 as I get with the non-overlay version:

  [ 4515/11354] TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
  [10697/11354] TestUnicodeEscape#test_basic
  [10736/11354] TestWEBrickCGI#test_cgi

When you get a moment, test it out in both 32 and 64bit flavors and try it as the internal RubyInstaller build toolchain (aka use-dkver-when-invoking-rake) and as a standalone toolchain.

Is this our shiny new default build toolchain at last? :P  (I'm hoping Ruben does another release so we don't have to use a base+overlay DevKit build recipe...we'll see)

Jon

[Luis Lavena]

On Tue, Sep 25, 2012 at 4:19 PM, Jon <jon.f...@gmail.com> wrote:
>
> I've updated mingw64-{32,64}-4.7.2 with Ruben's MinGW-w64 v2.0.7 headers/CRT
> overlay update. So far works fine for me on Win7 32bit with the projects I
> care about, but I'm not done testing.
>
> https://github.com/oneclick/rubyinstaller/commit/839217fea3617947fac2d3f4e8ae61e4738ee7b3
>

Thank you.

> Note that I get the same 3 test-all failures on trunk@37032 as I get with
> the non-overlay version:
>
> [ 4515/11354]
> TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
> [10697/11354] TestUnicodeEscape#test_basic
> [10736/11354] TestWEBrickCGI#test_cgi
>

I've tested this on CI worker and only got one:

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37034) [i386-mingw32]
11354 tests, 1939121 assertions, 1 failures, 0 errors, 82 skips

[ 4515/11354] TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
= 0.24 s

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37034) [x64-mingw32]
11333 tests, 1939068 assertions, 1 failures, 0 errors, 83 skips

[ 4515/11333] TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
= 0.18 s

> When you get a moment, test it out in both 32 and 64bit flavors and try it
> as the internal RubyInstaller build toolchain (aka
> use-dkver-when-invoking-rake) and as a standalone toolchain.
>

So far I've tested libuv, exerb-mingw and I'm rebuilding all knapsack
recipes with it (not completed yet, but looks like it works)

> Is this our shiny new default build toolchain at last? :P (I'm hoping Ruben
> does another release so we don't have to use a base+overlay DevKit build
> recipe...we'll see)
>

The only issue I see is that compiling C++ extensions since that will
require libstdc++-6.dll and libgcc_s_sjlj-1.dll be in PATH. Nothing
that "-static" doesn't solve.

Only example so far is of C++ extension is EventMachine, but it seems
to fail to compile with GCC 4.7.2, so...

Seems like we have a good candidate for new DevKit.

Thank you everybody for testing this out.

Regards,

[Dušan D. Majkić]

> mingw64-32-4.7.2

Oddly, this one works with qtbindings without error when closing app.

[Luis Lavena]

On Wed, Sep 26, 2012 at 5:40 AM, Dušan D. Majkić <dma...@gmail.com> wrote:
>> mingw64-32-4.7.2
>
> Oddly, this one works with qtbindings without error when closing app.
>

Oddly? what about mingw64-64-4.7.2?

I saw a big thread on mingw-w64 list about Qt5 using this instead of
mingw.org releases.

[Jon]

> > Note that I get the same 3 test-all failures on trunk@37032 as I get with
> > the non-overlay version:
> >
> > [ 4515/11354]
> > TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
> > [10697/11354] TestUnicodeEscape#test_basic
> > [10736/11354] TestWEBrickCGI#test_cgi
> >
>
> I've tested this on CI worker and only got one:
>
> ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37034) [i386-mingw32]
> 11354 tests, 1939121 assertions, 1 failures, 0 errors, 82 skips
>
> [ 4515/11354] TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
> = 0.24 s
>
> ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37034) [x64-mingw32]
> 11333 tests, 1939068 assertions, 1 failures, 0 errors, 83 skips
>
> [ 4515/11333] TestGemInstaller#test_generate_bin_bindir_with_user_install_warning
> = 0.18 s

Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?

[Luis Lavena]

On Wed, Sep 26, 2012 at 9:50 AM, Jon <jon.f...@gmail.com> wrote:
>
> Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?
>

C:\Users\Worker>chcp
Active code page: 1252

[Jon]

> > Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?
> >
>
> C:\Users\Worker>chcp
> Active code page: 1252

That's it. We should look see if `test/ruby/test_unicode_escape.rb` (TestUnicodeEscape#test_basic -- invalid byte sequence in UTF-8) and `test/webrick/test_cgi.rb` (TestWEBrickCGI#test_cgi -- log file data miscompare) can be updated to be agnostic to codepage settings.

That said, the following appears to work. Before I commit, do you see a bad side effect? The only side effect I've seen is that cmd.exe is left in 1252 mode after exiting sh.exe from manually running tests via `rake devkit:sh`. I don't expect this will bother the CI workers, but don't have a way to confirm.

C:\projects\rubyinstaller-git>git diff
diff --git a/recipes/devkit/devkit.rake b/recipes/devkit/devkit.rake
index a650755..e7d6f11 100644
--- a/recipes/devkit/devkit.rake
+++ b/recipes/devkit/devkit.rake
@@ -73,6 +73,7 @@ namespace(:devkit) do

task :sh => [:activate] do
sh_exe = File.join(RubyInstaller::ROOT, DevKitInstaller::MSYS.target, 'bin', 'sh.exe'
+ system 'chcp 1252'
exec sh_exe
end

[Luis Lavena]

On Wed, Sep 26, 2012 at 11:30 AM, Jon <jon.f...@gmail.com> wrote:
>> > Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?
>> >
>>
>> C:\Users\Worker>chcp
>> Active code page: 1252
>
> That's it. We should look see if `test/ruby/test_unicode_escape.rb` (TestUnicodeEscape#test_basic -- invalid byte sequence in UTF-8) and `test/webrick/test_cgi.rb` (TestWEBrickCGI#test_cgi -- log file data miscompare) can be updated to be agnostic to codepage settings.
>

Then we need to investigate and fix those too.

Can you open a bug report on bugs.ruby-lang.org describing the test
and the codepage used to produce the error? I'll try to take a look
soon.

> That said, the following appears to work. Before I commit, do you see a bad side effect? The only side effect I've seen is that cmd.exe is left in 1252 mode after exiting sh.exe from manually running tests via `rake devkit:sh`. I don't expect this will bother the CI workers, but don't have a way to confirm.
>

That might not play nice with other users with different codepage,
specially eastern ones (being 1251/1252 latin-based)

[Jon]

> >> > Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?
> >> >
> >>
> >> C:\Users\Worker>chcp
> >> Active code page: 1252
> >
> > That's it. We should look see if `test/ruby/test_unicode_escape.rb` (TestUnicodeEscape#test_basic -- invalid byte sequence in UTF-8) and `test/webrick/test_cgi.rb` (TestWEBrickCGI#test_cgi -- log file data miscompare) can be updated to be agnostic to codepage settings.
> >
>
> Then we need to investigate and fix those too.
>
> Can you open a bug report on bugs.ruby-lang.org describing the test
> and the codepage used to produce the error? I'll try to take a look
> soon.

I don't get much time these days to spend with ruby, but I'll try to find time to come up with a patch and a bug report.

> > That said, the following appears to work. Before I commit, do you see a bad side effect? The only side effect I've seen is that cmd.exe is left in 1252 mode after exiting sh.exe from manually running tests via `rake devkit:sh`. I don't expect this will bother the CI workers, but don't have a way to confirm.
> >
>
> That might not play nice with other users with different codepage,
> specially eastern ones (being 1251/1252 latin-based)

Agreed. No commit yet, but perhaps a fallback fix if the root cause (test files) can't be fixed.

[Jon]

> > >> > Hmm, wonder if it's `chcp` related? My cmd.exe is 437 when I fire up sh.exe for testing. Are the 32/64bit CI workers using 1252 or 65001?
> > >> >
> > >>
> > >> C:\Users\Worker>chcp
> > >> Active code page: 1252
> > >
> > > That's it. We should look see if `test/ruby/test_unicode_escape.rb` (TestUnicodeEscape#test_basic -- invalid byte sequence in UTF-8) and `test/webrick/test_cgi.rb` (TestWEBrickCGI#test_cgi -- log file data miscompare) can be updated to be agnostic to codepage settings.
> > >
> >
> > Then we need to investigate and fix those too.
> >
> > Can you open a bug report on bugs.ruby-lang.org describing the test
> > and the codepage used to produce the error? I'll try to take a look
> > soon.
>
> I don't get much time these days to spend with ruby, but I'll try to find time to come up with a patch and a bug report.

Forgot to mention in case anyone beats me to the punch, these are the two rogue children:

https://github.com/ruby/ruby/blob/trunk/test/ruby/test_unicode_escape.rb#L50
https://github.com/ruby/ruby/blob/trunk/test/webrick/test_cgi.rb#L40-41

[Luis Lavena]

On Wed, Sep 26, 2012 at 12:29 PM, Jon <jon.f...@gmail.com> wrote:
>
> Forgot to mention in case anyone beats me to the punch, these are the two rogue children:
>
> https://github.com/ruby/ruby/blob/trunk/test/ruby/test_unicode_escape.rb#L50
> https://github.com/ruby/ruby/blob/trunk/test/webrick/test_cgi.rb#L40-41
>

Oh dear backticks... it is shelling out to cmd.exe which is using
current codepage? I'm missing something?

[Jon]

> On Wed, Sep 26, 2012 at 12:29 PM, Jon <jon.f...@gmail.com> wrote:
> >
> > Forgot to mention in case anyone beats me to the punch, these are the two rogue children:
> >
> > https://github.com/ruby/ruby/blob/trunk/test/ruby/test_unicode_escape.rb#L50
> > https://github.com/ruby/ruby/blob/trunk/test/webrick/test_cgi.rb#L40-41
> >
>
> Oh dear backticks... it is shelling out to cmd.exe which is using
> current codepage? I'm missing something?

You didn't miss anything, I had the same reaction.

First thought was to define a `unicode_shell?` guard helper method (tries to do something similar and fails if can't) and simply skip tests that the shell can't handle.

Maybe there's some IO trickery to play with cmd.exe, but I don't a way to handle shelling out to a cmd.exe that's in the wrong codepage. Maybe a platform test combined with `system 'chcp 1252'` but I think that could get ugly real fast.

I haven't looked close enough at the other asserts (non webrick) to see why the test even needs to use the shell. Seems questionable to make the test system environment dependent without having some guards in place.

[Jon]

 

The only issue I see is that compiling C++ extensions since that will
require libstdc++-6.dll and libgcc_s_sjlj-1.dll be in PATH. Nothing
that "-static" doesn't solve.

Only example so far is of C++ extension is EventMachine, but it seems
to fail to compile with GCC 4.7.2, so...

EM compiles fine for me using mingw64-32-4.7.2 on trunk when I hack the `extconf.rb` files:

  https://gist.github.com/3789857

Haven't looked into why  `ENV['CROSS_COMPILING']` appears to be falsey (rake-compiler?) or patching with `-static` so that libstdc++-6.dll and libgcc_s_sjlj-1.dll don't need to be on PATH

[Luis Lavena]

On Wed, Sep 26, 2012 at 4:09 PM, Jon <jon.f...@gmail.com> wrote:
>
>
> EM compiles fine for me using mingw64-32-4.7.2 on trunk when I hack the
> `extconf.rb` files:
>
> https://gist.github.com/3789857
>
> Haven't looked into why `ENV['CROSS_COMPILING']` appears to be falsey
> (rake-compiler?)

No, rake-compiler do not set that env variable. That I presume is
something done by EM rake or the environment of the developers.

[Hiroshi Shirosaki]

On Thu, Sep 27, 2012 at 1:53 AM, Jon <jon.f...@gmail.com> wrote:
>> On Wed, Sep 26, 2012 at 12:29 PM, Jon <jon.f...@gmail.com> wrote:
>> >
>> > Forgot to mention in case anyone beats me to the punch, these are the two rogue children:
>> >
>> > https://github.com/ruby/ruby/blob/trunk/test/ruby/test_unicode_escape.rb#L50

echo command seems not to work fine.
Instead would using ruby work?


diff --git a/test/ruby/test_unicode_escape.rb b/test/ruby/test_unicode_escape.rb
index 088f81c..2c437d9 100644
--- a/test/ruby/test_unicode_escape.rb
+++ b/test/ruby/test_unicode_escape.rb
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-

require 'test/unit'
+require_relative 'envutil'

class TestUnicodeEscape < Test::Unit::TestCase
def test_basic
@@ -47,7 +48,7 @@ EOS
# \u in %x strings
assert_match(/^("?)A\1$/, `echo "\u0041"`) #"
assert_match(/^("?)A\1$/, %x{echo "\u0041"}) #"
- assert_match(/^("?)ü\1$/, `echo "\u{FC}"`.force_encoding("utf-8")) #"
+ assert_match(/^("?)ü\1$/, `#{EnvUtil.rubybin} -e "puts
\\"\u{FC}\\""`.force_encoding("utf-8")) #"

# \u in quoted symbols
assert_equal(:A, :"\u0041")


--
Hiroshi Shirosaki

[Jon]

Nice one Hiroshi. I didn't know about `EnvUtil`.

C:\Users\Jon\Documents\RubyDev\ruby-git>chcp
Active code page: 437

C:\Users\Jon\Documents\RubyDev\ruby-git>ruby test\ruby\test_unicode_escape.rb
Run options:

# Running tests:

Finished tests in 0.080005s, 99.9938 tests/s, 2349.8531 assertions/s.
8 tests, 188 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37036) [i386-mingw32]

C:\Users\Jon\Documents\RubyDev\ruby-git>chcp 1252
Active code page: 1252

C:\Users\Jon\Documents\RubyDev\ruby-git>ruby test\ruby\test_unicode_escape.rb
Run options:

# Running tests:

Finished tests in 0.072004s, 111.1049 tests/s, 2610.9661 assertions/s.
8 tests, 188 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37036) [i386-mingw32]

C:\Users\Jon\Documents\RubyDev\ruby-git>chcp 65001
Active code page: 65001

C:\Users\Jon\Documents\RubyDev\ruby-git>ruby test\ruby\test_unicode_escape.rb
Run options:

# Running tests:

Finished tests in 0.099006s, 80.8032 tests/s, 1898.8748 assertions/s.
8 tests, 188 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37036) [i386-mingw32]

[Luis Lavena]

On Wed, Sep 26, 2012 at 8:21 PM, Jon <jon.f...@gmail.com> wrote:
> On Thu, 27 Sep 2012 07:42:42 +0900
> Hiroshi Shirosaki <h.shi...@gmail.com> wrote:
>
>> echo command seems not to work fine.
>> Instead would using ruby work?
>>
>>
>> diff --git a/test/ruby/test_unicode_escape.rb b/test/ruby/test_unicode_escape.rb
>> index 088f81c..2c437d9 100644
>> --- a/test/ruby/test_unicode_escape.rb
>> +++ b/test/ruby/test_unicode_escape.rb

Awesome!!!

Would you mind commit this directly to ruby trunk?

[Hiroshi Shirosaki]

On Thu, Sep 27, 2012 at 8:24 AM, Luis Lavena <luisl...@gmail.com> wrote:
> On Wed, Sep 26, 2012 at 8:21 PM, Jon <jon.f...@gmail.com> wrote:
>> On Thu, 27 Sep 2012 07:42:42 +0900
>> Hiroshi Shirosaki <h.shi...@gmail.com> wrote:
>>
>>> echo command seems not to work fine.
>>> Instead would using ruby work?
>>>
>>>
>>> diff --git a/test/ruby/test_unicode_escape.rb b/test/ruby/test_unicode_escape.rb
>>> index 088f81c..2c437d9 100644
>>> --- a/test/ruby/test_unicode_escape.rb
>>> +++ b/test/ruby/test_unicode_escape.rb
>
> Awesome!!!
>
> Would you mind commit this directly to ruby trunk?

No. I'll submit it to ruby-core later to see if anyone has opinions.


--
Hiroshi Shirosaki

[Jon]

On Thu, 27 Sep 2012 09:24:53 +0900

Hiroshi Shirosaki <h.shi...@gmail.com> wrote:

> On Thu, Sep 27, 2012 at 8:24 AM, Luis Lavena <luisl...@gmail.com> wrote:
> > On Wed, Sep 26, 2012 at 8:21 PM, Jon <jon.f...@gmail.com> wrote:
> >> On Thu, 27 Sep 2012 07:42:42 +0900
> >> Hiroshi Shirosaki <h.shi...@gmail.com> wrote:
> >>
> >>> echo command seems not to work fine.
> >>> Instead would using ruby work?
> >>>
> >>>
> >>> diff --git a/test/ruby/test_unicode_escape.rb b/test/ruby/test_unicode_escape.rb
> >>> index 088f81c..2c437d9 100644
> >>> --- a/test/ruby/test_unicode_escape.rb
> >>> +++ b/test/ruby/test_unicode_escape.rb
> >
> > Awesome!!!
> >
> > Would you mind commit this directly to ruby trunk?
>
> No. I'll submit it to ruby-core later to see if anyone has opinions.

Your patch also worked successfully on my Arch system.

[jon@archee ruby-git]$ uname -a
Linux archee 3.5.4-1-ARCH #1 SMP PREEMPT Sat Sep 15 13:04:04 UTC 2012 i686 GNU/Linux

[jon@archee ruby-git]$ ruby test/ruby/test_unicode_escape.rb

Run options:

# Running tests:

Finished tests in 0.128344s, 62.3325 tests/s, 1464.8138 assertions/s.

8 tests, 188 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.0.0dev (2012-09-26 trunk 37036) [i686-linux]

[Hiroshi Shirosaki]

About above webrick test failure with chcp 437, I found ENV encoding
conversion is related.

Trunk ruby with chcp 437:

sh-3.1$ ./miniruby -e 'ENV["a"] =
"/\xA4\xDB\xA4\xB2/\xA4\xDB\xA4\xB2"; p ENV["a"]'
"/\xEF\xBD\xA4\xEF\xBE\x9B\xEF\xBD\xA4\xEF\xBD\xB2/\xEF\xBD\xA4\xEF\xBE\x9B\xEF\xBD\xA4\xEF\xBD\xB2"

Here is a patch, which removes conversion from utf-8 to file system encoding.
https://gist.github.com/3859711

Related issue:
http://bugs.ruby-lang.org/issues/5570

Jon, can you confirm this patch work with your environment?

--
Hiroshi Shirosaki

[Jon]

Thank you. The patch looks good on my Win7 Ultimate 32bit with latest trunk with 437 and 1252.

Added comment to your gist showing results and new test fail. The test fail is bogus (`Process.groups` not implemented on Windows) and I'm submitting a bug report with a simple `skip` patch after I test it.

Jon

[Hiroshi Shirosaki]

Thank you. I investigated it further and noticed the above patch would
break #5570 intension.

http://bugs.ruby-lang.org/issues/5570#note-5
Luis's example doesn't work properly since filesystem encoding is
forced upon environment variables.

If the failure occurs only with chcp 437, skipping the test when chcp
437 might be suitable?

What is your filesystem encoding which is the result of
`Encoding.find("filesystem")`?


--
Hiroshi Shirosaki

[Jon]

Seems reasonable. I will try testing without the patch using 65001 and let you know. Sometimes I see differences between 1262 and 65001 when there shouldn't be any.

> What is your filesystem encoding which is the result of
> `Encoding.find("filesystem")`?

It is Windows-1252 and seems independent of cmd.exe's codepage. I also have HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\{ACP,OEMCP} set to 1252 which may affect the results.

C:\Users\Jon\Documents>chcp
Active code page: 1252

C:\Users\Jon\Documents>ruby -ve "puts Encoding.find('filesystem')"
ruby 2.0.0dev (2012-10-09 trunk 37127) [i386-mingw32]
Windows-1252

C:\Users\Jon\Documents>chcp 437
Active code page: 437

C:\Users\Jon\Documents>ruby -ve "puts Encoding.find('filesystem')"
ruby 2.0.0dev (2012-10-09 trunk 37127) [i386-mingw32]
Windows-1252

C:\Users\Jon\Documents>chcp 65001
Active code page: 65001

C:\Users\Jon\Documents>ruby -ve "puts Encoding.find('filesystem')"
ruby 2.0.0dev (2012-10-09 trunk 37127) [i386-mingw32]
Windows-1252

Jon

[Jon]

> > >> > Forgot to mention in case anyone beats me to the punch, these are the two rogue children:
> > >> >
> > >> > https://github.com/ruby/ruby/blob/trunk/test/ruby/test_unicode_escape.rb#L50
> > >> > https://github.com/ruby/ruby/blob/trunk/test/webrick/test_cgi.rb#L40-41
> > >>
> > >> About above webrick test failure with chcp 437, I found ENV encoding
> > >> conversion is related.
> > >>
> > >> Trunk ruby with chcp 437:
> > >>
> > >> sh-3.1$ ./miniruby -e 'ENV["a"] =
> > >> "/\xA4\xDB\xA4\xB2/\xA4\xDB\xA4\xB2"; p ENV["a"]'
> > >> "/\xEF\xBD\xA4\xEF\xBE\x9B\xEF\xBD\xA4\xEF\xBD\xB2/\xEF\xBD\xA4\xEF\xBE\x9B\xEF\xBD\xA4\xEF\xBD\xB2"
> > >>
> > >> Here is a patch, which removes conversion from utf-8 to file system encoding.
> > >> https://gist.github.com/3859711
> > >>
> > >> Related issue:
> > >> http://bugs.ruby-lang.org/issues/5570
> > >>
> > >> Jon, can you confirm this patch work with your environment?
> > >
> > > Thank you. The patch looks good on my Win7 Ultimate 32bit with latest trunk with 437 and 1252.
> > >
> > > Added comment to your gist showing results and new test fail. The test fail is bogus (`Process.groups` not implemented on Windows) and I'm submitting a bug report with a simple `skip` patch after I test it.
> > >
> >
> > Thank you. I investigated it further and noticed the above patch would
> > break #5570 intension.
> >
> > http://bugs.ruby-lang.org/issues/5570#note-5
> > Luis's example doesn't work properly since filesystem encoding is
> > forced upon environment variables.
> >
> > If the failure occurs only with chcp 437, skipping the test when chcp
> > 437 might be suitable?
>
> Seems reasonable. I will try testing without the patch using 65001 and let you know. Sometimes I see differences between 1262 and 65001 when there shouldn't be any.

Added https://gist.github.com/3859711#gistcomment-585563

Unpatched trunk passes TestWEBrickCGI#test_cgi when codepage is 1252 but fails when 437 or 65001

Jon

[Hiroshi Shirosaki]

Thank you.

Filesystem encoding is CP932 at my box. And the test fails when
codepage is 1252 or 437 or 65001.
So I think the test fails If Encoding.find("locale") !=
Encoding.find("filesystem").

Patch for skip the test.
https://gist.github.com/3869593

I'll submit it to ruby-core to see if anyone has opinions.

--
Hiroshi Shirosaki