Proposal to deprecate win32console

[Luis Lavena]

Hello,

As you might know, Windows lacks support for ANSI color sequences. To
workaround this limitation, several languages created their tools that
catch and transform these color sequences into Win32API console calls.

Background:

In Ruby land, win32console was a port of Win32::Console from perl [1]
by Gonzalo Garramuno

In May 2008 I took the work from Justin Bailey [2] and upgrade the
infrastructure to properly support the MinGW work for RubyInstaller
and migrated it to GitHub [3]

Projects like RSpec and Cucumber which depends on ANSI color sequence
implemented workarounds to use them.

While the project had its limitations, it serve it's purpose.

Issues:

Moving forward to Ruby 1.9, the need for accented characters generated
a lot of issues for those colored codes under Windows. Just looking at
Cucumber troubleshooting page [4] indicates how big the issue was
(and still is for some users)

A few weeks back some users reported issues related to autotest not
properly escaping the colors [5]. Investigated into the issue and
found that autotest is outputting the ANSI escape sequence byte at
byte, which confused win32console and made it ignore them.

Proposal:

While working on a win32console (crayons). Another user approached me
telling me of similar issues under JRuby and cucumber. He found
ANSICON [6] and tested with good results.

I remember playing with it back in Jan 2009 but it failed to work
under 64bits systems so discarded it.

Seems that Jason Hood not only upgraded it to work under 64bits
systems, but also made it possible inject it as permanent AutoRun
clause, so every new open console will be ANSI capable.

I've discussed with Gordon Thiesfeld (contributor of win32console),
and we both agree that performance wise [7] the usage of ANSICON looks
better than the Ruby implementation.

Also, seems it will be possible simplify the code RSpec and Cucumber
do in order to deal with ANSI coloring on Windows independently of the
Ruby implementation running.

So far during testing and small modifications to the above mentioned
libraries, it performed well and sensibly faster than win32console
(the bigger spec/features set you have, the better)

People that knows me, can tell how lazy I am... the less libraries I
need to maintain, the more hours of sleep and naps I will be able to
take.

Conclusion:

I believe above provided points are good enough to deprecate
win32console usage, send patches that introduce the changes to these
libraries requiring win32console and advertise/recommend usage of
ANSICON and use ENV['ANSICON'] to check its availability.

Thoughts? Comments? Rejections for this proposal?

[1] http://rubyforge.org/projects/win32console/
[2] http://rubyforge.org/projects/winconsole/
[3] https://github.com/luislavena/win32console/
[4] https://github.com/aslakhellesoy/cucumber/wiki/Troubleshooting
[5] https://github.com/luislavena/win32console/issues#issue/9
[6] http://adoxa.110mb.com/ansicon/index.html
[7] https://gist.github.com/660199
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[Michel Demazure]

+1 for deprecating win32console.
Just installed Ansicon on Windows (XP and 7, both 32 bits). Works perfectly.
Michel Demazure
mic...@demazure.com

[Leonardo Valeri Manera]

So long as dropping support for that ANSI colour codes on Win9x is ok
- does COMMAND.COM support showing colours at all, anyway? - I say go
ahead.

Leo

[Dr Nic Williams]

I'm all for Luis getting more naps in.

> --
> You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
> To post to this group, send email to rubyin...@googlegroups.com.
> To unsubscribe from this group, send email to rubyinstalle...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/rubyinstaller?hl=en.
>
>

--
Dr Nic Williams
Engine Yard, VP Technology
http://engineyard.com
http://drnicwilliams.com
cell +1 (415) 322-9556

[Boško Ivanišević]

I am also for depricating win32console but at the moment one thing worries me - handling UTF-8 characters. Just tried your test [7] and changed letter 'E' with some Cyrillic characters, changed active code page to 65001 (utf-8) and ansicon printed out a few garbage characters after the original sequence. That does not happen in win32console.
--
Regards,
Boško Ivanišević

[Luis Lavena]

2010/11/3 Boško Ivanišević <bosko.iv...@gmail.com>:

>
>> [7] https://gist.github.com/660199
>> --
>
> I am also for depricating win32console but at the moment one thing worries
> me - handling UTF-8 characters. Just tried your test [7] and changed letter
> 'E' with some Cyrillic characters, changed active code page to 65001 (utf-8)
> and ansicon printed out a few garbage characters after the original
> sequence. That does not happen in win32console.
> --

Did you change the codepage after installing ANSICON? (ansicon -p)

Or before?

I haven't checked the code fully but I think it is not possible to get
the current codepage dynamically and only at installation time.

win32console handles it because the ruby process is initialized after
your change and takes the current one instead. On Ruby 1.9 there is
also a lot of encoding things.

I'll take a look later today. Will try to reach Jason Hood to see if
we can get the code at GitHub so it adds more visibility to it and is
more easy to dig and review.

Cheers,

[Boško Ivanišević]

On Wed, Nov 3, 2010 at 9:06 PM, Luis Lavena <luisl...@gmail.com> wrote:

2010/11/3 Boško Ivanišević <bosko.iv...@gmail.com>:

>
>> [7] https://gist.github.com/660199
>> --
>
> I am also for depricating win32console but at the moment one thing worries
> me - handling UTF-8 characters. Just tried your test [7] and changed letter
> 'E' with some Cyrillic characters, changed active code page to 65001 (utf-8)
> and ansicon printed out a few garbage characters after the original
> sequence. That does not happen in win32console.
> --

Did you change the codepage after installing ANSICON? (ansicon -p)

Or before?

I haven't checked the code fully but I think it is not possible to get
the current codepage dynamically and only at installation time.

Yes I tried that too. Here are three scenarios I've tried. Opened console, started ansicon.exe and changed code page. In the second scenario I first changed code page and then started ansicon.exe. Finally I instarted ansicon, restarted system (all in Win 7 x86). In all three cases additional garbage was displayed after the original string.

win32console handles it because the ruby process is initialized after
your change and takes the current one instead. On Ruby 1.9 there is
also a lot of encoding things.

Yes I know this.

 

I'll take a look later today. Will try to reach Jason Hood to see if
we can get the code at GitHub so it adds more visibility to it and is
more easy to dig and review.

That would be good. That way there is a bigger chance that more people will submit patches and we'll have better version. I'll also try to dig a little bit into the code to check what's going on, although I to not have so much time lately. 

--
Regards,
Boško Ivanišević

[Octagon]

Did you also change the font for the console window?

Works perfectly for me.

On Nov 3, 11:59 pm, Boško Ivanišević <bosko.ivanise...@gmail.com>
wrote:

[Boško Ivanišević]

On Thu, Nov 4, 2010 at 2:48 PM, Octagon <g...@kaluga.ru> wrote:

Did you also change the font for the console window?

Works perfectly for me.

Yes I did.

 

--
You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
To post to this group, send email to rubyin...@googlegroups.com.
To unsubscribe from this group, send email to rubyinstalle...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyinstaller?hl=en.

--
Regards,
Boško Ivanišević

[Luis Lavena]

On Thu, Nov 4, 2010 at 11:48 AM, Octagon <g...@kaluga.ru> wrote:
> Did you also change the font for the console window?
>
> Works perfectly for me.
>

I'm using Consolas and chcp 1252.

Normally "type x.txt" that contains any type of accented characters
ends with garbage in the output, with or without ANSICON.

So, decided to force the encoding when saving the file to Windows-1252
and accents works ok. Ruby 1.8.7 also works, but not 1.9.2:

C:\Users\Luis\Projects\_sandbox>ruby foo.rb
foo.rb:1: invalid multibyte char (US-ASCII)
foo.rb:1: syntax error, unexpected $undefined, expecting $end
puts "\e[33mÉ\e[0msta es una l\e[31mí\e[0mnea de texto."
^

So added a magic comment:

# encoding: Windows-1252

And invoking worked:

C:\Users\Luis\Projects\_sandbox>ruby foo.rb
[33mÉ [0msta es una l [31mí [0mnea de texto.

Now, I enabled ANSICON and it worked (see attached)

So I think garbaged characters might be the consequence of Ruby 1.9.2
and is 'ability' to play with encodings.

Bosko, if you don't mind, I would love to see another example (attach
the files) so I can test it.

Tried to recreate your scenario and of course ended with garbage, but
think is related to incorrect encoding in my source files (Dunno how
to genera Cyrillic characters)

Thank you.

[Boško Ivanišević]

On Thu, Nov 4, 2010 at 5:57 PM, Luis Lavena <luisl...@gmail.com> wrote:

On Thu, Nov 4, 2010 at 11:48 AM, Octagon <g...@kaluga.ru> wrote:
> Did you also change the font for the console window?
>
> Works perfectly for me.
>

I'm using Consolas and chcp 1252.

My font is Lucida Console and chcp normaly set to 437.

 

Normally "type x.txt" that contains any type of accented characters
ends with garbage in the output, with or without ANSICON.

Yes it does unless you set correct code page in the console. For this purpose I've set it to 65001 (utf-8) since my file is saved in utf-8 encoding.

 

So, decided to force the encoding when saving the file to Windows-1252
and accents works ok. Ruby 1.8.7 also works, but not 1.9.2:

Haven't tried it with 1.8.7. Will do that too.

 

C:\Users\Luis\Projects\_sandbox>ruby foo.rb
foo.rb:1: invalid multibyte char (US-ASCII)
foo.rb:1: syntax error, unexpected $undefined, expecting $end
puts "\e[33mÉ\e[0msta es una l\e[31mí\e[0mnea de texto."
             ^

So added a magic comment:

# encoding: Windows-1252

And invoking worked:

C:\Users\Luis\Projects\_sandbox>ruby foo.rb
 [33mÉ [0msta es una l [31mí [0mnea de texto.

Now, I enabled ANSICON and it worked (see attached)

So I think garbaged characters might be the consequence of Ruby 1.9.2
and is 'ability' to play with encodings.

Probably you're right, but I'll check with 1.8.x version.

 

Bosko, if you don't mind, I would love to see another example (attach
the files) so I can test it.
 

Tried to recreate your scenario and of course ended with garbage, but
think is related to incorrect encoding in my source files (Dunno how
to genera Cyrillic characters)

I've attached source file with Cyrillic characters (my first and last name) and three screenshots with results of type command and ruby.

--
Regards,
Boško Ivanišević

[Boško Ivanišević]

2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>

Results with 1.8.7 are same.

 

-- 

Regards,
Boško Ivanišević

[Luis Lavena]

2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>:

>
>
> Results with 1.8.7 are same.
>

Every time I enable chcp 65001 and do a simple 'type' I get garbled results.

I needed to resave your file with Windows-1251 and set my codepage to
1251 just to be able to have 'type' work properly.

Neither Ruby, ANSICON or win32consoled worked for me. Maybe is because
I'm using a Western based Windows.

See attached all my attempts

Seems your test_cyrillic.rb contained a BOM which made type explode. I
needed to remove the BOM and by doing that I modified the whole
scenario of testing.

Brain Candler collected a list of all the issues with Ruby 1.9 encoding:

http://groups.google.com/group/comp.lang.ruby/msg/d032a35d6ca75b88
https://github.com/candlerb/string19/blob/master/string19.rb

I'm running out of ideas in encoding issues. I'm going to try Japanese
which seems have worked even with 1252 on my terminal and see what
happens.

[Boško Ivanišević]

On Fri, Nov 5, 2010 at 12:34 AM, Luis Lavena <luisl...@gmail.com> wrote:

2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>:

>
>
> Results with 1.8.7 are same.
>

Every time I enable chcp 65001 and do a simple 'type' I get garbled results.

I needed to resave your file with Windows-1251 and set my codepage to
1251 just to be able to have 'type' work properly.

Neither Ruby, ANSICON or win32consoled worked for me. Maybe is because
I'm using a Western based Windows.

That's very strange. I'm also using Western based Windows and haven't even installed Serbian language pack. Only thing I added is Serbian (Cyrillic) keyboard layout. When you open command prompt what is your initial code page? Is it 1251 or 437 as it is in my case. If it is 1251 did you do something special to set it up because I didn't do anything special to set 437 code page in command prompt. It was there immediately after installing Windows.

See attached all my attempts

Seems your test_cyrillic.rb contained a BOM which made type explode. I
needed to remove the BOM and by doing that I modified the whole
scenario of testing.

Sorry, BOM was added by mistake by the editor I used.

 

--
Regards,
Boško Ivanišević

[Boško Ivanišević]

On Fri, Nov 5, 2010 at 12:34 AM, Luis Lavena <luisl...@gmail.com> wrote:

2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>:

>
>
> Results with 1.8.7 are same.
>

Every time I enable chcp 65001 and do a simple 'type' I get garbled results.

I needed to resave your file with Windows-1251 and set my codepage to
1251 just to be able to have 'type' work properly.

Neither Ruby, ANSICON or win32consoled worked for me. Maybe is because
I'm using a Western based Windows.

 

I attached file without BOM and tested everything again but on Windows 7 x64. I attached result screenshots. BTW I think that if you save file in unicode format like 1251 with little endian and remove BOM Windows will misinterpret it completely. If you use utf-8 that doesn't happen. I tried that: converted file to 1251, removed BOM and output of 'type' was garbled.


--
Regards,
Boško Ivanišević

[Octagon]

If the ruby output is redirected to a file, the file is OK and is
displayed OK. So, this is a ruby problem. The VS build of 1.9.2
behaves the same.

However, since ANSICON works fine this does not prevent making it THE
solution to ANSI colors.

[Boško Ivanišević]

ANSICON works fine when Ruby is not involved and purpose of ANSICON is to replace win32console. Since ANSICON and Ruby do not work well together I'm not sure it does not prevent making the solution to ANSI colors. What is a benefit of getting colors with garbled output? For example all Cucumber scenarios would not be readable at all.

--
Regards,
Boško Ivanišević

[Luis Lavena]

2010/11/5 Boško Ivanišević <bosko.iv...@gmail.com>:

Actually win32console and combined with Cucumber does more than what
ANSICON does.

It reads source files as UTF-8 and translate them to the Console codepage:

https://github.com/aslakhellesoy/cucumber/blob/master/lib/cucumber/formatter/unicode.rb

It fakes some Kernel elements to make win32console work on redirected mode:

https://github.com/aslakhellesoy/cucumber/blob/master/lib/cucumber/formatter/color_io.rb

The problem that Ruby 1.9 thinks $stdout and $stderr lack encoding,
they are US-ASCII even if your codepage is 1251, 1252 or 65001
(Unicode) one.

As I pointed before, 1.9 encoding is way, way, way more complicated
than I thought and Cucumber workaround some of these issues by reading
the desired console codepage and using Iconv to transform it.

I'm not an Encoding expert, and the few I know are expert about
encoding rant all day about Ruby 1.9 inability to deal properly with
encodings.

So I'm going to take a long stab on this during the weekend and see
why we are getting mixed and inconsistent results.

In the mean time, the ones having enough time for testing, please try
with Ruby 1.8.7 and different codepages.

Thank you.

[Luis Lavena]

2010/11/5 Boško Ivanišević <bosko.iv...@gmail.com>:

>
> On Fri, Nov 5, 2010 at 12:34 AM, Luis Lavena <luisl...@gmail.com> wrote:
>>
>> 2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>:
>> >
>> >
>> > Results with 1.8.7 are same.
>> >
>>
>> Every time I enable chcp 65001 and do a simple 'type' I get garbled
>> results.
>>
>> I needed to resave your file with Windows-1251 and set my codepage to
>> 1251 just to be able to have 'type' work properly.
>>
>> Neither Ruby, ANSICON or win32consoled worked for me. Maybe is because
>> I'm using a Western based Windows.
>>
> That's very strange. I'm also using Western based Windows and haven't even
> installed Serbian language pack. Only thing I added is Serbian (Cyrillic)
> keyboard layout. When you open command prompt what is your initial code
> page? Is it 1251 or 437 as it is in my case. If it is 1251 did you do
> something special to set it up because I didn't do anything special to set
> 437 code page in command prompt. It was there immediately after installing
> Windows.

I've changed my default codepage long ago due Cucumber specifically:

https://github.com/aslakhellesoy/cucumber/wiki/Troubleshooting

But I made the registry change so is permanent

I'm going to test again and see how it goes.

[Boško Ivanišević]

2010/11/5 Luis Lavena <luisl...@gmail.com>

2010/11/5 Boško Ivanišević <bosko.iv...@gmail.com>:

>
> On Fri, Nov 5, 2010 at 12:34 AM, Luis Lavena <luisl...@gmail.com> wrote:
>>
>> 2010/11/4 Boško Ivanišević <bosko.iv...@gmail.com>:
>> >
>> >
>> > Results with 1.8.7 are same.
>> >
>>
>> Every time I enable chcp 65001 and do a simple 'type' I get garbled
>> results.
>>
>> I needed to resave your file with Windows-1251 and set my codepage to
>> 1251 just to be able to have 'type' work properly.
>>
>> Neither Ruby, ANSICON or win32consoled worked for me. Maybe is because
>> I'm using a Western based Windows.
>>
> That's very strange. I'm also using Western based Windows and haven't even
> installed Serbian language pack. Only thing I added is Serbian (Cyrillic)
> keyboard layout. When you open command prompt what is your initial code
> page? Is it 1251 or 437 as it is in my case. If it is 1251 did you do
> something special to set it up because I didn't do anything special to set
> 437 code page in command prompt. It was there immediately after installing
> Windows.

I've changed my default codepage long ago due Cucumber specifically:

https://github.com/aslakhellesoy/cucumber/wiki/Troubleshooting

But I made the registry change so is permanent

I'm going to test again and see how it goes.

I thought you've made changes because you have 1252 by default set in cmd.exe. I will also try to investigate a little bit further. Hopefully we'll find solution ;-)

--
Regards,
Boško Ivanišević

[Octagon]

On Nov 5, 1:17 pm, Boško Ivanišević <bosko.ivanise...@gmail.com>
wrote:

What does Ruby work fine with? The upstream Ruby compiled with VS
does not print out international symbols to Windows 7 console. The
same happens
to the DevKit compiled Ruby. It is an upstream problem.

Strangely enough, when the code page is set to 65000, the interpreter
even
cannot run. Thus, Ruby is trying to treat console specially on a very
basic level:

<internal:enc/prelude>:3:in `require': incompatible character
encodings: UTF-7 a
nd US-ASCII (Encoding::CompatibilityError)
from <internal:enc/prelude>:3:in `block in <compiled>'
from <internal:enc/prelude>:1:in `each'
from <internal:enc/prelude>:1:in `<compiled>'

BTW, the test file is

# encoding: utf-8
puts "рролролрол"

ANSICON works well with color and international symbols when they
come
from normal programs.

ANSICON is still not ideal. I tried IronRuby 1.1.1 (the latest) and,
putting aside
slow starts and 4 times slower execution of my favorite test, it works
fine with
international characters both in ANSICON and Windows Console.
However,
there are no colors in ANSICON.

So, we have 2 upstream problems, but ANSICON still looks like the way
to go.

[Boško Ivanišević]

On Sat, Nov 6, 2010 at 11:20 AM, Octagon <g...@kaluga.ru> wrote:

On Nov 5, 1:17 pm, Boško Ivanišević <bosko.ivanise...@gmail.com>
wrote:

> On Fri, Nov 5, 2010 at 10:00 AM, Octagon <g...@kaluga.ru> wrote:
> ANSICON works fine when Ruby is not involved and purpose of ANSICON is to
> replace win32console. Since ANSICON and Ruby do not work well together I'm
> not sure it does not prevent making the solution to ANSI colors. What is a
> benefit of getting colors with garbled output? For example all Cucumber
> scenarios would not be readable at all.
>

What does Ruby work fine with? The upstream Ruby compiled with VS
does not print out international symbols to Windows 7 console. The
same happens
to the DevKit compiled Ruby. It is an upstream problem.

I'm not sure what problems you have but MinGW compiled ruby works fine on Win 7, both x86 and x64, at least on two Windows I have access to. International characters are printed correctly.

 

Strangely enough, when the code page is set to 65000, the interpreter
even
cannot run. Thus, Ruby is trying to treat console specially on a very
basic level:

<internal:enc/prelude>:3:in `require': incompatible character
encodings: UTF-7 a
nd US-ASCII (Encoding::CompatibilityError)
       from <internal:enc/prelude>:3:in `block in <compiled>'
       from <internal:enc/prelude>:1:in `each'
       from <internal:enc/prelude>:1:in `<compiled>'

BTW, the test file is

# encoding: utf-8
puts "рролролрол"

Of course it doesn't. You shouldn't set code page to 65000 but to 65001 which is UTF8 code page. Only problem I see when 65001 code page is set is with Cucumber and problem is in its unicode.rb file where it wrongly sets Cucumber::CODEPAGE variable to cp65001 which leads to errors when, later, Iconv.iconv is called. Instead of simply taking result of 'chcp' command which works good for 1251 or 1252 code pages Cucumber should test if code page is 65001 and set Cucumber::CODEPAGE to 'UTF-8'. All other batch files (at least those in Ruby like rake, ri, gem or rspec) work as they are supposed to.

 

ANSICON works well with color and international symbols when they
come
from normal programs.

Yes, but proposal is that it replaces win32console which is... Ruby. ANSICON is definitely much faster and I generally agree that in long term it is the best candidate to replace win32console but not before problem with non ASCII characters (regarding Ruby) is solved.

ANSICON is still not ideal. I tried IronRuby 1.1.1 (the latest) and,
putting aside
slow starts and 4 times slower execution of my favorite test, it works
fine with
international characters both in ANSICON and Windows Console.
However,
there are no colors in ANSICON.

So, we have 2 upstream problems, but ANSICON still looks like the way
to go.

I completely agree. I'm looking at it now and trying to find out why it prints out garbage characters in some cases.

-- 

Regards,
Boško Ivanišević

[Michel Demazure]

I do not know if it is of some use. But when I see garbage printed, I often find a BOM added to the file by Notepad… Ruby files should have no BOM.

Michel

 

De : rubyin...@googlegroups.com [mailto:rubyin...@googlegroups.com] De la part de Boško Ivaniševic
Envoyé : samedi 6 novembre 2010 12:34
À : rubyin...@googlegroups.com
Objet : Re: [RubyInstaller] Re: Proposal to deprecate win32console

--