Line ends in Ruby 1.9.3+

[Ando]

Hello,

I've compiled Ruby 1.9.2 with no problems whatsoever.  I figured I'd try my hand at one of the newer releases (I've tried with CHECKOUT=1 and TRUNK=1 to get 1.9.4, and also manually edited the config/ruby_installer.rb so it would get the 1.9.3preview.  Both with 1.9.3 and 1.9.4, running the code below creates a file

a = ["hello","you","there"]

File.open("linee.txt","w") { |f|

  a.each{|x|f.puts(x)}

}

with *nix line ends (0A), while with Ruby 1.9.2 version and below it creates a file with Windows line ends (0D 0A).  I know these versions are not officially supported yet, but I was wondering if you know of a way to get around this -- or what may be the cause of the difference.

[Jon]

First, that's fantastic you're building from source and I'm glad to hear it was painless. Another way to build that's a bit more friendly is to build from a local ruby repo clone.

rake ruby19 local=c:\your\ruby\repo

For example, say you've cloned https://github.com/ruby/ruby into c:\Users\Ando\Documents\RubyDev\ruby you could run

rake ruby19 local=c:\Users\Ando\Documents\RubyDev\ruby

and the recipes will build whatever version you have checked out in your local repo. This is the way I build.

re: line endings, that's odd on the default behavior between the versions. What behavior do you want?

If you're looking for *nix-only line endings, force to binary mode like so

a = ["hello","you","there"]

File.open("linee.txt","wb") { |f|
a.each{|x|f.puts(x)}
}

On my Win7 32bit machine, here's the output (all 0x0A) from gvim's `:%!xxd`

# jruby 1.6.3 (ruby-1.8.7-p330) (2011-07-07 965162f) (Java HotSpot(TM) Client VM 1.7.0) [Windows 7-x86-java]
0000000: 6865 6c6c 6f0a 796f 750a 7468 6572 650a hello.you.there.

# ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-mingw32]
0000000: 6865 6c6c 6f0a 796f 750a 7468 6572 650a hello.you.there.

# ruby 1.9.2p290 (2011-07-09 revision 32478) [i386-mingw32]
0000000: 6865 6c6c 6f0a 796f 750a 7468 6572 650a hello.you.there.

# ruby 1.9.3dev (2011-08-04 revision 32835) [i386-mingw32]
0000000: 6865 6c6c 6f0a 796f 750a 7468 6572 650a hello.you.there.

# ruby 1.9.4dev (2011-08-05 trunk 32847) [i386-mingw32]
0000000: 6865 6c6c 6f0a 796f 750a 7468 6572 650a hello.you.there.

Jon

---
blog: http://jonforums.github.com/
twitter: @jonforums

"Anyone who can only think of one way to spell a word obviously lacks imagination." - Mark Twain

[Jon]

I can repro and I think it's a regression in 1.9.3 and 1.9.4. (NTS: wonder if line ending handling is helping 1.9.3 performance on windows)

If Luis (ruby-core committer) repros and agrees, I'd suggest posting a bug report at http://redmine.ruby-lang.org/projects/ruby-19

I'm swamped with work today, but if you've got time, start with an io.c diff and see where it leads

C:\Users\Jon\Documents\RubyDev\ruby-git>git diff origin/ruby_1_9_2 origin/ruby_1_9_3 -- io.c

[Ando]

Thanks for the tip. I managed to trace it back to around line 222. It basically comes down to >=1.9.3 using:

#define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) ((fptr)->mode & FMODE_TEXTMODE)

and <=1.9.2 using:

#define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) (!((fptr)->mode & FMODE_BINMODE))

which apparently are not equivalent. The change originates from this commit, and I can't make out if it was intentional.

The patch below restores the <=1.9.2 line end behavior for Windows for the ruby_1_9_3 branch.

diff --git a/io.c b/io.c

index ddb6d95..41bde65 100644

--- a/io.c

+++ b/io.c

@@ -219,14 +219,16 @@ rb_update_max_fd(int fd)

 #  endif

 #endif

-#define NEED_NEWLINE_DECORATOR_ON_READ(fptr) ((fptr)->mode & FMODE_TEXTMODE)

-#define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) ((fptr)->mode & FMODE_TEXTMODE)

 #if defined(RUBY_TEST_CRLF_ENVIRONMENT) || defined(_WIN32)

 /* Windows */

+#define NEED_NEWLINE_DECORATOR_ON_READ(fptr) (!((fptr)->mode & FMODE_BINMODE))

+#define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) (!((fptr)->mode & FMODE_BINMODE))

 # define DEFAULT_TEXTMODE FMODE_TEXTMODE

 # define TEXTMODE_NEWLINE_DECORATOR_ON_WRITE ECONV_CRLF_NEWLINE_DECORATOR

 #else

 /* Unix */

+#define NEED_NEWLINE_DECORATOR_ON_READ(fptr) ((fptr)->mode & FMODE_TEXTMODE)

+#define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) ((fptr)->mode & FMODE_TEXTMODE)

 # define DEFAULT_TEXTMODE 0

 #endif

 #define NEED_READCONV(fptr) ((fptr)->encs.enc2 != NULL || NEED_NEWLINE_DECORATOR_ON_READ(fptr))

[Luis Lavena]

On Fri, Aug 5, 2011 at 4:03 PM, Ando <andoem...@gmail.com> wrote:
> Thanks for the tip. I managed to trace it back to around line 222. It
> basically comes down to >=1.9.3 using:
> #define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) ((fptr)->mode &
> FMODE_TEXTMODE)
> and <=1.9.2 using:
> #define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) (!((fptr)->mode &
> FMODE_BINMODE))
> which apparently are not equivalent. The change originates from this commit,
> and I can't make out if it was intentional.
>

This clearly seems a regression, but the change in logic is baffles me.

The change was along the lines in certain Encoding changes and how
newlines were affecting parsing.

I'm checking this and will report to Ruby-Core and assign to Mr Nakada
for review.

Thank you for taking the time to investigate it and really happy that
RubyInstaller made it easy for you to build locally :-)
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[Luis Lavena]

On Fri, Aug 5, 2011 at 6:55 PM, Luis Lavena <luisl...@gmail.com> wrote:
> On Fri, Aug 5, 2011 at 4:03 PM, Ando <andoem...@gmail.com> wrote:
>> Thanks for the tip. I managed to trace it back to around line 222. It
>> basically comes down to >=1.9.3 using:
>> #define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) ((fptr)->mode &
>> FMODE_TEXTMODE)
>> and <=1.9.2 using:
>> #define NEED_NEWLINE_DECORATOR_ON_WRITE(fptr) (!((fptr)->mode &
>> FMODE_BINMODE))
>> which apparently are not equivalent. The change originates from this commit,
>> and I can't make out if it was intentional.
>>
>
> This clearly seems a regression, but the change in logic is baffles me.
>
> The change was along the lines in certain Encoding changes and how
> newlines were affecting parsing.
>
> I'm checking this and will report to Ruby-Core and assign to Mr Nakada
> for review.
>

Ando and Jon, this issue has been reported and Mr Nakada is assigned to it:

http://redmine.ruby-lang.org/issues/5164

Once again, thank you for taking the time investigating this issue.