Streams and Filters (Ticket #31921)

[Ron Blaschke]

I have looked into the t/library/streams.t failures on win32 (Ticket
#31921). They arise from the \r\n line separator on Windows.

For test #14, I've tried to plug in a \r\n -> \n Stream::Filter but didn't
find a way to peed ahead (for chunks that end in \r).

Test #18 has the same problem, but also checks the chunk size, which would
change with the \r\n -> \n substitution.

An easier fix for the tests would be to change the expected output to
include the platform specific line endings, which is what I'd like to do
for these tests, but that doesn't solve the line endings problem.

The Stream::Filter probably needs a "peek" method, to allow the filter to
peek ahead.

Any opinions?

Ron

[Leopold Toetsch]

Ron Blaschke <r...@rblasch.org> wrote:
> I have looked into the t/library/streams.t failures on win32 (Ticket
> #31921). They arise from the \r\n line separator on Windows.

Can you please elaborate a bit more on the exact problem of the test:
* is the test testing wrong things?
* does Parrot produce wrong line endings?
* where exactly is the mismatch coming from?
* can you provide a simpler example and show hexdumps of the
differences?

> Ron

leo

[Jens Rieks]

On Sunday 07 November 2004 09:48, Leopold Toetsch wrote:
> * where exactly is the mismatch coming from?

Unix uses "\n" to indicate end-of-line, windows uses "\r\n". The problem is,
that the "perlhist.txt" file is checked in as a text file. I'll recommit it
as a binary file in the hope that it fixes the problem.

The root of the problem is the different line ending, I have no idea how
parrot can deal with it, or if it should deal with it at all.

jens

[Jeff Clites]

Ah, just to amplify what you've said, it sounds like the problem is
that CVS on Windows is translating the line endings upon checkout,
since it's a text file.

It sounds like the test is fine, just a bit tricky to get the
perlhist.txt file in place unmodified. An alternative would be to
create the test file on-the-fly, but flagging it as binary for CVS is
probably simpler.

(In my opinion, the line-ending issue isn't a problem for Parrot in
terms of this test--it's legitimate to expect that you can process a
particular file, not matter what platform you happen to be on, and no
matter if the file happens to have been created on another platform.)

JEff

[Robert Spier]

> (In my opinion, the line-ending issue isn't a problem for Parrot in
> terms of this test--it's legitimate to expect that you can process a
> particular file, not matter what platform you happen to be on, and no
> matter if the file happens to have been created on another platform.)

Also, the proper way to set "binary" on a cvs file is to set -kb on
it.

-R

[Ron Blaschke]

Here are my thoughts on that topic: In my mind, "plain old text
files" are a model for text, just as PNG or JPEG are for images.

The model for the text file is a list of lines (strings). The list is
created by a delimiter (though some programs treat it more like a
terminator). Thus, for a correct parse one needs to know
- the encoding
- the character set
- the delimiter string
all of which usually default to the current platform, but may be
different.

For example, there are Windows programs that write UTF16 Unicode (see
C:\WINDOWS\wusetup.log on WinXP). In Europe usually 8bit encoding
with Codepage 1250 is used, except for the "Command Prompt" which uses
Codepage 850 (which leads to fun for example with german umlauts).

For me it boils down to the question whether parrot should support
plain old text files.

Ron