golang dos-mode output

[Tong Sun]

Hi, 

Is there a "dos mode" for golang output, I.e., the "\n" is automatically translated as "\r\n"?

How to make my "\n" code portable between Unix/Dos environment? 

Thanks

[Henrik Johansson]

I have had some troubles with mixed win/nix teams and something like this helped.

https://help.github.com/articles/dealing-with-line-endings

For other vcs than git I don't really know but there must be some way I guess.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tong Sun]

On Thu, Apr 3, 2014 at 5:44 PM, Henrik Johansson <dahan...@gmail.com> wrote:

For other vcs than git I . . .

Thanks Henrik. I was actually talking about golang output.

fmt.Printf("a\nb\nc\n");

should have a "dos mode" for when working under Dos/Win. 

[Caleb Spare]

That line of code is very clear (although it needn't include a
semicolon and Print would do as well as Printf). It outputs exactly
what it says it does. You're asking for it to magically do something
different behind your back, depending on platform?

[minux]

No. And Go also doesn't have the distinction between text files and binary files, for good.

If you want to create files with \r\n lines, you should do it explicitly.

Note: your example should just work (3 lines of output) when running in a CMD console.

[oju...@gmail.com]

Package os has some operating system specific constants:

const (
        PathSeparator     = '/' // OS-specific path separator
        PathListSeparator = ':' // OS-specific path list separator
)

Maybe we should have a LineEnding constant. For Windows it would be "\r\n" and for Unix it would be just "\n". If the distinction is of importance to you, os.LineEnding to the rescue.

[Daniel Theophanes]

It would be my strong recommendation as a user of both Windows and Linux to just standardize on "\n".

I wouldn't consider "\r\n" to be os dependent, just application dependent. All applications I use (including visual studio) understand "\n" as a line ending.

[Andy Balholm]

On Friday, April 4, 2014 11:18:11 AM UTC-7, Daniel Theophanes wrote:

It would be my strong recommendation as a user of both Windows and Linux to just standardize on "\n".

I wouldn't consider "\r\n" to be os dependent, just application dependent. All applications I use (including visual studio) understand "\n" as a line ending.

Apparently you don't use Notepad.

But I'd rather need to do a little extra work to write files that Notepad can read than deal with a text vs. binary distinction. 

[Robert Johnstone]

I'm sorry, but this does not match my experience.  Certain tools (typically programming tools) on windows understand "\n", but many others do not.  You lose some of the advantage of sticking with text file formats on Windows if you do not match the platform's expectation for line endings.  Obviously the issue is not insurmountable, but Windows definitely has a preference for "\n\r".

[Carlos Castillo]

CRLF vs LF is mainly an application concern not an OS one, so you should be creating you output based on which applications you expect will read it. It is your problem, Go can't and definitely shouldn't arbitrarily choose for you.

Should you determine that your code needs to handle either format, and you don't want to re-write every line that deals with text, you can use the http://godoc.org/code.google.com/p/go.text/transform package to wrap your readers and writers with transformations that convert "\n" <=> "\r\n". This way only at the barrier between your application and elsewhere does the change happen, both conceptually, and in the code.

[oju...@gmail.com]

I am glad Go doesn't distinguish text from binary files. That distinction is not needed. As we know, Go works well without it.

A great number prominent members of this select group are UNIX users. If you guys allow me to, I would ask: please don't forget that 9 from 10 computer users has Windows as an OS, for the good or the bad [1].

As any Windows programmer can tell you guys, line endings in Windows are "\r\n" [2]. I agree completely that some applications can choose to handle line endings in a different way. That doesn't mean there is not a standard way, that unfortunately varies from OS. This variance can be caught in the os package, by the same motives other constants are there.

[1] http://www.macrage.com/os-x-mavericks-adoption-slows-december-3370

[2] http://support.microsoft.com/kb/170750 :

[Tong Sun]

I'd like to raise the concern of this topic again, as I'm wrapping up my program (coded and debugged under Linux) into its final destination -- for use under Windows. Currently I have to replace all my "\n" with "\r\n", which is plain ugly. Moreover, my following code broke:

    w.Write([]byte{'\n'})

and I'm on my journey to find a fix. 

All I'm saying is that, this is unnecessarily bothersome. For those who oppose standardizing on "\n", may I hazard a guess that you are not C/C++ programmers? Because for C programmers, it comes natural that the "\n" is standardized on Linux, and Windows, and Mac too, so the same C program will work nicely on Linux, Windows, and Mac. 

[rant on] But so far Go is forcing us to write *three* programs just for this trivial line-end differences is just driving me nuts. Where is the spirit of compatibility? [rant off] 

Thanks

[Manlio Perillo]

The Go standard library has no notion of text file, where the standard '\n' character is translated to the current os end of line "character".

You can implement it yourself:

type TextWriter struct {

   ...

}

func (w *TextWriter) WriteLine(line string) {

    // This is easy, just write line followed by end of line for current os

...

}

func (w *TextWriter) Write(s string) {

  // This require scanning s and allocating a new buffer

}

Regards  Manlio

[Tong Sun]

On Tue, Nov 3, 2015 at 3:32 PM, Manlio Perillo <manlio....@gmail.com> wrote:
> You can implement it yourself:

No, I was talking about, taking right from my code, :

fmt.Fprintf(w, "CE: \r\n=>\r\n\r\n")

That's just ugly.

BTW, Perl conceal the trouble of handling "\n" as well.

[Tong Sun]

On Tue, Nov 3, 2015 at 3:37 PM, Tong Sun <sunto...@gmail.com> wrote:

On Tue, Nov 3, 2015 at 3:32 PM, Manlio Perillo <manlio....@gmail.com> wrote:
> You can implement it yourself:

No, I was talking about, taking right from my code, :

    fmt.Fprintf(w, "CE: \r\n=>\r\n\r\n")

That's just ugly.

Try run this under DOS,

https://github.com/suntong/lang/blob/master/lang/Go/src/sys/OsEnv.go

and redirect output to a file, 

You'll get:

[Matt Harden]

The TextWriter that Manlio sketched would work as the "w" in your fmt.Fprintf example, except the Write method should accept a []byte and return an error. Then TextWriter would implement the io.Writer interface.

BTW, if you open your file using Wordpad rather than Notepad, it will split the lines correctly.

--

[Tong Sun]

On Tue, Nov 3, 2015 at 6:30 PM, Matt Harden wrote:

The TextWriter that Manlio sketched would work as the "w" in your fmt.Fprintf example, except the Write method should accept a []byte and return an error. Then TextWriter would implement the io.Writer interface.

True. 

My argument is that it should not be the responsible of hundreds of thousands of people coming up with hundreds of thousands of different functions to deal with this portability and compatibility issue, which should have been shield from us, as all other languages already do. Even FTP does it. 

 

BTW, if you open your file using Wordpad rather than Notepad, it will split the lines correctly.

I know, but the whole point is convenient. How much percentage of people associate .txt files with Wordpad? How much percentage of people know how to open .txt files with Wordpad, or willing to? 

Windows is a world full of dummies, if you can't fight them, then you HAVE TO join them. 

 

On Tue, Nov 3, 2015 at 1:17 PM Tong Sun wrote:

On Tue, Nov 3, 2015 at 3:37 PM, Tong Sun  wrote:

On Tue, Nov 3, 2015 at 3:32 PM, Manlio Perillo wrote:
> You can implement it yourself:

No, I was talking about, taking right from my code, :

    fmt.Fprintf(w, "CE: \r\n=>\r\n\r\n")

That's just ugly.

Try run this under DOS,

https://github.com/suntong/lang/blob/master/lang/Go/src/sys/OsEnv.go

and redirect output to a file, 

You'll get:

BTW, Perl conceal the trouble of handling "\n" as well.

[Ian Lance Taylor]

On Wed, Nov 4, 2015 at 6:59 AM, Tong Sun <sunto...@gmail.com> wrote:
>
>
>
> On Tue, Nov 3, 2015 at 6:30 PM, Matt Harden wrote:
>>
>> The TextWriter that Manlio sketched would work as the "w" in your fmt.Fprintf example, except the Write method should accept a []byte and return an error. Then TextWriter would implement the io.Writer interface.
>
>
> True.
>
> My argument is that it should not be the responsible of hundreds of thousands of people coming up with hundreds of thousands of different functions to deal with this portability and compatibility issue, which should have been shield from us, as all other languages already do. Even FTP does it.

Perhaps the io/ioutil package should provide a Reader and Writer that
convert binary files to text files. I'm not sure whether that would
be a good idea or not, but at least it would only have to be written
once.

Ian

[Carlos Castillo]

Either that, or have transforms defined in golang.org/x/text to do the translation both ways.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/VwpNO73V1CE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Manlio Perillo]

The concept of text file is quite old (Algol 68 has the concept of char and bin transput); what it the reason why it has no support in the Go standard library?

Regards  Manlio 

[Andy Balholm]

> what it the reason why it has no support in the Go standard library?

Partly it’s Go’s Unix heritage. Unix doesn’t distinguish text and binary files. Other modern operating systems don’t make nearly as much distinction between the two as older ones did, either. On Windows, I think the only difference is whether \n gets converted to \r\n. I think some older OSes had completely separate APIs for the two.

There is no clear line between text and binary files. Is XML a text format? How about EPS? What if the EPS file has a TIFF preview?

If a system automatically translates line endings, it also needs a way to turn that translation off. That adds yet more complexity.

And there’s no guarantee (especially in the age of the internet) that a text file has the standard line endings for the platform it is being processed on. Many text-like file formats have a standard for what the line endings should be, which applies no matter what platform they are processed on.

(Yes, it would be a good idea for the standard library to provide some sort of Reader that normalizes newlines, and a Writer that converts \n to \r\n, but I am glad that by default Go gives us access to the exact content of files.)

[Tong Sun]

On Wed, Nov 4, 2015 at 4:04 PM, Andy Balholm <andyb...@gmail.com> wrote:

There is no clear line between text and binary files. Is XML a text format? How about EPS? What if the EPS file has a TIFF preview?

IMHO, yes it is -- whenever '\n' is used, it means text output. Otherwise, I'd use '\x0A' instead. 

 

...

(Yes, it would be a good idea for the standard library to provide some sort of Reader that normalizes newlines, and a Writer that converts \n to \r\n, but I am glad that by default Go gives us access to the exact content of files.)

I was holding that kind of tunnel vision as well, until I need to deal with Windows users. Glad that you don't need to deal with them, but please be sympathy with people that do. 

[Andy Balholm]

> IMHO, yes it is -- whenever '\n' is used, it means text output. Otherwise, I'd use '\x0A' instead.

So your proposal is that, when compiling for Windows, \n should mean the two-byte sequence CR/LF instead of just a line feed?

[Tong Sun]

On Wed, Nov 4, 2015 at 4:24 PM, Andy Balholm <andyb...@gmail.com> wrote:

> IMHO, yes it is -- whenever '\n' is used, it means text output. Otherwise, I'd use '\x0A' instead.

So your proposal is that, when compiling for Windows, \n should mean the two-byte sequence CR/LF instead of just a line feed?

Yeah, that's basically what C, C++, Perl and many other programming languages are doing by default. 

[Andy Balholm]

That’s not what C does. C did something like that on Classic Mac OS, where it interpreted \n as CR instead of LF. But on Windows, \n is always LF internally. It is converted to and from CRLF by the I/O routines if the file is opened as a text file.

You can do the same thing in Go, if you define a function OpenTextFile that wraps the os.File in an object that translates CRLF to LF in its Read method and does the reverse in its Write method.

[Jan Mercl]

On Wed, Nov 4, 2015 at 10:26 PM Tong Sun <sunto...@gmail.com> wrote:

> Yeah, that's basically what C, C++, Perl and many other programming languages are doing by default.

Files are and always must stay an opaque stream of bytes. What C cares about is the _device_ the file is written to. If the device is a terminal (or equivalent something), the output can/should be interpreted as appropriate - into a different encoding, line endings transformation etc.

--

-j

[Manlio Perillo]

Il giorno mercoledì 4 novembre 2015 22:04:56 UTC+1, Andy Balholm ha scritto:

> what it the reason why it has no support in the Go standard library?

Partly it’s Go’s Unix heritage. Unix doesn’t distinguish text and binary files. Other modern operating systems don’t make nearly as much distinction between the two as older ones did, either. On Windows, I think the only difference is whether \n gets converted to \r\n. I think some older OSes had completely separate APIs for the two.

C is a system language and it has support for text files (but some details are implementation defined, AFAIR).

 

There is no clear line between text and binary files. Is XML a text format? How about EPS? What if the EPS file has a TIFF preview?

If you are not sure, open the file in binary mode.

 

If a system automatically translates line endings, it also needs a way to turn that translation off. That adds yet more complexity.

It *must* not translate characters automatically, but only when requested by the user.

In C, text mode is the default.

stdout is opened in text mode (I remember that some time I had to set it to binary mode in a few Python programs compatible with Windows).

With go, the default is binary (this is not precise, see below), but there should be a way to set text mode, either with os.OpenTextFile, or with a generic transformer.

The generic transformer (as in x/text) seems the best solution, since when stdout is connected to a Windows console, you also need to encode strings to the default codepage.

The problem is that Go standard io interfaces only operate on byte slices.

The io package has a WriteString function, and a stringWriter interface that is not exported.

Maybe the io package should export a StringWriter interface and some functions to create concrete types, setting the encoding/transformer and end of line "character".

The interface should have a WriteString method, but also methods to query and set the encoding/ransformer and end of line character.

And there’s no guarantee (especially in the age of the internet) that a text file has the standard line endings for the platform it is being processed on. Many text-like file formats have a standard for what the line endings should be, which applies no matter what platform they are processed on.

Python has a nice concept of universal new line.

In Python3 they added some more options:

https://docs.python.org/3.4/library/io.html#io.TextIOWrapper

As a reference, the io.TextIOBase class has an encoding attribute, but it is read only.

So you can not change the encoding and newline, but this may be useful (and probably required, see below).

The File type should implement the StringWriter interface, and Stdout and Stderr should have the correct default, platform specific, values

(but I'm afraid this is not possible, since this will change the current behavior of io.WriteString when applied to os.Stdout/os.Stderr).

> [...]

Regards  Manlio

[Manlio Perillo]

Il giorno mercoledì 4 novembre 2015 22:58:34 UTC+1, Manlio Perillo ha scritto:

The problem is that Go standard io interfaces only operate on byte slices.

The io package has a WriteString function, and a stringWriter interface that is not exported.

Maybe the io package should export a StringWriter interface and some functions to create concrete types, setting the encoding/transformer and end of line "character".

The interface should have a WriteString method, but also methods to query and set the encoding/ransformer and end of line character.

I start to understand why Go standard library does not support text files...

The core concept of io is the Writer and Reader interfaces, and they only operate on bytes.

If support for StringWriter is added, then it *must* be added to all the code that currently implements a WriteString function, and most of them have no way to know the correct encoding/newline to use.

An example is the fmt package, since it uses a []byte buffer, so it does not call WriteString on the "final" Writer, but Write is called instead.

I do not see a solution...

Regards  Manlio

[Dan Kortschak]

On Tue, 2015-11-03 at 16:17 -0500, Tong Sun wrote:
> BTW, Perl conceal the trouble of handling "\n" as well.
>

Please don't follow this. The behaviour perl causes huge confusion in
users in my field where perl is (unfortunately) commonly used, because
files will 'work' under one analytical pipeline and not under another
depending on whether perl is part of the pipeline and whether the author
of the perl shmoo has set $/ explicitly.

[Andy Balholm]

I just made a package for transforming line endings: github.com/andybalholm/crlf

[Tong Sun]

On Wed, Nov 4, 2015 at 9:57 PM, Andy Balholm wrote:

I just made a package for transforming line endings: github.com/andybalholm/crlf

Thanks, Andy. 

It does indeed deal with text files with platform-appropriate line ending conversion. Excellent!

The only unfortunate is that I can't do std io any more, if to use this package. Thanks anyway. 

[Matt Harden]

What do you mean, you can't do std io anymore? You could easily wrap os.Stdout using this package, for example:

w := crlf.NewWriter(os.Stdout)

fmt.Fprint(w, "Hello\nWorld\n")

[Volker Jebramek]

I may wrong with my answer. But Go outputs UTF-8, not ASCII. In my understanding, even when UTF8 maps ASCII, its line endings are different. 

Even with the same binary representation, the semantics are different.

[Tong Sun]

./CRLF.go:34: cannot use crlf.NewWriter(os.Stdout) (type io.Writer) as type io.WriteCloser in assignment:

        io.Writer does not implement io.WriteCloser (missing Close method)

[Tong Sun]

On Thu, Nov 5, 2015 at 1:12 PM, Volker Jebramek wrote:

I may wrong with my answer. But Go outputs UTF-8, not ASCII. In my understanding, even when UTF8 maps ASCII, its line endings are different. 

Even with the same binary representation, the semantics are different.

I don't quite understand what you meant, but try run this under DOS,

https://github.com/suntong/lang/blob/master/lang/Go/src/sys/OsEnv.go

redirect output to a file, and see what you'll get. 

Are you expecting line endings are different/normal in this case? 

[Matt Harden]

w := ioutil.NopCloser(crlf.NewWriter(os.Stdout))

[Volker Jebramek]

I mean, sublime/brackets (it looks good) understand UTF 8, notepad doesn't. An application, that can not proper read an output from Go cannot read UTF 8.

[Tong Sun]

./CRLF.go:35: cannot use crlf.NewWriter(os.Stdout) (type io.Writer) as type io.Reader in argument to ioutil.NopCloser:

        io.Writer does not implement io.Reader (missing Read method)

./CRLF.go:35: cannot use ioutil.NopCloser(crlf.NewWriter(os.Stdout)) (type io.ReadCloser) as type io.WriteCloser in assignment:

        io.ReadCloser does not implement io.WriteCloser (missing Write method)

[Tong Sun]

I'm afraid you are wrong bro -- notepad can not proper read output from Go but it definitely can read UTF8. Just tried and proved it. 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/VwpNO73V1CE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Volker Jebramek]

I think it behaves wrong. See: Unicode® Standard Annex #14 :Definitions

[Dan Kortschak]

You can create your own NopCloserWriter.