ReadCloser question

[Frank Schröder]

I've got some code which should read gzipp'ed and non-gzipp'ed files. For that I have an open method which wraps the reader in a zip reader if the filename ends with gz.

My assumptions are that A) calls to os.File.Read() are *not* buffered so I want to wrap them in a bufio.Reader() and B) that I must call Close().

I'm trying to do something like this:

func open(filename string) (io.ReadCloser, error) {
    f, err := os.Open(filename)
    if err != nil {
        return nil, err
    }

    var r io.ReadCloser
    r = bufio.NewReader(f) // does not compile

    if path.Ext(filename) != "gz" {
        return r, nil
    }

    return gzip.NewReader(r)
}

But
a) r = bufio.NewReader(f) does not work since it doesn't implement io.ReadCloser
b) Do I still need to close the file after I've wrapped it in a bufio.Reader?

Right now I have an open function which creates a struct which implements ReadCloser so that I can always call Close() (Note that this doesn't solve the "close the buffered file" problem):


type CfpFile struct {
    Filename string
    R        io.Reader
    RC       io.ReadCloser
}

func (f *CfpFile) Read(p []byte) (n int, err error) {
    if f.RC != nil {
        return f.RC.Read(p)
    }
    return f.R.Read(p)
}

func (f *CfpFile) Close() error {
    if f.RC != nil {
        return f.RC.Close()
    }
    return nil
}

func openCfpFile(filename string) (*CfpFile, error) {
    const bufSize = 4 * 1 << 20

    f, err := os.Open(filename)
    if err != nil {
        log.Printf("Error opening %s", filename)
        return nil, err
    }

    if path.Ext(filename) != "gz" {
        return &CfpFile{Filename: filename, R: bufio.NewReaderSize(f, bufSize)}, nil
    }

    rc, err := gzip.NewReader(bufio.NewReaderSize(f, bufSize))
    if err != nil {
        log.Print("gzip.NewReader: ", err)
        return nil, err
    }

    return &CfpFile{Filename: filename, RC: rc}, nil
}

Is this the right approach?

Frank

[Rui Ueyama]

You can wrap a reader with http://golang.org/pkg/io/ioutil/#NopCloser to "convert" a io.Reader to io.ReadCloser.

But I don't know if you need it. http://golang.org/pkg/compress/gzip/#NewReader says that gzip reader does buffer input, so you may not have to use bufio in the first place.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frank Schröder]

OK, that helps. What I still don't get is that when I wrap a file in a bufio.Reader and pass only the reader on how does the file get closed?

In case of the gzip.Reader the receiver can close the reader whereas in with the bufio.Reader the caller has to do it while also keeping track of the file and the reader.

What I'd like to do is to pass a bunch of readers (gzip and others) over a channel which just consume everything and then close the reader when they're done.

Thx a lot
Frank

[Rui Ueyama]

I don't know why bufio.Reader does not provide Close method, but I'd write my own io.Reader that wraps a given io.Reader with bufio.Reader and call Close on the underlying reader when bufio.Reader returns EOF.

--

[Frank Schröder]

Something like a BufferedReadCloser. Would this warrant a CL?

Frank Schröder

[Frank Schröder]

This could be as simple as this. It compiles. Will test this now.

type readCloser struct {
    io.Reader
    io.Closer

}

func open(filename string) (io.ReadCloser, error) {
    f, err := os.Open(filename)
    if err != nil {
        return nil, err
    }

    if path.Ext(filename) != "gz" {
        return &readCloser{bufio.NewReader(f), f}, nil
    }

    return gzip.NewReader(f)
}

Frank

[Rui Ueyama]

That's simpler than I thought. Nice work.

[Dan Kortschak]

You should probably use filepath here, and you should definitely use ".gz" (http://play.golang.org/p/eEdZBdKP5D).

On 15/05/2014, at 3:36 AM, "Frank Schröder" <frank.s...@gmail.com> wrote:

> path.Ext(filename) != "gz"

[Frank Schröder]

Yeah, I've stumbled over the ".gz" on the first run. Where is the difference between path.Ext() and filepath.Ext() in this case? filepath being OS agnostic?

[Tamás Gulácsi]

Or with anonymous struct:
return struct {
Io.Reader
io.Closer
}{bufio.NewBuffer(f), f}