Re: [go-nuts] Racey stuff when chaining io.Writers

[Thomas Bushnell, BSG]

You can use a WriteFlusher, say, and then call Close guarded by a type assertion. This technique is pretty idiomatic.

On Thu, Mar 21, 2013 at 7:46 PM, <dbr...@datalicious.com> wrote:

I have a program that creates several csv.Writer objects and multiplexes input between them.

As soon as the input is exhausted the program can quit, so I do this:

for _, w := range writers {
  w.Flush()
}
os.Exit(0)

csv.NewWriter(io.Writer) actually initialises a bufio.Writer, so calling flush on the CSV writer just leads to a flush on the underlying bufio writer (http://golang.org/src/pkg/encoding/csv/writer.go?s=2157:2181#L85).

Now there's a need to support a flag for the writers to all produce compressed output (gzip), so I do this:

// f is a file open for writing
if Opts.Compress {
  writer = csv.NewWriter(gzip.NewWriter(f))
} else {
  writer = csv.NewWriter(f)
}

There's a problem with this: the gzip algorithm requires an explicit Close() when the stream has finished. All that bufio.Writer.Flush() does is write all the bytes to the underlying writer - if I just treat my new writer pipeline as a single io.Writer interface value that I can pass around, there's no way to make sure I close the gzip cleanly.

The only solution I can think of is to define an interface type that encapsulates it:

type WriteFlushCloser interface {
    Write([]byte) (int, error)
    Flush() error
    Close() error
}

And then I can back that with a struct that has references to both the csv.Writer (for a Flush) and the gzip.Writer (for a Close), and the Close can just be a no-op in the case where I'm not compressing the output.

But now all my functions are dealing with some funky WriteFlushCloser thing. Surely there's a cleaner way to do this?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Daniel Bryan]

Opps, replied off-list by mistake. Sorry about that.
===

But even with a type assertion, I need to call Close on an unexported name.

With w as my csv.Writer value, I'd need to do this:

w.w.w.Close()

Once I've created the csv.Writer object that happens to be backed by a gzip writer (which the rest of my code shouldn't really need to care about), the reference to the gzip writer (and the bufio.Writer in the middle) is inaccessible unless I pass around some composite type. Or are you suggesting passing that composite type around as an io.Writer, and then doing a type assertion to access the gzip writer embedded in it?

--

Daniel Bryan | Senior Technical Analyst  | Datalicious Pty Ltd

dbryan@datalicious.com | M +61466690864

Datalicious | SuperTag | Blog | Twitter | Facebook | LinkedIn

C/O Y&R | Level 15 | 35 Clarence Street | Sydney NSW 2000 

PO Box R7 | Royal Exchange NSW 1225 | Sydney

Submit any support request online or via email.

[Andy Balholm]

I would suggest creating two composite types, one for the compressed writer and one for the uncompressed. Then make the satisfy a WriteCloser interface. Make the Close method do whatever cleanup the type needs. If you embed the csv.Writer, you won't even need to reimplement the Write method.

[Daniel Bryan]

Thanks for the advice.

I've ended up doing this:

// A CSV writer supporting Close()
type csvWriteCloser interface {
    Write([]string) error
    Close() error
}
type dualwriter struct {
    csv *csv.Writer
    gz *gzip.Writer
}

func (dw *dualwriter) Write(s []string) error {
    return dw.csv.Write(s)
}

func (dw *dualwriter) Close() error {
    if dw.gz == nil {
        return nil
    }
    return dw.gz.Close()
}

And I just initialise dualwriter with nil for gz when compression is not needed.

Still a pity that all of my code changed to take a weird csvWriteCloser interface type, but I guess that's inevitable.

On Fri, Mar 22, 2013 at 2:23 PM, Andy Balholm <andyb...@gmail.com> wrote:

I would suggest creating two composite types, one for the compressed writer and one for the uncompressed. Then make the satisfy a WriteCloser interface. Make the Close method do whatever cleanup the type needs. If you embed the csv.Writer, you won't even need to reimplement the Write method.

[Zorion]

On 3/22/13, Daniel Bryan <dbr...@datalicious.com> wrote:
>
> Still a pity that all of my code changed to take a weird csvWriteCloser
> interface type, but I guess that's inevitable.
>

Hi,
I must be missing something but if you have just one type doing all you need, why should you use the interface csvWriteCloser?
Can't you just pass your type dualwriter?

I understand that the idea of using the interface is not having to write the "if" in the method Close(), but you are doing it.

Am I missing something?
Thanks!