Bufio vs. Writing to Disk

[Frank Davidson]

Hi,

I have some code to write out a .csv file and then gzip it. I am using a bufio.NewWriter to link the two:

 x := gzip.NewWriter(outputFile)

 outBuf := bufio.NewWriter(x)

 w := csv.NewWriter(outBuf)

It seems slow. 

Would I be better off just writing first a .csv file to disk and then gzipping it in a separate operation and, if so, why? I would have expected the bufio.NewWriter to be faster because it keeps things in memory rather than on the disk.

Thanks!

Frank

[Tamás Gulácsi]

The bufio.NewWriter is unneeded here, as both the gzip.Writer and csv.Writer are buffered.

The concept of csv.Writer(gzip.Writer(os.File)) is ok.

Compression is cpu bound, and csv.Writer may need some optimization.

But first measure, use testing package's benchmark functionality.

[Frank Davidson]

Thanks, Tomas!

Very helpful! 

Do you know if it's important to run Flush() on the gzip writer?

Frank

[Tamás Gulácsi]

2017. január 19., csütörtök 15:10:56 UTC+1 időpontban Frank Davidson a következőt írta:

Thanks, Tomas!

Very helpful! 

Do you know if it's important to run Flush() on the gzip writer?

You must either call .Flush, or .Close at the end - and also Close the file, and check both errors to catch OS reported errors!