efficient way to csv.Reader() from a chan of string

[Sherif Eldeeb]

Greetings all,

I have a channel of 'string', where each item is a single CSV log line that I want to convert to columns using "encoding/csv"

Currently, I am creating a new csv.Reader at each iteration for each item, which is much more work than it needs to be.

<snip>

    for i := range feederChan {

        r := csv.NewReader(strings.NewReader(i))

        a := r.Read()

// Do stuff with a

// ...

    }

</snip>

I would really appreciate sharing with me if there's a way to iterate through a 'chan string' using 'csv.Reader()' without the need to create a new Reader for each item.

Thanks.

[Yannic Bonenberger]

You can use a bytes.Buffer to create a single csv.Reader and write the log lines into that buffer.

The code will look roughly like this https://play.golang.org/p/gbCPwSsx5gy .

However, depending on the implementation of strings.Reader and the level of optimization the Go compiler does, your approach of creating a new csv.Reader for every iteration might actually be faster and consume less memory than my code.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sherif Eldeeb]

Even though Yannic's answer was really useful and appreciated, I ended up using io.Pipe() which looks simpler and more efficient (explained below):

<snip>

    rp, wp := io.Pipe()

    go func() {

        defer wp.Close()

        for i := range feederChan {

            fmt.Fprintln(wp, i)

        }

    }()

    r := csv.NewReader(rp)

    for { // keep reading

        a, err := r.Read()

        if err == io.EOF {

            break

        }

// do stuff with 'a'

// ...

}

</snip>

The io.Pipe() is synchronous, and should be fairly efficient: it pipes data from writer to a reader; I fed the csv.NewReader() the reader part, and created a goroutine that drains the chan writing to the writer part.

Thanks a lot.

[Bryan Mills]

You don't need the io.Pipe: it's fairly trivial to implement io.Reader in terms of a channel of strings and a strings.Reader.

You can even implement io.WriterTo efficiently that way: https://play.golang.org/p/PxIEQYUoC50

(In theory that can make the whole pipeline zero-copy, but in practice I doubt that csv.Reader is that specialized.)