Chunking text files into Goroutines?

[mk...@yahoo.com]

Coming to Go from Python, I have been approaching problems from a Pythonic approach and try to "translate" them into Go. Working with large text files in ETL scripts, I have used this approach with the Multiprocessing module to split up a task: http://www.ngcrawford.com/2012/03/29/python-multiprocessing-large-files/ (not my blog).

Is there a module or common pattern I would in Go to accomplish the same, where a large file would be divided up into chunks and passed to multiple Goroutines for processing?

[Kyle Lemons]

Remember the rules of optimization:

1. Don't optimize.

2. (experts only) Don't optimize yet.

If you're reading a file from magnetic storage, there's a nonzero probability that reading the data from the drive is going to be slower than the entirety of the processing you need to do on the file.

That being said, however, the idea is very natural with goroutines:

http://play.golang.org/p/_9daZBbJXT

On Sat, Aug 31, 2013 at 8:23 PM, <mk...@yahoo.com> wrote:

Coming to Go from Python, I have been approaching problems from a Pythonic approach and try to "translate" them into Go. Working with large text files in ETL scripts, I have used this approach with the Multiprocessing module to split up a task: http://www.ngcrawford.com/2012/03/29/python-multiprocessing-large-files/ (not my blog).

Is there a module or common pattern I would in Go to accomplish the same, where a large file would be divided up into chunks and passed to multiple Goroutines for processing?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Konstantin Khomoutov]

On Sun, 1 Sep 2013 19:20:02 -0700
Kyle Lemons <kev...@google.com> wrote:

> Remember the rules of optimization:
> 1. Don't optimize.
> 2. (experts only) Don't optimize yet.
>
> If you're reading a file from magnetic storage, there's a nonzero
> probability that reading the data from the drive is going to be
> slower than the entirety of the processing you need to do on the file.
>
> That being said, however, the idea is very natural with goroutines:
> http://play.golang.org/p/_9daZBbJXT

[...]

I'd note that the prospective process() function missing in the example
should call wg.Done() after it completed its task -- preferably using a
deferred call placed somewhere at the beginning, like this:

func process(wg sync.WaitGroup, lines []string) {
defer wg.Done()
for _, line := range lines {
// do something with the line...
}
}

[Kyle Lemons]

I included the process function in my example :).

[Matt K]

> Remember the rules of optimization:

This is a good point. In the ETL process in question, I was hitting performance limits in the Python interpreter, and going to multiple processes resulted in a significant speed-up. However I have not yet fully tested a single process Go version, which may not have the same bottlenecks, resulting in the choke point being the storage.