Worker Pool vs. New Goroutine For Each Task
146 views
Skip to first unread message
Atakan Çolak
Jul 3, 2020, 10:03:02 AM7/3/20
to golang-nuts
Hiya everyone,
I have a simple processor function that takes in data, processes it, and sends it to a fellow goroutine which will eventually write them to DB. My main concern is just to maximise current task/sec with a hard limit on CPU usage. Currently I'm using a worker pool model that share a task channel to receive their tasks. I think another alternative solution could be setting GOMAXPROCS and generate a new goroutine for each task, as Go is able to handle thousands of goroutines easily. I wanted to ask about ups and down of both of these approaches.
Thank you.
I have a simple processor function that takes in data, processes it, and sends it to a fellow goroutine which will eventually write them to DB. My main concern is just to maximise current task/sec with a hard limit on CPU usage. Currently I'm using a worker pool model that share a task channel to receive their tasks. I think another alternative solution could be setting GOMAXPROCS and generate a new goroutine for each task, as Go is able to handle thousands of goroutines easily. I wanted to ask about ups and down of both of these approaches.
Thank you.
Michael Jones
Jul 3, 2020, 10:42:44 AM7/3/20
to Atakan Çolak, golang-nuts
If you change then will anything limit the peak degree of parallelism?
The worker pool size is such a limiting factor. May be important for you. If so, plan another throttling mechanism.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/567ff4bd-a99e-4af5-bd46-7859d98575cbo%40googlegroups.com.
Michael T. Jones
michae...@gmail.com
michae...@gmail.com
Andrei Tudor Călin
Jul 3, 2020, 11:29:41 AM7/3/20
to Atakan Çolak, golang-nuts
Check out Bryan's talk[0], in particular from ~27:00 onward, where worker pools are discussed. I highly recommend the entire talk.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/567ff4bd-a99e-4af5-bd46-7859d98575cbo%40googlegroups.com.
Andrei Călin
Atakan Çolak
Jul 4, 2020, 11:00:02 AM7/4/20
to golang-nuts
Thank you, perhaps two cases are quite equivalent when used with semaphores.
3 Temmuz 2020 Cuma 18:29:41 UTC+3 tarihinde Andrei Tudor Călin yazdı:
3 Temmuz 2020 Cuma 18:29:41 UTC+3 tarihinde Andrei Tudor Călin yazdı:
Check out Bryan's talk[0], in particular from ~27:00 onward, where worker pools are discussed. I highly recommend the entire talk.
On Fri, Jul 3, 2020 at 5:03 PM Atakan Çolak <atakanc...@gmail.com> wrote:
Hiya everyone,--
I have a simple processor function that takes in data, processes it, and sends it to a fellow goroutine which will eventually write them to DB. My main concern is just to maximise current task/sec with a hard limit on CPU usage. Currently I'm using a worker pool model that share a task channel to receive their tasks. I think another alternative solution could be setting GOMAXPROCS and generate a new goroutine for each task, as Go is able to handle thousands of goroutines easily. I wanted to ask about ups and down of both of these approaches.
Thank you.
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/567ff4bd-a99e-4af5-bd46-7859d98575cbo%40googlegroups.com.
--Andrei Călin
Reply all
Reply to author
Forward
0 new messages