How to determine when to actually use goroutines?

[Glen Huang]

Hi,

I'm still pretty new to go. Hope this question isn't too stupid.

I'm writing a restful API server, and in order to send a response, I need to query a db to get its content type and then send the actually file that lives on the file system. Now the question is, should I put db.QueryRow and os.Open each in a goroutine to make them concurrent?

And a more general question is, when using APIs from the stdlib or 3rd-party packages, how do I determine whether to wrap them in goroutines when more than one of them need to happen sequentially and the order actually doesn't matter? Should I manually time the API executions to make the call? Should I writing things sequentially by default and only when hitting performance problems do I profile the program and wrap calls in goroutine?

How do you decide when to use goroutines?

[nat...@honeycomb.io]

> I need to query a db to get its content type and then send the actually file that lives on the file system. Now the question is, should I put db.QueryRow and os.Open each in a goroutine to make them concurrent?

Why make them concurrent if you need to know to know the end result of the DB call before opening the file?

> Should I writing things sequentially by default and only when hitting performance problems do I profile the program and wrap calls in goroutine?

Personally I would say yes, only use goroutines when you know for a fact that you have an issue which needs solving via concurrency, e.g., IO-bound workloads. While it's nice to have goroutines and channels within easy reach, they do complicate program structure and can cause data races if not used properly. Unless your app is problematically slow, design everything to be as simple as possible. People tend to assume things like "DB queries are slow" or "filesystem access is slow", but that doesn't mean doing either of those things will cause issues, and even if they do, solutions such as adding indexes to problematic fields or doing caching might help A LOT more than trying to be more concurrent will.

[Dave Cheney]

This presentation, or more accurately, the summary at the end, may be of interest to you.

https://dave.cheney.net/paste/concurrency-made-easy.pdf

A recording of this presentation at GopherCon Signapore is also available by searching for those keywords.

[Glen Huang]

> Why make them concurrent if you need to know to know the end result of the DB call before opening the file?

I actually don't need to know the result of the DB call before opening the file, only before sending the file as a response.

> only use goroutines when you know for a fact that you have an issue which needs solving via concurrency

Thanks, that's a good advice. So the question now comes to how to easily benchmark the written code. I haven't done anything like that, but from docs it seems I should use the benchmark functions from the testing package?

[Glen Huang]

I just watched the presentation, it's a good one. Thanks for sharing it.

The summary seems more towards how to write goroutines correctly, but it contains good advices and the general message is clear for me now, make things simple and do things sequentially when possible.

[Jesper Louis Andersen]

In the Erlang world, when people ask this question, we often say: "identify the truly concurrent activities in the system--make those into processes". To wit, a common mistake is to apply too much concurrency in a system where none is needed. If you run a new goroutine, then there has to be a concurrent benefit of doing so, in particular, the current goroutine should be able to do something else in the meantime.

Another common mistake is to build a pipeline of goroutines and channels in a case where you can just spawn a new goroutine for each incoming request. The former solution runs the risk of reinventing the Go schedulers on top of the Go schedulers and this is usually a losing proposition. Furthermore, each goroutine in the latter solution can track its own state, where a pipeline has to "impersonate" the data that it is currently processing. The pipeline often leads to the need of spawning more goroutines as well in order to handle things on the side, and each node in the pipeline ends up being a directly-coded node.js event loop.

You can also try to think about the system as communicating agents. Imagine a group of human beings and how they would communicate and delegate: some times it is more efficient to do a piece of work yourself rather than ask someone else to do it. Some times, it is the other way around. Concurrent programming isn't much different, though note that there is one big difference: in a concurrent system, you often have shared knowledge among all the agents. In a (truly) distributed system, you have a correspondence to epistemic logic: different agents knows different parts of the state and will have to ask for data.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[me]

On Monday, July 24, 2017 at 6:44:10 PM UTC-6, nat...@honeycomb.io wrote:

only use goroutines when you know for a fact that you have an issue which needs solving via concurrency, e.g., IO-bound workloads. While it's nice to have goroutines and channels within easy reach, they do complicate program structure and can cause data races if not used properly. Unless your app is problematically slow, design everything to be as simple as possible.

Generally I am a huge fan of this type of mind set..

However there is the contrarian position to this:
- Design all programs in such a way that they could be concurrent later, if the app needs to be scaled..

Then imagine one day that all current programs may become obsolete and you will have to write them in a Quantum way rather than a classic way, if quantum computers require massively different programming technique. For example, is there a such thing as a multi core quantum computer or are quantum computers single core, or is quantum computing several quantum micro cores? If so, how would programs look and how would one think about program design? That's a bit off topic but still interesting dreamy thoughts.