Goroutines and channels optimization

[Kanak Bhatia]

Does anyone have idea how to optimize consumers, producers and channels using golang. I have to upload a million objects to a cloud object storage, but unable to get a optimzed values for above parametres. Producers used to create objects and send data through channels and receive  them at consumers and call api from there.

[Jason E. Aten]

There's not enough detail here to be more prescriptive than "use the profiler". You can of course insert calls to t := time.Now() at lots of places, compute time.Since(t) after an aggregate operation, and collect statistics on the time each operation takes -- a kind of poor-person's manual profiling -- if pprof isn't giving you what you need. In my experience needing to do so is rare. If you need to get down to L1 and cache coherency optimization, perf is your friend (https://en.wikipedia.org/wiki/Perf_%28Linux%29). Start by figuring out "where the time goes"-- is it disk, network, memory, or CPU bound?

[Jason E. Aten]

If you are asking how to benchmark variations of a design, rather an

improve an existing one, this is a nice survey of tools:

https://betterstack.com/community/guides/scaling-go/golang-benchmarking/

The classic pprof intro which gives a step by step example is:

https://go.dev/blog/pprof

[ren...@ix.netcom.com]

The two most likely limiters in performance will be your network pipe to the cloud and the QPS quota offered by the service. If you are not reaching those limits you should increase the parallelism until you do. If your cpu becomes saturated first you probably need larger buffer sizes in the I/o. 

[Kanak Bhatia]

Does we have any kind of documented material or mathematical theory type stuff for these things or is it more like hit or try

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com.

[Robert Engels]

Of that I’m not sure. It is based on deep knowledge of how these things are built from the hardware to the kernel to network protocols to the service layers. 

It broadly falls under performance tuning of which there is lots of literature. 

The key element of how channels and go routines play into this - is that when dealing with synchronous requests to a service (many services offer async endpoints) you need to be able to parallelize these. Then you need to be able to provide the data as fast as the network can accept it - most networks today are faster than disk storage - even some SSD. So then the disk storage array becomes the bottleneck and you need to parallelize that (RAID etc) and the kernel access. 

Often for massive uploads you might even partition the job across machines each with a portion of the data set. 

On May 11, 2025, at 11:04 AM, Kanak Bhatia <kanakb...@gmail.com> wrote:



[Kanak Bhatia]

Thanks for the tip!

Could you suggest some resources for the same regarding these design considerations if you know of any ?

[Robert Engels]

Also check this out https://cloud.google.com/storage/docs/parallel-composite-uploads

On May 11, 2025, at 11:32 AM, Robert Engels <ren...@ix.netcom.com> wrote:



This seems to be a decent meta reference https://www.linkedin.com/advice/0/what-key-skills-tools-cloud-performance-tuning?utm_source=share&utm_medium=member_ios&utm_campaign=share_via#:~:text=Cloud%20performance%20tuning%20is%20the,%2C%20errors%2C%20and%20resource%20wastage.

But this is probably more focused https://cloud.google.com/blog/topics/developers-practitioners/optimize-cloud-storage-upload-performance-client-libraries

On May 11, 2025, at 11:24 AM, Kanak Bhatia <kanakb...@gmail.com> wrote:



[Robert Engels]

Lastly this seems to be a decent read on using gsutil- the concepts are the same for all services. https://jbrojbrojbro.medium.com/moving-mountains-large-file-upload-performance-d98e066b8401

[Robert Engels]

[Kanak Bhatia]

Thanks, I'll check it out.