parallel blake3, broadcast non-zero values, wait-groups integrated with chan

[Jason E. Aten]

Some of my recent work may be of general interest.

Here are three open-source contributions that have gotten new

features of late:

1) A fully parallel (multiple goroutines used) Blake3

cryptographic hash, with AVX hardware acceleration.

This is probably the fastest cryptographic hash you

will find today.

https://github.com/glycerine/blake3

2) A Go channel pattern for broadcasting any value, not just the zero value.

I've long wished that Go channels could be closed and

broadcast something other than the zero value. I finally

figured out how. This package demonstrates the approach.

https://pkg.go.dev/github.com/glycerine/loquet

3) I almost always need to know when my goroutines are done,

and to shut them all down in case of an error from one.

My idem package provides goroutine supervision trees

and also shows how to integrate sync.WaitGroup-style

task counting effectively with Go channels. Normal

sync.WaitGroup is a hazard in production code because

you cannot abort a Wait in case of error or shutdown.

https://github.com/glycerine/idem

Enjoy,

Jason

[Nagaev Boris]

On Thu, Feb 13, 2025 at 6:11 PM Jason E. Aten <j.e....@gmail.com> wrote:

> 3) I almost always need to know when my goroutines are done,
>
> and to shut them all down in case of an error from one.
>
>
> My idem package provides goroutine supervision trees
>
> and also shows how to integrate sync.WaitGroup-style
>
> task counting effectively with Go channels. Normal
>
> sync.WaitGroup is a hazard in production code because
>
> you cannot abort a Wait in case of error or shutdown.
>
>
> https://github.com/glycerine/idem

Hey Jason!

Maybe this one can be interesting for you:
https://pkg.go.dev/github.com/lightningnetwork/lnd/fn/v2#GoroutineManager
It is based on a mutex and a WaitGroup and utilizes context.Context.


--
Best regards,
Boris Nagaev

[Jason E. Aten]

Thanks Boris.  

I promised Boris I would explain the problem that the idem library solves with traditional

sync.WaitGroup usage, as found in his referenced library, if he posted his suggestion 

back here on the list instead of in a dm, so here goes:

Imagine you have a backup server that you are sending lots of files to.

Further imagine that you have alot of goroutines in running

in parallel to handle the uploads.

Now lets suppose that we want to handle an out-of-disk-space condition gracefully.

Really this could be any error where we want to shut down a worker pool

of goroutines early, and we've been using a sync.WaitGroup for to manage them.

So our goal is: we've run out of disk, and now we want to stop all the in-progress

goroutines from spewings errors all over our logs, but without panic-ing. 

Suppose we want our server to stay up and responsive and answering questions 

about its health, and telling us about its disk space problem; but maybe still

serving reads, just not writes. Anyway... you get the point, the goal is: don't panic, 

and shut down a pool of goroutines gracefully.

We can always panic and catch the panics, but that could also hide

some very different errors, or logic errors in the code itself, and so we want to, as

a general goal, avoid using panic as a catch all.

If we are using a sync.WaitGroup, and have say 40 upload jobs in progress when we

run out disk space, then we need to cancel all the jobs in progress before

they have finished.  

But, because we have used a traditional sync.WaitGroup,

we are hosed. We cannot cancel the Wait once it has started. There is 

no mechanism designed in to do that.

Now you say, well, we could artificially decrement the wait count down to zero to release

the Wait call, right? Well, that runs into the panic problem, because now you

have a race with any goroutine that is finishing right as you want to force

the wait to end by artificially decrementing the wait group count. That race

means that you are just as likely to spuriously panic when trying to abort the

wait as not.  So that is a non-starter, given the rules of the game outlined above:

no spurious panics--as they hide real problems and/or bad logic. 

So this is the problem that a idem.IdemCloseChan solves with the methods

TaskAdd, TaskDone, and TaskWait.  See https://github.com/glycerine/idem/blob/master/halter.go#L427

Specially, TaskWait takes a giveup channel (that could come from a context), 

and you can also just close the IdemCloseChan to stop waiting. 

This gives you a clean early exit from waiting on a count of tasks--and no spurious panics to suppress.

[robert engels]

This sounds incorrect to me. I think the better approach might be to signal/cancel the jobs, and as the jobs cancel themselves, they will decrement the waitGroup - releasing the waiter eventually. A shared context across the jobs in a pool seems natural.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/4623e193-bef6-46f4-bbdb-c4af00241c70n%40googlegroups.com.

[Nagaev Boris]

Hey Jason,

To implement such a mechanism, you can use GoroutineManager to manage
upload workers. Each worker receives a context.Context as an argument,
which allows for graceful shutdown. When you detect an out-of-space
condition, you simply call GoroutineManager.Stop(), which cancels the
context for all workers. This method blocks until all goroutines
finish executing, utilizing an internal sync.WaitGroup to ensure a
clean shutdown.

Within each worker, you should check the context and exit as soon as
it is canceled to avoid unnecessary errors flooding your logs.

Best,
Boris

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/4623e193-bef6-46f4-bbdb-c4af00241c70n%40googlegroups.com.