Worker with two channels with master/slave relationship

[Jsor]

Or preferably with a better explanation than my admittedly poor title:

I have two channels that are related. The idea is that messages to one channel cause a large update, while messages to another cause smaller updates that may or may not affect the larger one. While it's not required that all waiting smaller updates are processed before the next larger updates, it's ideal. So I devised a scheme to make sure that all currently waiting smaller updates (if any exist) process before the big update happens. 

A sillier example with strings:

http://play.golang.org/p/KZkWHZz9MN

It's... ugly, I was wondering if anybody had any suggestions. In my particular case, it's important that reads from both channels happen in a single OS Thread (which means they have to be in the same goroutine locked with LockOSThread()), but I'm interested in more general solutions too, since I think a nicer solution could probably be done with two goroutines.

[Jsor]

Er, ignore the comment before FeedMaster, that was from an earlier revision where I was using two buffered channels instead of a buffered and unbuffered.

[David DENG]

Does this make you feel better?

http://play.golang.org/p/GnfwSIl6sg

David

[Jesse McNelis]

On Mon, Jul 8, 2013 at 3:20 PM, Jsor <jrago...@gmail.com> wrote:

Or preferably with a better explanation than my admittedly poor title:

I have two channels that are related. The idea is that messages to one channel cause a large update, while messages to another cause smaller updates that may or may not affect the larger one. While it's not required that all waiting smaller updates are processed before the next larger updates, it's ideal. So I devised a scheme to make sure that all currently waiting smaller updates (if any exist) process before the big update happens. 

In a concurrent system the idea of "all the waiting updates" is difficult to define.

"all waiting" requires some kind of barrier that indicates where the block of "all" starts and where is stops.

Receiving from a channel until it would block is one definition of "all", but it doesn't include the update that was sent to the channel just after you considered it empty.

Since you can't predict the ordering between the arrivals on the dominantChan and the waitingChan you don't want the ordering to effect the outcome.

If messages received on one channel are going to change the way messages received on another channel are processed then you have a race.

A sillier example with strings:

http://play.golang.org/p/KZkWHZz9MN

 

It's... ugly, I was wondering if anybody had any suggestions. In my particular case, it's important that reads from both channels happen in a single OS Thread (which means they have to be in the same goroutine locked with LockOSThread()), but I'm interested in more general solutions too, since I think a nicer solution could probably be done with two goroutines.

Is there a good reason for this requirement? Something cgo related? or is this just an attempt to fix a racey design? 

[Jsor]

It's cgo related, OpenGL is required to have all calls in one context, which is bound to one thread. Try to call anything from a different thread and it won't find the context, thus causing some pretty confusing crashes. Thus, all OpenGL instructions have to be processed in a single, thread-locked goroutine to prevent errors due to the runtime deciding to run the goroutine/function on the wrong thread. This means that both typical update-loop rendering and more general instructions (window resize, windowing system callbacks, loading textures or buffers into memory, etc) have to be processed in the same goroutine, so I have a channel of functions that the rendering system calls as it gets them, and a channel in which it waits on until it's told to update. Since the goroutine can't be exited (due to the possibility of losing the thread it was locked to in future calls), typical solutions involving simply waiting for one channel to be closed don't work (and, incidentally, is why I wait on a channel to update the renderer instead of just calling a function every time).

The instructions aren't *integral* to process before rendering, generally speaking it doesn't matter whether any given instruction happens this frame or the next frame. However, every update loop I generally try to accommodate as many instructions as possible before the update happens to ensure that the system doesn't end up falling behind and start processing these instructions later and later simply because it can only process one or two per frame otherwise. In other words, for a single instruction it doesn't matter if it's now or later, but in aggregate failing to process instructions quickly can become a problem. I *could* just drain the channel every update and be done with it, but I have that "outer" select case present because if the subsystem isn't currently updating and has something else to do it might as well be doing it.

I could use the traditional C-style multithreaded trick of just using some shared memory that the renderer checks whenever it updates to execute these instructions -- but the channel of functions method is so far much, much clearer, easier, and less of a pain to maintain than that, despite its flaws.

[Steven Blenkinsop]

http://play.golang.org/p/12MJzk6zhH

I removed the boolean flow control. There's one goto in there. My personal view is that using a goto to redo a select or switch in some cases but not others is sufficiently well structured to be one of the legit uses of goto. You could put the whole thing in a labelled for loop and then break, but I think that miscommucates the intent, since it makes looping the default rather than the exception. That's also why I put the default case first in the second select. It's the default. It's what I intend to do unless I have to do something else first.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[William Kennedy]

Here is an implementation that uses 2 go routines. A queue routine and a work routine.

http://play.golang.org/p/nhyWyxXADe

It assumes that only one routine will be processing the work. So if small work comes in, the routine will pick it up before processing any large work.

[William Kennedy]

I created a package called jobpool.

The package implements a pool of go routines that are dedicated to processing jobs posted into the pool. It maintains two queues, a normal processing queue and a priority queue. Jobs placed in the priority queue will be processed ahead of pending jobs in the normal queue.

If priority is not required, consider using my workpool package. It is faster and more efficient.

Read the following blog post for more information
http://www.goinggo.net/2013/05/thread-pooling-in-go-programming.html

[William Kennedy]