crash-proof buffered channel?

[Jerry Worthey]

I have some applications that have buffered channels of data.  I'd like to make it so that if the application restarts for whatever reason that any data that is the buffered channels is not lost.

If it's a clean restart, it's fairly straightforward to drain a channel and serialize the contents.

But does anyone have any strategies for not losing the data in the event the process gets killed when there is no chance to drain the channel?  Like a server restarting or the application crashing.

Thanks.

[Matt Kane's Brain]

Instead of one channel, use two channels and a goroutine. When the
goroutine reads from one channel, write that data to a file. After the
goroutine writes that data to the output channel, delete the file.

> --
>
>



--
matt kane's brain
http://hydrogenproject.com

[Jerry Worthey]

I'm not quite sure how your solution solves the problem.  

If the input channel is buffered and the output channel is buffered, the goroutine reads an object off the input channel, writes it to a file, then puts it in the output channel, then deletes the file.  But since the output channel is buffered, you don't know that the application has actually used the object yet and thus don't know that deleting it is ok.  For example, if the output channel ends up having 500 elements in it and the application crashes, then those 500 elements will be lost.

If the output channel is not buffered, then once the goroutine can put it on the output channel, you know something has read it, so the file can be deleted. Say there are 1,000 elements in the input channel, you are only ever saving one of them as your goroutine is waiting for the output channel.  If the application crashes, everything on the input channel will be lost.  

Please correct me if I'm misunderstanding your solution...

[Matt Kane's Brain]

On Thu, Sep 27, 2012 at 4:44 PM, Jerry Worthey <jwor...@gmail.com> wrote:
> If the output channel is not buffered, then once the goroutine can put it on
> the output channel, you know something has read it, so the file can be
> deleted. Say there are 1,000 elements in the input channel, you are only
> ever saving one of them as your goroutine is waiting for the output channel.
> If the application crashes, everything on the input channel will be lost.
>
> Please correct me if I'm misunderstanding your solution...

I wasn't entirely clear since I rushed, but in this case you wouldn't
have buffered channels, but unbuffered channels. Kyle Lemons's iq
project would be a good starting point:
https://github.com/kylelemons/iq

[Kyle Lemons]

Not really... The fact remains that without designing your own journaling and restoration scheme (which will be unrelated to what's in the channel -- what if you just pulled the value out before the process gets killed?), you're not going to be able to get a particularly good solution.

--

[Patrick Crosby]

You could do something like:

ch := make(chan data, 10000)

func store(d data) { // puts d somewhere (db, redis, file) }
func remove(d data) { // removes d from wherever you stored it }
func restore() { // gets all the stored data and puts it in ch (but
doesn't remove it) }

func put(d data) {
store(d)
ch <- d
}

func loop() {
for {
x := <-ch
remove(x)
// do something with x
}
}

func main() {
go loop()
restore()

put(d1)
put(d2)
put(d3)
}

If you use ch in multiple places, you have to be careful and make sure
you always call store and remove, but it should work.

> --
>
>



--
http://www.stathat.com
Invent stats on the fly. Track data instantly, up to the minute, accurately.