Why does the Wal-e kill all the greenlets if only one of them fail?

[Dikang Gu]

Hello there,

We are using Wal-e to backup to S3, and I find that if one of the backup segment failed because of timeout, all of the backup will fail. I dig into the code and find this line https://github.com/wal-e/wal-e/blob/master/wal_e/worker/pg/wal_transfer.py#L147.

So I'm wondering why do you killall the greenlets instead of letting other good ones finish? Any reason behind this?

Thanks

Dikang.

[Daniel Farina]

Thinking back, it's probably a general preference to 'join' to exit
promptly when there is trouble, and this code path has not been a
material problem for anyone who has reported bugs before.

I think it'd be fine to be more lenient about waiting for other
greenlets to finish. Try patching it.

[Dikang Gu]

Cool, are you going to change that? Or I can try to patch it as well. I already committed some changes to the instagram wal-e fork.

Thanks

Dikang.

--

Dikang

[Dikang Gu]

hmm, it seems greenlet.killall will call the joinall first.

--

Dikang

[Dikang Gu]

https://github.com/surfly/gevent/blob/master/gevent/greenlet.py#L457

--

Dikang

[Dikang Gu]

this is my pull request. https://github.com/Instagram/wal-e/pull/4

--

Dikang

[Daniel Farina]

On Fri, Oct 3, 2014 at 11:13 PM, Dikang Gu <dika...@gmail.com> wrote:
> this is my pull request. https://github.com/Instagram/wal-e/pull/4

For the sake of the record, I merged this about a week ago when releasing 0.8b2.