Recomputing nodes

[Eric Springer]

Does scoobi offer a guarantee that a Dlist is to only be computed once?


As I've just spent the better part of a day tracking down a pretty
serious bug in my code, and the whole cause is I was generated random
numbers in part of my dag, and using the result of that. However,
different parts of my job were seeing a different set of random
numbers (as they are recomputing that node). Ouch. Do you want me to
try make a minimal case to reproduce this? Or is this a design
decision in scoobi, that nodes should be deterministic and
recomputable?

[Ben Lever]

Hi Eric,

Pre-0.7, the execution engine would "sometimes" compute "some" of the nodes multiple times. And yes, the use case of generating random values is particularly problematic here. It was really kind of bad.

As part of 0.7, EricT has re-written the execution engine (actually he's re-written a lot of stuff :)) to ensure we only execute each node in the graph once. So issue #213 that you've submitted is definitely a bug in my opinion. I'll also add that the guarantee of computing each node only once applies to all 3 "modes" - in-memory, local and cluster. If that doesn't hold it's a bug too.

Cheers!

PS. Keep the bugs coming.

PPS. And keep the bug fixes coming too :)

[Eric Springer]

Awesome. In the future, there's a fair bit of impure stuff I'm
planning on throwing inside of scoobi jobs (e.g. database lookup xD)
so this is good news. But at the moment, it's not really a problem for
me, since it's no big deal just generating my random numbers on disk,
and loading it from there.

I think there's probably about ~2 more bugs in scoobi/master (another
tagged input one, and another where it tries to read from a
non-existent file) preventing my use, so when I get the chance I'll
try minimize them down to something fixable :D

Also, I'm not sure if it's just me -- but scoobi/master seems
noticeably slower :D I can probably get you some numbers after all my
code runs on it, but based on the logs I think it's re-reading files
many many times. But again, I'm not overly worried about it, if it's
something I can ignore by throwing a few hundred more machines at


Also, I haven't tried it, or even looked into it -- but I saw some
references in the code to supports executing MSCRs in parallel? If so,
awesomeness! This will be super duperly useful, and a really huge
seller about using an abstraction layer over MapReduce :D

> --
> You received this message because you are subscribed to the Google Groups
> "scoobi-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scoobi-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>