Race condition in concurrence reply handling?

[Rick Cobb]

We've been using ruote for a few years here, but have recently begun seeing a fairly nasty way to get stuck.

Simplifying, we use a concurrent iterator roughly like:

  citerator :tag => 'main', :on_field => 'fu', :to_f => 'bar', :merge_type => 'stack' do
    cursor :tag => 'collect-${v:ii}', :unless => '${f:bar.keep}' do

      unset 'f:rewind_to', :tag => 'c14n'
      ...
    end
  end
Enter code here...

What we're seeing (with multiple workers running) is that if we have a bunch of our entries marked with ${f:bar:keep} true, then we end up with the citerator having an "orphaned branch".  The branch expression is gone (not in the database anymore), but in the debugger, we see that the citerator still has a child for it.

I think this is caused by the ``concurrence`` (and ``citerator`` code) ignoring the return value of ``do_persist``.  That is, if two branches are "over" at the same time (i.e., replying into two different workers simultaneously), one will be able to persist the new version of the citerator exception -- and the other will fail silently.

Does this sound plausible? Or is there some exception handling I'm not seeing?

[John Mettraux]

2015-04-10 11:09 GMT+09:00 Rick Cobb <ri...@grandrounds.com>:
>
> I think this is caused by the ``concurrence`` (and ``citerator`` code)
> ignoring the return value of ``do_persist``. That is, if two branches are
> "over" at the same time (i.e., replying into two different workers
> simultaneously), one will be able to persist the new version of the
> citerator exception -- and the other will fail silently.
>
> Does this sound plausible? Or is there some exception handling I'm not
> seeing?

Hello,

yes, that sounds possible. Some persistence mechanisms are vulnerable
to such issues, they were tested, but some corner cases remain.

Best regards,

John