Realtime database transaction deadlock?

166 views
Skip to first unread message

Felix Halim

unread,
Aug 13, 2018, 12:57:43 PM8/13/18
to fireba...@googlegroups.com
I have a realtime database rule on "items" indexed on the "time" attribute:

{
  "rules": {
    "items": {
      ".indexOn": "time"
    }
  }
}

I then compiled the typescript code in the attachment (deadlock.ts) using tsc version 2.9.1  (you will need to replace the <DATABASE_NAME> to use it).

I then run 4 processes (in 4 separate terminals) in this order:
- In the first two terminals, run:
  node lib/deadlock consumer
- In the third terminal, run:
  node lib/deadlock monitor
- In the fourth terminal, run:
  node lib/deadlock producer

The "producer" process should terminate pretty quickly, it is inserting 20 entries to the items database.

The "monitor" process should print one line per second and after about 1 minute, it will stuck with this pattern:

...
first = [object Object], items.length = 3
first = [object Object], items.length = 3
first = null, items.length = 3
first = null, items.length = 3
first = null, items.length = 3
[the last line above repeats indefinitely]

The non-zero item.length shows that there is still data in the items database, but querying the first item ordered by its time attribute returns null. This is definitely wrong (or at least unexpected to me).

Looking at the "consumer" output in the first two terminals shows that they are stuck inside transaction.

There are several ways to bail out from this deadlock:
- Kill the first two "consumer" terminals (killing only one of them does not fix the deadlock). In fact, restarting one terminal, then kill the other one still does not fix the deadlock! (both of them must be killed).
- Run the "producer" process again, it will somehow fix the deadlock temporarily (the "consumer" processes run again, but then they are stuck in a deadlock again).
- Remove the ".indexOn" database rule. This will spew a lot of warning messages in all terminals, but the consumers suddenly runs fine and no more deadlock. This may be the biggest hint on what causing the deadlock.

I have contacted firebase support. I'm just sharing it here in case other people encountered the same bug.

Felix Halim
deadlock.ts

Piotr Kaminski

unread,
Aug 13, 2018, 4:52:21 PM8/13/18
to fireba...@googlegroups.com
Thanks for posting -- I've seen similar issues, but I'm still on SDK 2.x and never managed to get a reliable repro.  Please follow up with the resolution when available!

    -- P.


--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAAbwPoqmJo6a%3D%3DR9xPCpiZJQsoNuZqN77qVE6H3ZG4iyupey-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

Felix Halim

unread,
Aug 16, 2018, 2:24:04 AM8/16/18
to fireba...@googlegroups.com
Hi Piotr,

The firebase support person could not reproduce the deadlock, and dismiss it as a coding error :(

I am able to reproduce it consistently.

Can you help confirming the deadlock bug from my script?


To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAAbwPoqmJo6a%3D%3DR9xPCpiZJQsoNuZqN77qVE6H3ZG4iyupey-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5zyD799SBcvfico%3Dm1Mq96uB_dn70BAJtqJ36GvOqw4BQ%40mail.gmail.com.

Samuel Stern

unread,
Aug 17, 2018, 12:51:36 PM8/17/18
to fireba...@googlegroups.com
Hi Felix,

I was one of the people who saw your support case.  I ran the script and don't see the error myself.  That doesn't mean we don't believe you or are dismissing you!  It just means we're clearly missing some part of the equation.  Without being able to reproduce the error we have no chance of finding it.

Also thank you for providing such a nice self-contained reproduction script.  It's definitely helpful.  I'm asking some of my RTDB colleagues to see what they think might be going on.

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAAbwPooRBcYH5UO%2BjhNrZKegoBp%3D3MG5SoU0rkx-sSbMqJCGqw%40mail.gmail.com.

Piotr Kaminski

unread,
Aug 18, 2018, 10:31:52 PM8/18/18
to fireba...@googlegroups.com
Hey all,

I'm able to consistently reproduce this by following Felix's instructions. [1]  The process doesn't always get stuck at the same item number so there appears to be an element of timing to it.  Have you Firebase folks tried to run this from outside the internal network just in case that makes a difference?

Anyway, this is a completely independent verification of Felix's repro so you might want to take another swing at it.  I'm happy to provide debug logs if you think that would be helpful.

Thanks,

    -- P.

[1] I did convert his TypeScript code to JS and injected credentials explicitly instead of pulling them out of the environment, but those are clearly trivial changes.


To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBqNjzkdUN16TCyZNMse95rT3oygO2pX0hTUgg3r3WRpFg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Felix Halim

unread,
Aug 19, 2018, 2:40:00 AM8/19/18
to fireba...@googlegroups.com
Thanks Piotr for trying to reproduce!

I have simplified the code even more: it only need 1 item to cause deadlock (previously I used 20 items and gradually deletes them).

I have merged the producer with the monitor, so the repro steps is now shorter and easier:

1. In the first two terminals, run:
    node lib/deadlock consumer
2. In the third terminal, run:
    node lib/deadlock monitor

Just watch the monitor process for about 2 minutes until it prints "DEADLOCK!"



To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBqNjzkdUN16TCyZNMse95rT3oygO2pX0hTUgg3r3WRpFg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5zWDusbrQCdFZzgS9LAFhutTCqEhtWk8xGfw_H0gTmRDw%40mail.gmail.com.
deadlock.ts

Piotr Kaminski

unread,
Aug 19, 2018, 3:15:35 AM8/19/18
to fireba...@googlegroups.com
Reproduced again.  Interestingly, I only need a single consumer and the monitor running to repro (though it seems to take longer), so it's not some kind of race condition between transactions.  Running two consumers without the monitor will also repro in about the same amount of time.  (But running just a single consumer doesn't seem to repro for me, i.e. the consumer doesn't get stuck.)

    -- P.


To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5zWDusbrQCdFZzgS9LAFhutTCqEhtWk8xGfw_H0gTmRDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAAbwPoq-YBZod6iH0ThcdSPaZpJ3E5ZjvkmkUUZ1giFarBXxZg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Samuel Stern

unread,
Aug 21, 2018, 12:55:55 PM8/21/18
to fireba...@googlegroups.com
Just to circle back here, the RTDB backend team took a look at this and they were able to identify a server bug.  Thanks to everyone for the evidence and reproductions!

- Sam

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5zWDusbrQCdFZzgS9LAFhutTCqEhtWk8xGfw_H0gTmRDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAAbwPoq-YBZod6iH0ThcdSPaZpJ3E5ZjvkmkUUZ1giFarBXxZg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
  Piotr Kaminski <pi...@ideanest.com>
  "That bun is dirty.  Don't eat that bun."

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5xfxnbFZ7OOSSg6oMNQUAa8Wsksotc3sdYNv7VQg%3D_dqg%40mail.gmail.com.

Felix Halim

unread,
Aug 24, 2018, 1:37:35 PM8/24/18
to fireba...@googlegroups.com
The bug has been fixed.
I'm no longer able to reproduce the issue (I've been running it for several hours now).

Thanks all!

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBrVKQ4Axg%3Dq-Ekr8Yt9WBZfo3A3Uim6H2P%2B%2BaMLacPO9Q%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages