Looking for tips on firebase queue architecture (Master/slave scripts)

[Alan Haverty]

Hi Firebase forum,

I was recently put on the route of firebase-queue with nodejs on a google app engine flexible environment. Things are going great since!

However, I'm looking for tips/tricks on how to architect the following:

Often it's preferable to not have the client initiate the task.

Instead, sometimes I'd like my server to be in control of starting a particular type of task.

Writing a continuous listener works fine until things start scaling (i.e. Having a script poll a ref, and trigger a task on a change, great for a single instance on GAE FE, but once that script is triggered again, tasks start duplicating)

To get around this problem, I've used a concoction of transactions and onDisconnect() to record which instance is the 'master' script. (I.e., ensuring only one script triggers jobs)

I've used onDisconnect() to ensure a peer script picks up the job as master in the event of the master disconnecting/being scaled down.

I'm also aware maybe placing these scripts on a separate GAE server or manual-scaled heroku dyno would have been an alternative to dealing with master/slave, but I like being able to keep things in one place (disabling/enabling server in one click is great while developing for example).

Long story short, has anyone done anything similar or have a smarter trick?

I feel like it's a common enough use-case, surely I'm not the first to do this?

Regards,

Alan

[Kato Richardson]

Hi Alan,

It's not exactly clear what you mean here when you say "tasks start duplicating" or what the actual problem you're trying to solve is.

Generally speaking, you can just have the clients write to the queue and have the workers write the permanent data, or alternately have the client dual-write to the data and the queue at once and avoid any need for a "master" et al. 

Note that, depending on what you're trying to solve here, you can also set up multiple workers to handle tasks by changing the state of the task rather than resolving it. For example, your "master" could set the state to "phase_1_complete" and your other workers could monitor for that state to begin processing (you accomplish this by setting start_state in the spec for those workers).

Yet another option, assuming you can't just write to the queue from the clients, is to have your listener/master write to the queue on their behalf, instead of trying the master/slave handoff idea. The listeners can be pretty robust and simple.

For example, if I'm monitoring for new user create events so I can email an administrator (or some other process), then all I need to create a robust listeners is to add a created_at timestamp on the user records. Now my listener can a) query on the user path using orderByChild('created_at') and startAt( lastFetchedRecord ). Each time my listener/master gets a new add event, it caches the created_at timestamp and then pushes the task onto the queue. Now the workers pick it up and process it. If the listener/master is restarted, it has the cached created_at timestamp, so it doesn't need to load all users again.

I hope that helps! If not, you may want to ask a specific, technical question with some code to repro, version info, and sample data.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/a6bf58ad-e36f-4065-9be1-53836c9b47f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398