Function to return absolute ref of events happening anywhere under the tree

[Aditya Mertia]

Hi Firebase team,

Is there a way(function) in firebase to return the absolute reference (from absolute root) of child_added, child_removed, child_updated events happening anywhere in the firebase tree.

Thanks

Aditya

[Frank van Puffelen]

Within an event handler, you can call ref() to get a reference to that location

    ref.on('child_added', function(snapshot) {

        console.log('Child added at '+snapshot.ref().toString());

    });

This will print the complete URL to the location of the snapshot.

[Aditya Mertia]

So what you mentioned is possible, but my query was a bit different. I want to set a listener on parent e.g. https://xyz.firebaseio.com/appname and want to listen for child_added, child_updated, child_removed events happening anywhere under the hierarchy (e.g. events happeing at https://xyz.firebaseio.com/appname/x/abc/ace/qpr). A function that returns just the reference path wherever the event happened, without the pain of downloading whole snapshot under (https://xyz.firebaseio.com/appname) which could be huge data and irrelevant for each event.

Is this possible? This is a very important function in case we want to sync changes in realtime to another database without setting listeners statically at each child node. 

[Frank van Puffelen]

Ah, sorry for the misunderstanding.

The child_ events for Firebase currently only fire for node one level under where you register the listener. Do you have a use-case where you'd want them to fire for all descendants?

[Aditya]

Yes. My usecase is to sync data in realtime from firebase to my companies existing databases where all the reporting and analytics runs. I would ideally want to set listener on the root of my firebase account and listen for all the child_ events happening at any descendants. I want only 2 params from that function:

1) absolute reference at which the event was fired,

2) snapshot of only the object for which the event was fired.

I believe such a function will be required for most large scale projects that need to sync their realtime database with existing hadoop or nosql store.

According to me this is a useful feature that should be added to firebase. Or do you suggest an alternate way to achieve this?

--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/Rbjwoa_PoxE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/c7766630-b686-494a-91fa-c13910bf65e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kato Richardson]

Aditya,

This is an interesting request. It is, not a request we receive at the enterprise or large scale app level. In fact, the strategy is generally the opposite; listen at a more granular level to control bandwidth and receive concise events. More to the point, listening on the root node for an extremely large data set would be very impractical and non-performant. We could receive thousands (or more!) events per second, not to mention the burden on the server process.

It's hard to understand your goal here without a specific, detailed use case. Since (in my inbox) this appears to be a fragment from an ongoing conversation, which Groups failed to link to the original post, I'm probably missing context here, so bear with me.

Deeply nested data is rarely going to be easy to understand or wield effectively in NoSQL. As a developer working on a project like this, I'd want to hear a pretty strong justification for this approach before spending a lot of time on it, including constraints as to why the data can't be flattened and listeners established in a standard fashion.

2) snapshot of only the object for which the event was fired.

This makes the assumption that there is some defined schema and that Firebase would be able to tell what level in the hierarchy represents a record/object. Otherwise, you'd simply receive individual fields and it would be extremely hard to decide what path represents an "event". For example, if I update a node by setting the value to an object, would I receive separate change events for each child? Or one for the highest level parent? What defines the highest level parent for a "record"?

Cheers,

Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CACTPxgenjfG4RgsbsQG9_8sC%2BCTASDrOqPNFdwxNTYCYVoHg%2BQ%40mail.gmail.com.

[Chris Raynor]

Hi Aditya

If you're trying to keep the data in sync for some map-reduce or BI insight then our private backups feature might be good what you're looking for - you could overwrite the existing db from every new backup in order to guarantee that they're in sync and then run your analytics after that. If you're on the Bonfire plan or higher you can get daily backups, and if you're on the Blaze plan or higher you can get hourly backups, to a Google Cloud Storage or Amazon S3 bucket of your choosing.

However, if you're relying on the Firebase client to report every intermediate state that the db goes through then you will likely run into issues - the realtime database is a state synchronization service not a pub/sub service; the guarantee is that the state between clients will eventually be consistent, not that every state is propagated to every client. In practical terms, this means that the server is free to squash intermediate states into a single update if they arrive there in quick succession or the client is offline during the time the update happened. You will need to modify your data structure (probably using lists and push / childByAutoId) if you want to capture every intermediate state

Hope that helps

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEY%2BseAqzn9xBJGzEpOBW9s2p7QLV5v%3DX6r1qeCSmwk_fw%40mail.gmail.com.

[Piotr Kaminski]

I can't speak for Aditya, but my interpretation of his request -- and incidentally, something that would help me as well -- is simply to enable adding a listener that fires the same events as the streaming REST API.  This would provide specific update events for arbitrarily deep descendants at any desired path or set of paths.  The obvious workaround is to use the streaming REST API, but this gets expensive as the number of subtrees to watch grows since each requires its own open socket.

In my case, I'd like this facility to optimize the synchronization of a local copy of the data.  At the moment, I use a mix of diffing and adaptively installing (normal, shallow) listeners deeper in the data structure if I see it changing.  (Source code here for those interested, look at the firebaseBindRef and mergeAndListen functions: https://github.com/pkaminski/altfire/blob/master/src/altfire.js#L880.)

I believe the streaming events are pretty close to what the SDK gets on the wire anyway, so revealing them to the client shouldn't be too much trouble, hopefully.

Cheers,

    -- P.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHh5jvRvVW%3DaY-j%3D7PiQe%2BD_qg%3DeHF4Wng5LHKNhGUS-eH6qDQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

  Piotr Kaminski <pi...@ideanest.com>

  "That bun is dirty.  Don't eat that bun."

[Michael Lehenbauer]

FWIW, right now we basically provide two ways to observe your Firebase data:

• As an atomic value ('value' listener)
• As a collection of values ('child_*' listeners)

We've occasionally considered providing a 3rd way: as a "tree of data."  In this model, we'd probably give you a "tree_changed" event that would include the path that changed and the new data.  This is indeed essentially what REST streaming and our internal wire protocol use internally.

However we view this as a rare and advanced use case.  In the common case your data has a specific schema and you're going to know whether something represents a value or a collection, and so 'value' or 'child' events are a good fit.  The "tree_changed" event would mostly make sense when you want to write code that deals with arbitrary data.  This is typically for debug tools (like the Data Viewer in your Firebase dashboard) or perhaps syncing tools like you're describing.

FWIW, our data viewer solves this by just recursively creating event listeners all the way down the tree, so we get granular events no matter what changes.  The client is generally good about handling tons of listeners, so this works out fine, even for moderately large datasets.

In any case, thanks for the feedback!  Like I said, this is on our radar and I'll add your vote for the feature. :-)

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CANF_M5wN0irUBTEET31c9fyqN-F0cXZ%2BtRUdL8BRGZtSAAOFLQ%40mail.gmail.com.

[Piotr Kaminski]

Thanks!  I think your summary of the feature request and trade-offs is right on.  Just one note:

On Tue, Jul 14, 2015 at 5:43 PM, 'Michael Lehenbauer' via Firebase Google Group <fireba...@googlegroups.com> wrote:

FWIW, our data viewer solves this by just recursively creating event listeners all the way down the tree, so we get granular events no matter what changes.  The client is generally good about handling tons of listeners, so this works out fine, even for moderately large datasets.

I used to do this but the up-front performance hit was unacceptable on large datasets, so I had to change to the adaptive technique.

    -- P.

[Aditya Mertia]

Private backup even at hourly interval is not what i am looking for. Our requirement is to have a realtime sync at points we decide to sync.

This third way "tree_changed" events is what fits my requirement. I hope a tree_changed event would return 

• The "path" at which the change happened, and 
• The snapshot of only the terminal child where the change happened.

I hope with this kind of event, i am not liable to incur huge bandwidth cost, and the liability of downloading the full snapshot everytime i listen on root.

Waiting for this feature :-)

The way i am planning for a workaround is as follows:

• Any update on any client will also write the events snapshot with path to firebase queue.
• Consumer on the firebase queue will process these requests and sync these with the other NoSQL store.

Need recommendations if this is going to work seemlessly or a heads up in case there can be prospective issues with it (alternate workaround if possible?)

Thanks

Aditya