Querying locally cached data / Accessing single results from a collection

[Matthew Huebert]

Hello Firebase team,

First of all, great work on all the new querying features. It's been a few months since I've worked intensively with Firebase, and these are huge improvements.

My current use case involves using Firebase as a central store of settings for about a dozen different web servers, serving requests from various hosts which each have different settings. Each server will connect to Firebase and I plan to use something like the following to always have access to the latest settings to access inside of requests:

# empty "value" callback to ensure that all data is always cached locally,because I want instantaneous access.

settingsRef.on "value", (snapshot) ->

# use new Firebase querying feature to find data inside of requests
app.get "/", (req, res) ->

  # find the settings for the current site, based on hostname. There should only be one match.

  settingsRef.orderByChild("host").equalTo(req.headers.host).limitToFirst(1).once "value", (snapshot) ->

    # the result inexplicably includes a single undefined value in the array before the actual element.

    settings = snapshot.val()[1]

    # now I can use the settings

    title = settings.siteTitle

    ...

# example data 

settings =

  ASDFIWEAFKSNA1234123:

    host: "www.mysite.com"

    title: "My great example website"    

  28932FQWFGW23r24GGWE:

    host: "www.othersite.com"

    title: "One website wasn't enough"

Practically speaking, I know I could store the result from my settingsRef.on("value"...) call in a variable that I use everywhere, and loop through the settings myself to find what I'm looking for. 

My first question is: does the Firebase query logic exist in the javascript driver, so that if all the data is already cached locally it will find it there on its own, or would my second call here (.once "value" ...) trigger another round trip to the server, because it uses these new queries?

Second question: is there a better way, perhaps without calling .val(), to get the title of the single returned element? snapshot.forEach(...) seems a bit weird when I know there's only one element, but I don't know how to get that one element/snapshot. Ideally I could somehow get that snapshot and then call snap.child("title").val() instead of .val()'ing the whole thing.

[Kato Richardson]

Hello Matthew,

Great to meet you and some solid questions.

Let's talk first about how the local cache works in the current iteration of the SDKs. When you detach all listeners from the client, the local in-memory cache is deleted. So when you use once() to grab the data, no local cache will be preserved for the next call. If you were to establish an on() listener, then the once() operations could read from the local cache, assuming they were in the same branch or a child of the same branch as the on() listener.

Equally important is how queries cache data. A query maintains its own cache and doesn't share with any other listeners attached to the same branch (even an equivalent query). So, to be redundant, each listener on a query ref maintains its own internal cache regardless of how similar it may be to another listener. Thus, in both cases here, your data would not be cached and would be downloaded again on each request.

To grab the title, you can use snapshot.child('title').val(), which may be more performant than doing snapshot.val().title. I'll attempt to confirm this by consulting the secret ninja code masters, but I'm fairly sure that's true. I'm not sure what you mean by "get that one element/snapshot". Presumably, this is as simple as calling .child().

I hope that helps!

Cheers,

Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/834f855a-4f74-4944-a63a-3a42e690ad72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Lehenbauer]

Hey Matthew,

It looks like you do have a .on('value') for all of settingsRef (I think Kato might have missed this).  So the SDK will keep all of the data under settingsRef cached and your .orderByChild('host').equalTo(...) queries will be executed client-side without any roundtrips to the server.

As for the question about simplifying "settings = snapshot.val()[1]" I don't understand why you need the [1], given your example data.  But some information that may help:

• I'd expect your snapshot.val() to return an object, not an array.  Something like { ASDFIWEAFKSNA1234123: { host: ..., title: ... } }.  If your data however looks array-like (e.g. the keys are 0, 1, 2, etc. instead of the long ASDF... example you gave), then Firebase might return an array.  In particular if server '1' is the matching server, you'd get an array [null,  { host: ..., title: ... }].  However due to our array heuristics, if server '2' was the matching server, you'd get it as an object: { 2: { host: ..., title: ... } }
  
  In this case (you're currently storing the server info as an array with numeric IDs in Firebase), you might want to consider using push id's or some other unique ID (perhaps the machine hostname itself).  See this blog post on arrays for more insight.
  
  
• If you're sure the entry exists, you could do .once('child_added', ...) instead of .once('value', ...) in which case you would get a single child_added event with the matching item (rather than do 'value', and get an object or array containing the single matching item).  This would change your code to something like:
  
  settingsRef.orderByChild('host')...once('child_added', function(snapshot) { settings = snapshot.val(); }
  
  And of course you could also do the snapshot.child('title').val() trick that Kato mentioned.

Hope this is of some help,

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEYf851SaNkcqaTE0hJnrz_1kR%3DtbQkEUmx5dECu6LKkNQ%40mail.gmail.com.

[Matthew Huebert]

I emailed Michael and Kato off-list to clarify some additional questions I had (thanks for the help!), and below is a summary of my current understanding.

Assume that postsRef is some path in your database ("/posts"), and we have the following two queries:

[a] postsRef.orderByChild("slug").equalTo("my-first-post") 

[b] postsRef.orderBy("publishDate").limit(10)

We can say that [a] and [b] are overlapping, because they are on the same branch, but independent and would not share internal cache, because we have no way of knowing in advance whether [b] would contain [a]. So even though query [b] may have already downloaded the post we're looking for in [a], [a] will hit the server again.

However, we can create the following listener, with an empty callback, to ensure that we've always got a local cache of the entire branch:

[c] postsRef.on("value", function(){})

With this listener in place, any query we run which merely filters or traverses it would use the existing cached data. So the question internally is, "do we already have this entire tree? If yes, query the local cache. If not, hit the server." Given an existing listener [c], queries [a] and [b] will be local, cached queries.

Michael also had this to add about indexes on the client:

As for indexes, we'll basically create them as-needed in the client.  As soon as you do .orderByChild("slug"), we'll start maintaining an in-memory index ordered by "slug."  This may cause a (very slight) hiccup when you start your query, but subsequent updates (as data changes) will just be incremental index updates and be just as fast as handling any other data change.  In practice, the amount of data you're dealing with on the client is small enough that doing some in-memory sorting is no big deal.

My other question, about accessing the data returned, was a bit muddled because my data did, in fact, look like an array - I didn't realize it was significant at the time, but in Forge I had typed in 1, 2, 3, 4, 5... as keys, different from the sample data I typed into my question, which meant that Firebase interpreted it as a list. So that was part of my problem - not being clear about whether I was expecting an array or an object as the result.

Still, there is something to be learned: with a query like q = ref.orderByChild("slug").limitToFirst(1) you are clearly expecting a single result, but if you run q.on("value", function(snap){...}), the snapshot is still one level up from the result you want. It might look like { '-JiHrDOr6HoEzOVibPoN': { host: 'www.apple.com' } } where in fact you want just { host: 'www.apple.com' }. One way to get around this is to use "child_added" instead, which will be called with a snapshot just of the document you want. However, as Michael warns: "The one potential caveat is if you do ref.orderBy('host').equalTo('foo').once('child_added', ...) but there is no item with host 'foo', then you won't get any callback (unless an item with host 'foo' is created later).  So it's not possible to detect the case where there are no matches for your query." 

[Alan deLespinasse]

Is the information about caching in this thread still current, or has client-side caching been improved in the last year and a half?