Best strategy to work with large sets of related data, e.g. a blog post with 2000 comments?

[Christian Spanring]

Hi,

Let’s say I have a post and comment data model, like:

post
 comments: hasMany(‘comment’)

comment
 post: belongsTo(‘post’)

In my comments route I would like to show the comments that belong to a post, ideally paginated if there are too many.

If I retrieve the comment records in my model hook with:

 let post = this.modelFor(‘post’);
 return post.get(‘comments’);

If I inspect the data store at this point, I will see 2000 comments (let's assume I have 2000 entries in my Firebase). Are there really 2000 comments in the local store or am I missing a basic concept about Firebase and websockets?

In can reduce the number of comments that render simply with a `comments.slice(0,20)` before adding them to the template, but I’m wondering if I’m pursuing the wrong strategy and that maybe relying on EmberFire's `findQuery` method or even dropping into the Firebase layer to run more advanced queries would be better.

Any advice will be greatly appreciated!

Thanks,

Christian

[Tim Stirrat]

forgot to send to: +Firebase + EmberJS 

Hi Christian,

In EmberFire's current form, all relationships are eager loaded, which means you will get 2000 fetches and 2000 objects in your local store (sorry!). This relationship fetching logic is handled in Ember Data. I did some reading and it looks like it is possible to lazy load relationships using JSONAPI's link, but it requires non-trivial changes to the EmberFire adapter.

comments.slice(0, 20) will reduce the number of rendered items, but I doubt it will reduce the request count, nor the number of objects stored in the Ember Data store.

If you're concerned about memory or lookups, you could use a raw Firebase query and store.findRecord on each id, store the resulting comments in an array:

const ref = new Firebase('https://id.firebaseio.com/posts/-X/comments');

const query = ref.orderByKey().limitToFirst(20);

const comments = Ember.A();

query.on('child_added', function(snap) {

  const id = snap.key();

  store.findRecord('comment', id).then((c) => {

    comments.pushObject(c);

  });

});

query.on('child_removed', function(snap) {

  const id = snap.key();

  const comment = store.peekRecord('comment', id);

  if (comment) {

    comments.removeObject(comment);

    store.unloadRecord(comment);

  }

});

Or if you want something resembling pagination you could use this pattern  with the firebase-utils paginator (keep in mind it's beta).

Hope that gives you some ideas to work with :)

Cheers,

Tim

--
You received this message because you are subscribed to the Google Groups "Firebase + EmberJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-embe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Spanring]

Thanks for the info!

Dropping into the Firebase lib seems reasonable and is probably the solution I’m going to try.

[Mikko Paderes]

If I'm correct, doing store.findRecord on a post will still load the foreign keys references. When that references count reaches to millions, even if it's just a bunch of "<id>: true" that's gonna take a toll on a client's mobile data.

I'm wondering now if there's a good hacky way to denormalize emberfire data by just storing the post's metainfo and storing the comments organized by post IDs.

Currently, I'm using the raw firebase instead to save and retrieve the comments node in this type of structure. But then I won't have the benefits of ember-data when I do that.

[Tim Stirrat]

You could remove the hasMany relationship definition, so that Ember Data doesn't know about it. Then, use model.ref().child('x') to access the raw info. It would require manual work to save changes to the foreign key list, too.

Cheers,

Tim