Activity Feed for followers - Modelling / Query

[Paul Hinett]

Hi,

 

I have a modelling/query problem I am trying to work out which is a very simple version of Facebooks activity feed, I have lots of activities which are tied to a user…for simplicity just assume they are all the same type of activity.

 

User A has commented on User B’s profile

 

User C is following 1000 Users including User A, I want to return all the activities that have been actioned by any users that User C is following.

 

Now I could probably do this badly with an IN query, but with 1000+ followers like below but it doesn’t seem the right way…is there another way?

Session.Query<Activity, ActivityIndex>().Where(a => a.ActionedByUserId.In(onethousandfollowerids).ToList()

 

Currently my followers are in there own documents, like so..

 

Follower/1

{

                UserId: users/C,

                IsFollowingUserId: users/A

}

 

Any ideas please?

 

Thank You,

Paul

This email has been sent from a virus-free computer protected by Avast. www.avast.com

[eladm]

Hi Paul,

this is not a trivial problem,

indeed the "in query" will not give you the best performance in case of large data-set.

one good direction for this solution is to pre-process in background all of your possibilities.

lets assume you're holding per user fixed number of latest activities, (5000 for instance).

then you're holding a queue for each user. 

when a new post arrives, you insert it to the relevant users queues.

for each queue, if the queue has enough space, it will grow. otherwise, if the queue has reached maximum size, older posts will be deleted.

then you create an index for each queue and simply query the top latest items.

you need to manage it by yourself of course.

[Chris Marisic]

I agree, this isn't something you merely query, this is something you need to coerce out of events into the users owned data

[Ryan Heath]

Back in the days when we were a large social site, we just copied this data across the followers. That was performance wise and logic wise the best step we could do. 

// Ryan

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Hinett]

Do you mean to have a single document for each user which contains all his related activities?

 

I have around 700,000 users currently, would each user now get a document like “users/C/activityfeed” which contains the most recent 1000 or so activities?

 

Then when a new activity is created, insert this activity to every user who should see this activity?

 

And when a user starts following a user, I would need to query the most recent 1000 activities related to this user and create the users/X/activityfeed document?

 

Makes sense, just want to be sure this is what you mean?

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Ryan Heath
Sent: 07 December 2015 09:16
To: rav...@googlegroups.com
Subject: Re: [RavenDB] Activity Feed for followers - Modelling / Query

 

Back in the days when we were a large social site, we just copied this data across the followers. That was performance wise and logic wise the best step we could do. 

 

// Ryan

On Friday, 4 December 2015, Paul Hinett <pa...@ukcreativedesigns.com> wrote:

Hi,

 

I have a modelling/query problem I am trying to work out which is a very simple version of Facebooks activity feed, I have lots of activities which are tied to a user…for simplicity just assume they are all the same type of activity.

 

User A has commented on User B’s profile

 

User C is following 1000 Users including User A, I want to return all the activities that have been actioned by any users that User C is following.

 

Now I could probably do this badly with an IN query, but with 1000+ followers like below but it doesn’t seem the right way…is there another way?

Session.Query<Activity, ActivityIndex>().Where(a => a.ActionedByUserId.In(onethousandfollowerids).ToList()

 

Currently my followers are in there own documents, like so..

 

Follower/1

{

                UserId: users/C,

                IsFollowingUserId: users/A

}

 

Any ideas please?

 

Thank You,

Paul

This email has been sent from a virus-free computer protected by Avast. www.avast.com

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

Pretty much, yes. But you'll probably _merge_ the new activity with that.

Note that you'll probably want to use the auction model, http://ayende.com/blog/96257/document-based-modeling-auctions-bids

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Paul Hinett]

Sorry yes merge is what I mean rather than creating a new doc.

 

Sounds good to me thanks for the help!

[Chris Marisic]

One document is probably too large, unless you use the single document to be nothing be a sequence of keys and perhaps timestamps.

Even then, things start to get big when you have 10,000 or 100,000 things in a person's timeline. You might need to chunk it, like add to the doc and once it is X big, start pumping to a new document. Just be careful how you maintain the crossover being consistent to avoid data loss or missed events.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kamran Ayub]

You could also try http://getstream.io. I'm probably going to implement activity feeds at some point and I don't know if I want to be in the business of managing them myself.

[Chris Marisic]

That's definitely interesting looking