MAP Reduce - Querying on multiple parameters (Select OR Clause)

[archit agarwal]

Some context -

I have a messaging feature in my app and I am trying to do a basic implementation of a query to map and reduce to the conversations with a particular user say (X) ordered by timestamp.
Now in basic terms the logic obv. is -

if ((from is me AND to is X ) OR (from is X and to is me)  //Fairly simple but we can't use it in the map function obviously

My map function is mentioned below, in which we map only messages which are mine irrespective of any other user.

Now I am confused about how to implement the reduce function so I get messages in which => either from is X OR to is X
Any help on this would be really appreciated.

It's not very obvious to me coming from relational db background.
May be I am doing something fundamentally wrong?


+ (CBLQuery*) queryMessagesInDatabase: (CBLDatabase*)db {
    CBLView* view = [db viewNamed: @"messages"];
    if (!view.mapBlock) {
        // Register the map function, the first time we access the view:
        [view setMapBlock: MAPBLOCK({
            id to = [doc objectForKey:@"to"];
            id from = [doc objectForKey:@"from"];
            id msg = [doc objectForKey:@"message"];
            id fromName = [doc objectForKey:@"fromName"];
            id timestamp = [doc objectForKey:@"timestamp"];
            NSString *myuid = [[NSUserDefaults standardUserDefaults] valueForKey:@"myuid"];

            if(to && from && msg && fromName && timestamp){
                if([to isEqualToString:myuid]||[from isEqualToString:myuid])
                {
                    emit(@[timestamp,from,to],nil);
                }
               
            }
        }) reduceBlock: nil version: @"1.1"]; // bump version any time you change the MAPBLOCK body!
    }
    return [view createQuery];
}

[Jens Alfke]

On Mar 1, 2014, at 11:24 PM, archit agarwal <arch...@gmail.com> wrote:

Now I am confused about how to implement the reduce function so I get messages in which => either from is X OR to is X 
Any help on this would be really appreciated.

That’s not the job of a reduce function. Reducing is like aggregation in SQL — “SUM()”, ”MAX()”, “AVERAGE()”, etc.

What you’re asking for in this query is a sub-range: “messages with X” for some X. So what you want is a map function that emits keys that are ordered by the other person’s identity, so you can then query for a specific person.

What I’d do is

                if([to isEqualToString:myuid])
                    emit(@[from,timestamp],nil);
                else if([from isEqualToString:myuid])
                    emit(@[to,timestamp],nil);

Then query by

query.startKey = @[other];

query.endKey = @[other, @{}];

—Jens

[archit agarwal]

Hey Jens!

Thanks a lot for that!
Essentially, we are just creating a view now => which will be different for each user based on the same dataset.

I was trying to reduce the from & to somehow into 1 variable but was thinking in terms of the database structure rather than an indexed view which could be different for each user.

Seems straightforward now & feels a bit stupid to not be able to figure it out earlier.

Cheers
Archit

[Jens Alfke]

On Mar 2, 2014, at 2:36 AM, archit agarwal <arch...@gmail.com> wrote:

Seems straightforward now & feels a bit stupid to not be able to figure it out earlier. 

Don’t feel stupid! Map/reduce is a different approach than SQL-type queries, and it takes a little while to wrap your head around it. Unfortunately I don’t know of any really good docs that explain the tricks.

FYI, there’s a view sort of like this in the CouchChat demo app; you might be interested in looking at it.  This view does have a reduce function, which is used to compute the message-count, mod-date and last speaker for each chat room.

—Jens

[archit agarwal]

Thanks Jen!
I will take a look at it.

Yeah, I have read the existing books on map and reduce for couchdb but none of them deal with complicated cases. They are quite broad.

I have a few follow up questions regarding this architecture -

Q1 Do you happen to have a suggestion on the push notification system I can use for messaging.

I was thinking of having a java server which checks the new messages and sends a push notification accordingly. Having a parameter like isPushed in the message structure or something.

Is this a recommended approach?

Q2 when the user actually opens the app after push notification I assume it would take a few seconds to get these messages. Is there a way to reduce this lag or delay? I know xmpp is pretty fast but it's a whole new implementation. Can we do something to get similar performance?

Q3. The other bit is when a message arrives from a new user who's information isn't available on the local DB. There would also be a lag in pulling this user info. Any suggestions for to make this better?

At the moment I am packing in the name of the user with every message and also thinking of putting in the photo link to amazon in the messages, so I can show something to the user and then when he clicks to see the user details, I show a loading screen if the user details aren't available while I pull it.

[Jens Alfke]

On Mar 2, 2014, at 6:25 PM, archit agarwal <arch...@gmail.com> wrote:

Q1 Do you happen to have a suggestion on the push notification system I can use for messaging. 

You mean, a server-side component that talks to the Apple Push Notification System? Mostly you just need a simple task that watches the gateway’s _changes feed and sends messages to APNS. Chris Anderson put one of these together a while ago as an example; it was pretty simple. Chris, can you point to the code for this?

Q2 when the user actually opens the app after push notification I assume it would take a few seconds to get these messages. Is there a way to reduce this lag or delay? I know xmpp is pretty fast but it's a whole new implementation. Can we do something to get similar performance? 

In iOS 7 there’s a push-notification flag that causes the app to invisibly launch/wake up in the background. It will then presumably start its replication and download the new documents.

Q3. The other bit is when a message arrives from a new user who's information isn't available on the local DB. There would also be a lag in pulling this user info. Any suggestions for to make this better? 

How are you pulling the user info?

—Jens

[archit agarwal]

Hey Jens

Thanks for those tips!

The user info is also being pulled out of couchdb. But its a separate user db.
I don't think it would be advisable to put chats and user in the same database?

Cheers
Archit

[archit agarwal]

1 more item => Do we push every single message or is there a way to check when the user is online...I mean I could write online and offline in a doc everytime an app opens and closes but seems like that wouldn't be scalable?

[Jens Alfke]

On Mar 3, 2014, at 4:54 AM, archit agarwal <arch...@gmail.com> wrote:

I don't think it would be advisable to put chats and user in the same database?

No, that’s actually the best approach. Just use a “type” property to distinguish them from each other.

—Jens

[Jens Alfke]

On Mar 3, 2014, at 6:06 AM, archit agarwal <arch...@gmail.com> wrote:

1 more item => Do we push every single message or is there a way to check when the user is online...I mean I could write online and offline in a doc everytime an app opens and closes but seems like that wouldn't be scalable?

Push every single message. If the client only tries to send a message when the recipient is online, then it fails if they’re not online at the same time. That’s what store-and-forward was invented for.

—Jens

[archit agarwal]

You mean I should put chats and users on the same DB and just distinguish them using a type

User {

Name - xx
{
Type - chat
Sender - xx
To - gg
}
}

Really, is that advisable?
What's the advantage is doing this?

[archit agarwal]

Oh..I think you meant this -

{type = user
name - xx
}
{
type = chat
sender = xxx
}

Just put everything in same db but with diff types rather that create separate dbs?
What advantages do I have with this? Wouldn't this make filtering harder when pulling users based on certain criteria or chats based on which ones are mine?

[Jens Alfke]

On Mar 3, 2014, at 3:44 PM, archit agarwal <arch...@gmail.com> wrote:

Just put everything in same db but with diff types rather that create separate dbs?

Yes. This is what everyone does in the document-database world. It’s the equivalent of using multiple tables. (Our CouchChat and ToDoLite demo apps both demonstrate this.)

What advantages do I have with this?

You can’t query across multiple databases. If you keep different types of docs in the same database you can do some smart queries that combine information from them, almost like what you do with JOIN in SQL.

You also avoid the overhead of running multiple replications.

Wouldn't this make filtering harder when pulling users based on certain criteria or chats based on which ones are mine?

No, why would it? Your map and filter functions are fully Turing-complete so they can just check the ‘type’ property of the document.

—Jens

[archit agarwal]

Thanks for that input! Helps quite a lot. I haven't found much literature on using document type databases more effectively.

When you say smart queries to do joins - can you give an example of this? And can I in some way ensure that I always have the document of the user who sent a new chat message.
I.e user doc + chat doc together.

[Jens Alfke]

On Mar 4, 2014, at 2:08 AM, archit agarwal <arch...@gmail.com> wrote:

> When you say smart queries to do joins - can you give an example of this? And can I in some way ensure that I always have the document of the user who sent a new chat message.
> I.e user doc + chat doc together.

http://guide.couchdb.org/draft/views.html

See the section on “the view for comments”.

—Jens

[CouchbaseLover]

"You mean, a server-side component that talks to the Apple Push Notification System? Mostly you just need a simple task that watches the gateway’s _changes feed and sends messages to APNS. Chris Anderson put one of these together a while ago as an example; it was pretty simple. Chris, can you point to the code for this?"

I don't see any link to that example, would be great! :)

[J. Chris Anderson]

Here is the code. It's more of a proof of concept than an example. https://github.com/couchbaselabs/CouchChat-iOS/tree/push/push-notifications

If anyone ports this to ToDoLite they'll get multiple hero points. 

Chris