How does messaging work in WhatsApp?
11 views
Skip to first unread message
Asim Pereira
May 31, 2015, 11:48:07 PM5/31/15
to motoactve...@googlegroups.com
How does messaging work in WhatsApp?
I come from the background of Java and socket programming. All I know about sending messages over the internet is creating a server socket(for the server side) and a client side socket.
In a multi-threaded environment, a client socket would request the server socket and go into sleep mode till the server replies. First of all I want to know if this is the right appraoch for communication between client and server?
Secondly, relating to WhatsApp, if I am chatting to 10 different friends, then does it mean that WhatsApp has made 10 different socket connections to the server? If yes, then what about the friends I am currently not talking to? This communication model to me feels like the mobile app has many server sockets as well as client sockets on a phone. But I know this is the wrong appraoch. So, guide me in the right direction.
I've heard and read about the web servies like RESTful, SOAP, JSON. But what I don't know is how do they relate to a messaging environment like WhatsApp messaging environment.
Also, my current requirement is that clients are sending their GPS coordinates to a server every 10 minutes. If a certain situation arrives, then other clients(who match the criteria) should revieve the coordinates and act accordingly. In the listeneing case, is the client always connected to the internet waiting for a response? Isn't the waiting period a waste of resources? How does WhatsApp handle this?
In a multi-threaded environment, a client socket would request the server socket and go into sleep mode till the server replies. First of all I want to know if this is the right appraoch for communication between client and server?
Secondly, relating to WhatsApp, if I am chatting to 10 different friends, then does it mean that WhatsApp has made 10 different socket connections to the server? If yes, then what about the friends I am currently not talking to? This communication model to me feels like the mobile app has many server sockets as well as client sockets on a phone. But I know this is the wrong appraoch. So, guide me in the right direction.
I've heard and read about the web servies like RESTful, SOAP, JSON. But what I don't know is how do they relate to a messaging environment like WhatsApp messaging environment.
Also, my current requirement is that clients are sending their GPS coordinates to a server every 10 minutes. If a certain situation arrives, then other clients(who match the criteria) should revieve the coordinates and act accordingly. In the listeneing case, is the client always connected to the internet waiting for a response? Isn't the waiting period a waste of resources? How does WhatsApp handle this?
8 ANSWERS
Rick Reed here talks about some of the things his team did to do to scale their application - Rick Reed, Software engineer at WhatsApp.
The rest of the answer is my guess on what a naive version of WhatsApp could do to implement it's features. WhatsApp or most of the other messaging apps rarely work on a peer to peer basis. So it wouldn't open a connection (from your device) to each of your friends' devices. Instead your device connects to their server. It could then use a custom TCP protocol or maybe HTTP to communicate your messages to the server. The server in return would dispatch them to your friends' devices. If your friend had their app open or at least the app process running there might be a live connection to the server. WhatsApp will use that connection to send them your messages. If their app is "offline" then they might choose to send them a push notification instead.
WhatsApp has chosen Erlang a language built for writing scalable applications that are designed to withstand errors. Erlang uses an abstraction called the Actor model for it's concurrency - http://en.wikipedia.org/wiki/Act.... Instead of the more traditional shared memory approach, actors communicate by sending each other messages. Actors unlike threads are designed to be lightweight. Actors could be on the same machine or on different machines and the message passing abstractions works for both. A simple implementation of WhatsApp could be:
Each user/device is represented as an actor. This actor is responsible for handling the inbox of the user, how it gets serialized to disk, the messages that the user sends and the messages that the user receives. Let's assume that Alice and Bob are friends on WhatsApp. So there is an an Alice actor and a Bob actor.
Let's trace a series of messages flowing back and forth:
WhatsApp actually uses the XMPP protocol instead of the vastly superior protocol that I outlined above, but you get the point.
For your own application things to consider:
The rest of the answer is my guess on what a naive version of WhatsApp could do to implement it's features. WhatsApp or most of the other messaging apps rarely work on a peer to peer basis. So it wouldn't open a connection (from your device) to each of your friends' devices. Instead your device connects to their server. It could then use a custom TCP protocol or maybe HTTP to communicate your messages to the server. The server in return would dispatch them to your friends' devices. If your friend had their app open or at least the app process running there might be a live connection to the server. WhatsApp will use that connection to send them your messages. If their app is "offline" then they might choose to send them a push notification instead.
WhatsApp has chosen Erlang a language built for writing scalable applications that are designed to withstand errors. Erlang uses an abstraction called the Actor model for it's concurrency - http://en.wikipedia.org/wiki/Act.... Instead of the more traditional shared memory approach, actors communicate by sending each other messages. Actors unlike threads are designed to be lightweight. Actors could be on the same machine or on different machines and the message passing abstractions works for both. A simple implementation of WhatsApp could be:
Each user/device is represented as an actor. This actor is responsible for handling the inbox of the user, how it gets serialized to disk, the messages that the user sends and the messages that the user receives. Let's assume that Alice and Bob are friends on WhatsApp. So there is an an Alice actor and a Bob actor.
Let's trace a series of messages flowing back and forth:
- Alice decides to message Bob. Alice's phone establishes a connection to the WhatsApp server and it is established that this connection is definitely from Alice's phone. Alice now sends via TCP the following message: "For Bob: A giant monster is attacking the Golden Gate Bridge". One of the WhatsApp front end server deserializes this message and delivers this message to the actor called Alice.
- Alice the actor decides to serialize this and store it in a file called "Alice's Sent Messages", stored on a replicated file system to prevent data loss due to unpredictable monster rampage. Alice the actor then decides to forward this message to Bob the actor by passing it a message "Msg1 from Alice: A giant monster is attacking the Golden Gate Bridge". Alice the actor can retry with exponential back-off till Bob the actor acknowledges receiving the message.
- Bob the actor eventually receives the message from (2) and decides to store this message in a file called "Bob's Inbox". Once it has stored this message durably Bob the actor will acknowledge receiving the message by sending Alice the actor a message of it's own saying "I received Msg1". Alice the actor can now stop it's retry efforts. Bob the actor then checks to see if Bob's phone has an active connection to the server. It does and so Bob the actor streams this message to the device via TCP.
- Bob sees this message and replies with "For Alice: Let's create giant robots to fight them". This is now received by Bob the actor as outlined in Step 1. Bob the actor then repeats Step 2 and 3 to make sure Alice eventually receives the idea that will save mankind.
WhatsApp actually uses the XMPP protocol instead of the vastly superior protocol that I outlined above, but you get the point.
For your own application things to consider:
- You might not have control over clients sending GPS coordinates to the server every 10 minutes. If your client is running on a mobile device, the OS might decide to starve you from resources or just kill your process.
- You need to maintain state for clients that are connected to your server to make sure you can send messages to active clients when your requirements are met. This is a slight modification of the stock "Comet app" example that almost every framework has.
- Establishing a TCP connection is not a very big waste of resources either from the client's side or from the server's side. If your server software ecosystem supports non blocking IO, the state required per connection is tiny. You could support upwards of 100k connections on a mediocre box if you tried hard. If you are on the JVM Netty might help you here. Python has Twisted and Tornado. C++/ C can make use of epoll, kqueue or select if you are on a *NIX system. Golang supports a high number of connections through it's standard library. We have addressed vertical scalability here i.e. how many users could you support on a simple box.
- If you really want to scale out and build a distributed system that maintains state, you might want to consider Erlang (with OTP) or other implementations of the Actor model, like Akka (JVM) which also supports remote messages. A combination of event sourcing and a message passing architecture could get you all horizontal scalability you need.
Updated 25 May, 2014. 69,069 views.
More Answers Below.
Related Questions
Abhishek Jain, Android app developer and trainer106 upvotes by Sidharth Shah, Marthala Vishnu Vardhan Reddy, Nikhil Vandanapu,(more)
WhatsApp uses XMPP (eXtensible Messaging and Presence Protocol) to handle the message delivery system.
XMPP is mostly like HTTP where the client opens the socket with the XMPP server and keeps it open as long as the client is logged in. It's not like the regular REST API where the client opens the socket send/receive the data and close the socket. The socket is opened as long as you are signed in. In case of WhatsApp that's eternity (not really, WhatsApp reconnects automatically if the connection terminates)
XMPP protocol has been used in various chat applications such as Google Talk, Facebook messenger etc. I have been working on XMPP since last year and that's when I got to know what this protocol could bring. It's not limited to chat apps though. We're developing an application of our own which is not a chat app.
As far as actual technology goes, WhatsApp uses heavily customized version of Smack library on Android to build their client and uses customized eJabberd Server to handle the XMPP traffic. They might have different backend solution for handling the data though which might be on one of the cloud storage/computing network (I think it's heroku, no real idea though).
On iOS and other platforms, I suppose they might have developed their own libraries. Developing own libraries is not a lot of work, especially when you have customized needs and have a team of developers. I have used one of the libraries available for Windows Phone and heavily customized them to work for us. I made some improvement on the library, but due to time shortage I couldn't submit them to the original repo (Documentation is really tough).
Anyways, if you are interested in learning the tech, you can read Oriely's "XMPP: The Definitive Guide" and can visit The XMPP Standards Foundation.
Peace!
XMPP is mostly like HTTP where the client opens the socket with the XMPP server and keeps it open as long as the client is logged in. It's not like the regular REST API where the client opens the socket send/receive the data and close the socket. The socket is opened as long as you are signed in. In case of WhatsApp that's eternity (not really, WhatsApp reconnects automatically if the connection terminates)
XMPP protocol has been used in various chat applications such as Google Talk, Facebook messenger etc. I have been working on XMPP since last year and that's when I got to know what this protocol could bring. It's not limited to chat apps though. We're developing an application of our own which is not a chat app.
As far as actual technology goes, WhatsApp uses heavily customized version of Smack library on Android to build their client and uses customized eJabberd Server to handle the XMPP traffic. They might have different backend solution for handling the data though which might be on one of the cloud storage/computing network (I think it's heroku, no real idea though).
On iOS and other platforms, I suppose they might have developed their own libraries. Developing own libraries is not a lot of work, especially when you have customized needs and have a team of developers. I have used one of the libraries available for Windows Phone and heavily customized them to work for us. I made some improvement on the library, but due to time shortage I couldn't submit them to the original repo (Documentation is really tough).
Anyways, if you are interested in learning the tech, you can read Oriely's "XMPP: The Definitive Guide" and can visit The XMPP Standards Foundation.
Peace!
Written 27 May. 3,566 views.
From a quick search on the internet it looks like Whatsapp uses a custom XMPP server built on Erlang to handle the messaging backend. The advantage of using XMPP is it comes with built in support for rosters and presence so you don't have to manually setup infrastructure and code to manage subscriptions.
Like one-to-one communication XMPP also provides you support for multi user chat which more or less just relays the messages to all users in a given chatroom. Here is a tutorial on how to do that with Ruby but it should give you a good idea of the underlying principle MultiUser Chat using XMPP and Orbited (Using Ruby-on-Rails). In my opinion REST based server consuming messages with a long-polling server or Websockets should help you fetch the data from the backend. I would recommend looking into StropheJS An XMPP library for JavaScript or Orbited orbited2. Orbited allows for a socket like interface with javascript and works on top of Twisted. Though I'm not familiar with the Java ecosystem I've found that the Play Framework and Lift for Scala come with built in support for long polling.
However lately, I've grown partial to messaging middleware which happens to give me more flexibility to achieve the same the features. Though this means that you would have to manage the presence and rosters list yourself. I would recommend looking into Kafka or Rabbitmq. Even Redis comes with a fairly scalable pubsub feature which would help you build message queues (as Redis Lists) and publish subscribe features.
REST based APIs make it easy to work with HTTP requests so every message would be "POSTed" to the server and handed to the your messaging layer or your XMPP server if your working with Ejabberd or Openfire.
The message if fetched by periodically polling with a (GET) long connection timeout - which basically waits till some data arrives on the backend to make it look realtime. Websockets on the other hand would make it more event based.
REST simply defines a set of rules to access the resource behind it.
JSON is just another way to exchange data over the web much like XML.
SOAP is another means of exchanging data using predefined XML schemas which help standardise the format in which messages are exchanged.
With Whatsapp there aren't 10 different socket connections (to the best of my knowledge) but periodic long polls which wait for new data to arrive.
From your requirement I feel Kafka or Rabbitmq or Redis would be better suited to implement it. The RabbitMQ site has beautifully documented the features available and would be a great place to start.
Like one-to-one communication XMPP also provides you support for multi user chat which more or less just relays the messages to all users in a given chatroom. Here is a tutorial on how to do that with Ruby but it should give you a good idea of the underlying principle MultiUser Chat using XMPP and Orbited (Using Ruby-on-Rails). In my opinion REST based server consuming messages with a long-polling server or Websockets should help you fetch the data from the backend. I would recommend looking into StropheJS An XMPP library for JavaScript or Orbited orbited2. Orbited allows for a socket like interface with javascript and works on top of Twisted. Though I'm not familiar with the Java ecosystem I've found that the Play Framework and Lift for Scala come with built in support for long polling.
However lately, I've grown partial to messaging middleware which happens to give me more flexibility to achieve the same the features. Though this means that you would have to manage the presence and rosters list yourself. I would recommend looking into Kafka or Rabbitmq. Even Redis comes with a fairly scalable pubsub feature which would help you build message queues (as Redis Lists) and publish subscribe features.
REST based APIs make it easy to work with HTTP requests so every message would be "POSTed" to the server and handed to the your messaging layer or your XMPP server if your working with Ejabberd or Openfire.
The message if fetched by periodically polling with a (GET) long connection timeout - which basically waits till some data arrives on the backend to make it look realtime. Websockets on the other hand would make it more event based.
REST simply defines a set of rules to access the resource behind it.
JSON is just another way to exchange data over the web much like XML.
SOAP is another means of exchanging data using predefined XML schemas which help standardise the format in which messages are exchanged.
With Whatsapp there aren't 10 different socket connections (to the best of my knowledge) but periodic long polls which wait for new data to arrive.
From your requirement I feel Kafka or Rabbitmq or Redis would be better suited to implement it. The RabbitMQ site has beautifully documented the features available and would be a great place to start.
Written 16 Aug, 2013. 21,318 views.
I don't know how exactly WhatsApp works but messaging described by you looks like an example of JMS. Producer(sender) can create a message which is saved in a queue. When the consumer(receiver) is available he will receive message by checking for messages or through listeners. In JMS, Sender and receiver do not have to be available simultaneously.
Android sdk has google cloud messaging (GCM) through which you can send messages, please check if it can be used with iPhone/BB/Windows phone or else your app would be restricted to android only.
For JMS, You can checkout example of activemq online which is an open source implementation of JMS API. Here is one such tutorial
http://www.javablogging.com/simp...
Android sdk has google cloud messaging (GCM) through which you can send messages, please check if it can be used with iPhone/BB/Windows phone or else your app would be restricted to android only.
For JMS, You can checkout example of activemq online which is an open source implementation of JMS API. Here is one such tutorial
http://www.javablogging.com/simp...
Written 16 Aug, 2013. 13,933 views.
Abhishek Kapoor, programmer5 upvotes by Saiansh Raizada, Sideeq Olanrewaju Damilola, Chung Vo, (more)
Written 17 Jun, 2014. 11,667 views.
I've a little different opinion from that of Rajiv Kurian. Following are my guesses:
1> From 2 and 3:
I think Alice's phone doesn't do the retry push of message. This may be done by the whatsapp server on account of Alice. This is done through a separate polling thread at the server that dequeues the unsent message and an available worker from a thread-pool does the send job. Each message should be having the destination address apart from the message payload itself.
1> From 2 and 3:
I think Alice's phone doesn't do the retry push of message. This may be done by the whatsapp server on account of Alice. This is done through a separate polling thread at the server that dequeues the unsent message and an available worker from a thread-pool does the send job. Each message should be having the destination address apart from the message payload itself.
Written 1 Oct, 2014. 9,677 views.
Amit Gupta, I am Amit Gupta and i like to create ... (more)3 upvotes by Nikhil Ponnuru, Sidharth Shah, and Dinesh Ravi.
eXtensible Messaging and Presence Protocol is used to massaging in whatsapp. and all working in depends on this protocol.
- WhatsApp starts and opens two sockets: One to listen on and one to send a message to the server.
- WhatsApps starts listening on the first socket.
- WhatsApp sends a message containing your phone number and the port of the listening socket to the server and waits for an acknowledgement.
- The server records the phone and port numbers in the message and the IP address that the message came from.
- The server sends an acknowledgement to the app.
- The app receives the acknowledgement and closes the message socket.
- A message with your phone number as the destination comes into the server.
- The server uses the IP address and port number it has associated with your phone number and uses that information to push the message to your phone.
Written Sat. 163 views.
There is still a trade off between XMPP MQTT implementation in WhatsApp
But. Apart from these here you will get the bit and bit of how this system work
RaCHul Chavardol's answer to If I block a person on WhatsApp and later unblock them, will I receive the messages that were sent to me while the sender was blocked?
But. Apart from these here you will get the bit and bit of how this system work
RaCHul Chavardol's answer to If I block a person on WhatsApp and later unblock them, will I receive the messages that were sent to me while the sender was blocked?
Written 7 Mar. 3,012 views.
3 Answers Collapsed
Write an answer
Related Questions
RELATED QUESTIONS
Regards,
Asim
Satish Srinivas
Jun 1, 2015, 12:02:37 AM6/1/15
to Asim Pereira, motoactve...@googlegroups.com
Read this just yesterday...: )
--
You received this message because you are subscribed to the Google Groups "MotoACTVEnthusiasts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to motoactventhusi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asim Pereira
Jun 1, 2015, 12:12:59 AM6/1/15
to Satish Srinivas, motoactve...@googlegroups.com
But when is my MotoX getting Lollipop? :P
--
Regards,
Asim
Satish Srinivas
Jun 1, 2015, 12:22:07 AM6/1/15
to Asim Pereira, motoactve...@googlegroups.com
Why L... You wait for M!!!!
On a serious note.. I owe you a date today..
Reply all
Reply to author
Forward
0 new messages