Chinmay Soman
unread,Oct 11, 2012, 12:08:18 AM10/11/12Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Sign in to report message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to project-...@googlegroups.com
The problem:
The DefaultStoreClient currently in use in production at LinkedIn is a source of a lot of issues. I'm enumerating them below:
* The cluster topology and store metadata has to be synchronized on the client side. This makes bootstrapping (a process by which the client obtains this metadata from a random server node) an expensive operation. In addition, it introduces the additional burden of fetching the latest changes or bouncing the client when such an event occurs.
* Complicated connection management which seems to have operational and performance implications.
* Existing client handles all the replication (including cross zone) which is inefficient.
* The client design is pretty complicated, with multiple levels of nesting.
* Client side config parameters are far too many (sometimes confusing) which causes an operational overhead.
Although some of these issues can be fixed, it requires an iterative release to all the clients. For instance, we have recently implemented an auto-bootstrapper mechanism which tracks changes in cluster.xml happening on the cluster and automatically re-bootstraps the client which helps in doing cluster maintenance. However, in order to reap the benefits, it is essential that all the clients using the cluster pick up this new release. This is a very costly operation when a large number of clients are involved.
The proposal:
Refactor most of the Voldemort logic out of the client and create a new entity called the 'Coordinator'. Essentially we're trying to make the client more lightweight to address most of the issues discussed above.
A high level description of how it will work is as follows:
- We can deploy a set of N coordinators in front of M server nodes (the ration of N:M needs to be worked out).
- The coordinator deployment is symmetric in the sense: each coordinator can satisfy any Voldemort request (get/put/getall/...).
- The coordinator is stateless (across requests) and supports a REST interface.
- When it receives a Voldemort request (get/put/...), it takes care of routing the request to the required server nodes as specified by the store properties, aggregate the result and return an HTTP response.
The coordinator still needs to obtain the cluster.xml and stores.xml, but the point is that its within the control of the administrator (thus, fetching/bouncing is really easy). In addition, all the bug fixes can be quickly deployed due to this property. In summary, this is a huge operational benefit. Please note that the subsequent bug fixes would also be made available as part of the existing thick client (same code path).
Open issue:
The main issue now is how to route a request from the thin client to the coordinator. The ideal solution should take care of the following:
- Fair load balance of the incoming REST requests amongst the N coordinators
- Have the ability to dynamically add / remove / change the coordinator nodes (with minimal impact to the clients)
- Should not have any extra overhead (since every request will route through this piece)
- Should be highly available
At LinkedIn, there is a proprietary solution that resolves the above Open Issue. Unfortunately, the proprietary solution is not open sourced (yet). So we're opening this question to the community: What open source options may resolve our Open Issue?
We will document the detailed design for the coordinator and thin client in a separate wiki. This is just to get some initial feedback from the community regarding the routing logic between
the thin client and the coordinator in the long term. If you have any
ideas please share with us!
Voldemort team