--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwfzZ3f9EkWHFZdOfKfpZ9xyRBKPmirezu7XGki_UjUYPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
I had always hoped that one day we could use Raft on the Druid Coordinators, maybe through something like Copycat: http://atomix.io/copycat/Then there won't be a dependency on an external system, and we could roll in the metadata store too, with some benefits:- No dependency on ZK/Consul/Etcd or a metadata database.- Eliminates a class of bugs where updates to the metadata store are made from an overlord / coordinator that is not actually the leader (due to race between ZK-based leadership election and metadata writes).- Eliminates need for overlord / coordinator to cache metadata items like segments, active tasks, task locks, etc in memory; since we can structure the Raft-based state store to itself be the cache.However I would also be supportive of any work to generalize Druid's coordination layer to support using either Zk or Consul. That same work could be used to help build a builtin implementation later.
Gian
On Tue, Nov 14, 2017 at 1:43 PM, Roman Leventov <roman.leventov@metamarkets.com> wrote:--Reviving this issue: https://github.com/druid-io/druid/issues/1263Consul seems to be superb to ZooKeeper in many aspects: https://www.consul.io/intro/vs/zookeeper.htmlWe could also leverage it's key-value store for indexing locks.
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwfzZ3f9EkWHFZdOfKfpZ9xyRBKPmirezu7XGki_UjUYPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYBcDmosSJa2QC-Q9YGvALbOmFVHok2mk%2BQ9zt_Ns0%3DjTA%40mail.gmail.com.
Does Copycat suit for Coordinator and/or Overlord state management, locks and leader election, or for service discovery as well?
Also a nice thing about Consul is that it has Web UI out of the box.
On Tue, Nov 14, 2017 at 10:40 PM, Gian Merlino <gi...@imply.io> wrote:
I had always hoped that one day we could use Raft on the Druid Coordinators, maybe through something like Copycat: http://atomix.io/copycat/Then there won't be a dependency on an external system, and we could roll in the metadata store too, with some benefits:- No dependency on ZK/Consul/Etcd or a metadata database.- Eliminates a class of bugs where updates to the metadata store are made from an overlord / coordinator that is not actually the leader (due to race between ZK-based leadership election and metadata writes).- Eliminates need for overlord / coordinator to cache metadata items like segments, active tasks, task locks, etc in memory; since we can structure the Raft-based state store to itself be the cache.However I would also be supportive of any work to generalize Druid's coordination layer to support using either Zk or Consul. That same work could be used to help build a builtin implementation later.
Gian
On Tue, Nov 14, 2017 at 1:43 PM, Roman Leventov <roman.l...@metamarkets.com> wrote:--Reviving this issue: https://github.com/druid-io/druid/issues/1263Consul seems to be superb to ZooKeeper in many aspects: https://www.consul.io/intro/vs/zookeeper.htmlWe could also leverage it's key-value store for indexing locks.
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwfzZ3f9EkWHFZdOfKfpZ9xyRBKPmirezu7XGki_UjUYPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYBcDmosSJa2QC-Q9YGvALbOmFVHok2mk%2BQ9zt_Ns0%3DjTA%40mail.gmail.com.--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwdErobF_xPLiX6A0GbjyfF-JDmPA-8cP_RWkbfWivt4Fw%40mail.gmail.com.
I have been working slowly to reduce the dependency of Druid on consensus based system in general to bare minimum so that consensus implementation is used *only* for node discovery and leader election.
My goal is that it should be made possible to write extensions to use the consensus implementation of your choice by implementing just two abstractions https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/discovery/DruidLeaderSelector.java and https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/discovery/DruidNodeDiscoveryProvider.java .
I am creating alternatives for other state management that original had hard dependencies on zookeeper. All the alternatives use a combination of mentioned abstractions for discovering nodes and then use HTTP to interact directly with other nodes instead of depending on zookeeper or anything like that.
- Segment Load/Drop Management at Coordinator ( DONE: See https://github.com/druid-io/druid/pull/4997 )
- Segment Discovery at Coordinator/Broker ( DONE: See https://github.com/druid-io/druid/pull/4997 )
- Node discovery e.g. Router discovering Brokers, Coordinator discovering lookup nodes etc etc has been moved to use abstractions mentioned before. (DONE)
- Overlord task management ( Proposal: https://github.com/druid-io/druid/issues/4996 , WIP PR would be available soon)
All the code that has hard dependency on Zookeeper (aside from two abstractions I mentioned) is deprecated, being kept only for backward compatibility and till above alternatives prove their worth in production. Note that I'm running all of the above alternatives on our internal metrics cluster except for overlord task management which is a work in progress at this time.
Also, I have already added curator based implementations in Druid core for above abstractions and working on copycat based extension. I'm hopeful to run our internal metrics cluster without zookeeper or any other external consensus implementation dependency by the end of Q1-2018 .
At this point, Aside from node discovery and leader election, I would be reluctant to add more dependency on consensus based system so that writing extensions for consensus system of your choice stays very simple and also no persistent state needs to be stored there. So, We should try to find alternatives like we've done for segment/task management instead of adding more dependency on consensus protocol.
Personally, I have never heard metadata store being a problem except for marketing of Druid. However, I think it would be possible to even remove metadata store by just depending on node discovery and leader election from consensus system.
-- Himanshu
On Tuesday, 14 November 2017 21:40:30 UTC-6, Roman Leventov wrote:
Does Copycat suit for Coordinator and/or Overlord state management, locks and leader election, or for service discovery as well?
Also a nice thing about Consul is that it has Web UI out of the box.
On Tue, Nov 14, 2017 at 10:40 PM, Gian Merlino <gi...@imply.io> wrote:
I had always hoped that one day we could use Raft on the Druid Coordinators, maybe through something like Copycat: http://atomix.io/copycat/Then there won't be a dependency on an external system, and we could roll in the metadata store too, with some benefits:- No dependency on ZK/Consul/Etcd or a metadata database.- Eliminates a class of bugs where updates to the metadata store are made from an overlord / coordinator that is not actually the leader (due to race between ZK-based leadership election and metadata writes).- Eliminates need for overlord / coordinator to cache metadata items like segments, active tasks, task locks, etc in memory; since we can structure the Raft-based state store to itself be the cache.However I would also be supportive of any work to generalize Druid's coordination layer to support using either Zk or Consul. That same work could be used to help build a builtin implementation later.
Gian
On Tue, Nov 14, 2017 at 1:43 PM, Roman Leventov <roman.l...@metamarkets.com> wrote:--Reviving this issue: https://github.com/druid-io/druid/issues/1263Consul seems to be superb to ZooKeeper in many aspects: https://www.consul.io/intro/vs/zookeeper.htmlWe could also leverage it's key-value store for indexing locks.
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwfzZ3f9EkWHFZdOfKfpZ9xyRBKPmirezu7XGki_UjUYPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYBcDmosSJa2QC-Q9YGvALbOmFVHok2mk%2BQ9zt_Ns0%3DjTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/9fffb1d8-c4b3-42c3-9e12-eb01bbe85dea%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/047b5cf2-4b82-41cf-812f-06d33b48025d%40googlegroups.com.