|The State of Doozer||Robby Colvin||2/25/13 5:45 PM|
I'm hoping to implement doozer for my infrastructure and I'm sad about the state of the project. From what I can tell, the original authors of doozer have moved on to other things and that development has stopped. There are quite a few forks that have been updated, but there is still no canonical repository to my knowledge. There are quite a few commits in the network, though these ones stick out the most:
I was hoping that we could somehow round everyone up that is still interested in doozer and decide where to go from here. First, is either Keith or Blake still interested in moving forward with the project? If not, should we ask for more members to be added to the organization in order to bring steam back to the project? Or should we create a new organization to host doozer related repos? I understand that Heroku's requirements or situation have changed, but I don't see Zookeeper or DynamoDB as viable options for my situation. I still believe this project has value and would like to hear from everyone else that is using it.
|Re: The State of Doozer||Erik St. Martin||2/25/13 8:45 PM|
We (skynet team) are also really into the idea of seeing doozer further developed, and have been in a similar boat with the abandonment of the project. We've actually been researching porting skynet over to ZooKeeper as it's more actively maintained, has some great tools surrounding it, and supports things like ephemeral nodes, so we can make sure nodes are removed when our services die, and it works pretty much exactly the same way doozer does.
Two other forks of doozer that we've pulled patches from are:
Keep me posted on how this comes along, I wouldn't mind contributing to some of the work that may need to be done.
|Re: The State of Doozer||snakes||2/26/13 6:25 AM|
We (bitly) have a fork as well that is production worthy. We've been running it as part of a core service for over a year now.
Other than the obvious fixes needed to get it to compile it has been rock solid.
We have *not* had to pull in any of the other work going on in the rest of the more active forks. I've only added a single new feature (SELF command) so that you can deterministically identify a node at runtime.
We do have some scripting that isn't open sourced (but could be) that adds starting and managing a cluster.
I would also be interested in giving a little TLC to the project.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 6:41 AM|
One of the things we really want for skynet is Ephemeral nodes - paths that disappear when they disconnect. Eric mentioned earlier that one of the forks has this already. This would be a great feature if we were to get a "canonical" fork.
We also have some tooling for doozer that could be open sourced pretty easily, like a CLI, it's skynet specific right now, but could be pretty easily fixed up to be generic.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 7:23 AM|
How about we get everybody interested on a conference call tomorrow? 2PM Eastern? I can send out a conference bridge number, and we can work out how to effectively manage a single canonical fork of Doozer.
Let me know if you're all interested in a call, and I will send out a conference number.
|Re: The State of Doozer||Erik St. Martin||2/26/13 7:26 AM|
https://github.com/araddon/doozerd is the fork I believe I saw with ephemeral node support. I'm not sure the state of that fork I haven't used it.
I'd definitely be willing to contribute a CLI similar to ZooKeepers for doozer if we can find a canonical place for doozer so that all these additions aren't scattered across 10 different forks.
|Re: The State of Doozer||Gustavo Niemeyer||2/26/13 7:51 AM|
On Tue, Feb 26, 2013 at 12:23 PM, Brian Ketelsen <bket...@gmail.com> wrote:Sounds like a good idea. Perhaps a G+ hangout if that works for everybody?
I've never used Doozer, but hoped to use it at one point, and may see
it being useful in the future. We ended up using ZooKeeper via gozk,
and then MongoDB since our needs didn't quite match the ZooKeeper
gustavo @ http://niemeyer.net
|Re: The State of Doozer||Mike Damm||2/26/13 10:22 AM|
Torbit started dusting the cobwebs off of doozer internally (since we are a Go heavy shop it made perfect sense), and would be willing to help out.
|Re: The State of Doozer||Keith Rarick||2/26/13 10:37 AM|
Hey folks, this all sounds fantastic. I'd love to open up the ha org
if there's interest to more people who want to keep things going.
Or whatever else helps remove friction.
|Re: The State of Doozer||Phil Whelan||2/26/13 8:57 AM|
We, ActiveState, also have a fork of Doozerd, in which we have implemented ephemeral nodes. We use this extensively in our product, Stackato. The ephemeral nodes really help in getting an overview of processes running state throughout a cluster.
Bit more background here...
We have not put a great deal of effort in doozerd development and are limited on the resources we can assign to it. Some of the work we have done is on the Ruby client libraries, but some of that work is specific to our implementation.
We have found several limitations with Doozerd, we would like to address, but we are also considering alternatives such as ZooKeeper or some other (more light-weight than ZK) clustered key-value implementation. We depend on several features of doozerd, most notably watchers, ephemeral nodes, clustering and being light-weight.
Limitations we've found include... it degrades rapidly the more values you put in, it's slow to get a group of config values in one go (lots of tcp packets back-and-forth), becomes disabled if you post large values (3kb) requiring restart. Generally it's slow, for what it does, even with a single doozerd instance, which surprises me since it's data is all in RAM.
We are limited on development time we can commit to this, but we do have insights into using doozerd, as it's been a part of our product for a little while now. We would love to talk more.
Twitter : http://www.twitter.com/philwhln
LinkedIn : http://ca.linkedin.com/in/philwhln
Blog : http://www.bigfastblog.com
On Tue, Feb 26, 2013 at 7:51 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
|Re: The State of Doozer||Matt Proud||2/26/13 10:51 AM|
What a spectacular idea. A number of us from SoundCloud would like to join the call, and that time will work for me. You can shoot me the call information at matt (at) soundcloud (dot) com.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 10:55 AM|
Perhaps we should wait another day before organizing a hangout (great idea for those outside USA, Gustavo!) so we can get more participants.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 11:15 AM|
Or, we just ask Keith nicely to add someone from ActiveState, Bitly, Skynet, torbit, etc to the HA org and we work through our updates with pull requests and issues like any normal project. If Keith is open to adding a few people to the org, we don't have much of a problem left to solve that can't be done with some good pull requests.
|Re: The State of Doozer||Robby Colvin||2/26/13 11:17 AM|
Wow, thank you for all the responses.
Is anyone opposed to continuing with using the ha org on Github and just having everyone push their changes upstream? Since Keith has given his blessing I think it might be better than risking another fork being abondoned later on. I also think it might be nice to have a sort of quorum so the work can be distributed easier and also keep clients and other tools in once place. Even if it was the few of us here I think it could go a long way.
|Re: The State of Doozer||snakes||2/26/13 11:21 AM|
I would prefer it continue to stay where it is and Keith add a few of us, it's already got some momentum there and it should become the canonical repo.
I also agree with Brian that a conference call might be overkill, this mailing list + pull requests should suffice for feature proposals and working through design decisions.
|Re: The State of Doozer||Gustavo Niemeyer||2/26/13 11:28 AM|
On Tue, Feb 26, 2013 at 4:17 PM, Robby Colvin <geeta...@gmail.com> wrote:
> Wow, thank you for all the responses.Are all the different forks well tested and sensible to go upstream?
It'd be good to have one or two people that have a good grasp on the
internals of the project, and a slightly critical eye, to be doing a
round of integration.
gustavo @ http://niemeyer.net
|Re: The State of Doozer||Edward Muller||2/26/13 10:56 AM|
I've had passing interest in Doozer and would like to attend as well. I started a fork the other day with the intent of going through more active repos and pulling in changes that looked interesting/useful/etc. Note: I haven't actually started yet though.
|Re: The State of Doozer||Keith Rarick||2/26/13 11:39 AM|
On Tue, Feb 26, 2013 at 11:15 AM, Brian Ketelsen <bket...@gmail.com> wrote:Those options don't have to be mutually exclusive. :)
Gustavo raises a good point that care should go into merging things in, but I'm
sure everyone here will be reasonable and not go making crazy commits without
any review. I'll go ahead and add the owners of those forks listed
above to the ha
org and the community can work out what to do collectively from there on.
I'd love to be in the conference call or google hangout if it happens.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 11:51 AM|
That's awesome news, Keith. And I am sure that with the amount of production experience we all have with Doozer, we'll all take great care not to break things.
Let's do a Google+ hangout on Thursday, just to introduce everyone. Seems like everyone involved who has requested involvement is within the US timezone span, so how about 2PM Eastern time on Thursday? I'll post a URL when we get some concurrence on timing.
|Re: The State of Doozer||Keith Rarick||2/26/13 11:53 AM|
On Tue, Feb 26, 2013 at 11:39 AM, Keith Rarick <k...@xph.us> wrote:Okay I'm not sure exactly what github accounts to use for some of the forks,
because several are org accounts and can't be added directly to the ha org.
Can folks reply here and identify user accounts for them?
|Re: The State of Doozer||Keith Rarick||2/26/13 11:58 AM|
|Re: The State of Doozer||snakes||2/26/13 12:00 PM|
for bitly, account is mreiferson
|Re: The State of Doozer||Robby Colvin||2/26/13 12:02 PM|
My fork doesn't have much right now, but I can volunteer to start adding some tests and help out with other things unless anyone has an objection to that. I can also beef up the Go client to be more compatible with the service. My Github username is geetarista.
|Re: The State of Doozer||Brian Ketelsen||2/26/13 12:13 PM|
For skynet, please add bketelsen and erikstmartin.
|Re: The State of Doozer||Erik St. Martin||2/26/13 12:15 PM|
I'm so pumped there is a community rallying behind doozer now, we've been concerned that we'd have to maintain it on our own, or to migrate skynet to something else.
|Re: The State of Doozer||snakes||2/26/13 12:17 PM|
Also, while we have the whole group here, we should discuss some early ground rules given the fact that, at some level, we may have differing visions of where the project should go.
The priorities as I see them:
1. obvious TLC that enables easy compilation, fixes bugs, and brings the codebase up to date
2. discussion of our collective production experiences and documentation of some of the pain points that need work (this will help us agree on some sort of prioritized roadmap for feature/design changes)
3. keep the core codebase and purpose small :)
Relatedly, given the fact that there will be several cooks in the kitchen and able to merge things, we should commit to some agreement that pull requests need some sort of quorum of support (much in the spirit of doozer :).
|Re: The State of Doozer||Brian Ketelsen||2/26/13 12:28 PM|
We were just discussing this internally. How about we use some basic github tools to manage what gets done and how.
1. For ideas, we submit issues. Issues with enough (how many??) +1's get approved for work.
2. Work happens in forks, forks submit pull requests
3. Pull requests get code reviewed, and require some sort of quorum/consensus for merging. Go's LGTM plan works well.
We can create a wiki page when we've agreed to the guidelines. We also need to agree on versioning, etc. so we don't create breaking changes. We can create and publish a roadmap to help with this.
Perhaps this list is the best place to start airing out ideas. When we see some solid patterns, we can move to issues on Github. Also, I certainly hope for this to be as democratic as possible, so having a small committee will help that. Too many chiefs kills too many projects. As long as the big production instances of doozerd are represented, we should be good.
We're really excited to see life back in the doozer community!
|Re: The State of Doozer||Erik St. Martin||2/26/13 12:29 PM|
Agreed, I definitely think we should do pull-requests, and agree on a certain number of +1's / LGTM's before a pull-request is accepted, and possibly at the feature level. I'd definitely like to see a compiled list of things people would like to see implemented/fixed.
|Re: The State of Doozer||Phil Whelan||2/26/13 1:44 PM|
Thanks Keith. From ActiveState, please add GitHub users srid and philwhln
Cell : +1 (778) 233-4935
|Re: The State of Doozer||Daniel Huckstep||3/15/13 8:53 PM|
I'm a little late to the party, but I'm also interested in doozerd.
A small part of my job at Yardstick is all the devops stuff. I need to split things up to better utilize our big VM server, so was going to use doozerd to hold config information (where's the DB yo?). A main "master" server, then each VM has their own to read from that slaves off the master (okay there is no master/slave, but you get the point).
While I haven't gotten far enough to decide if doozerd is missing any features or anything, I'm definitely keeping my eye on the repo and threads like this. If I find myself needing stuff, it'll create issues or a PR.
|Re: The State of Doozer||David Arnold||1/2/14 5:56 PM|
So, this whole thread was a while ago now, and there's nothing much happening on github.
What's the current state? Is doozer dead?