Time to hand over the reigns before Capistrano costs me my youth?

[Lee Hambley]

Dear List Members,

I've been experiencing a rather overwhelming level of burnout lately, as I'm sure many of you regular readers on the list and on Github have noticed, my patience has grown short, and I'm not the passionate, helpful chap I once was. (Or at least, that I once tried to be)

I'm not even sure if it's a case of poisonous people sapping my energy, or a more general case of burnout since I run my own company, and have very little free time.

I'm considering stepping down from maintaining Capistrano at all, if I had to pick on a shortlist of reasons, it'd be:

• I don't use Rails all that much anymore, and many of the problems people report with Cap are really problems of Rails (i.e the entire manifest/asset pipeline disaster). When people have problems, I'm not equipped to diagnose what might be going wrong, as I simply don't deploy that way. My rails projects are all Rails 4 with no assets, or they are Rails 3 with the most standard 
• I've rewritten Capistrano and it's now a better tool, but I too cowardly to release it and make it mainstream, as Im afraid it'll destroy whatever good will for open source I have left when the flood of support questions inevitably comes in, followed by all the people who are unhappy with what I've built and feel obliged to tell me how bad I am at software.
• Ruby Gems is an awful platform. I feel crushed by the burden for making things secure, both making things difficult to use wrongly, and making them safe to use. Ruby Gems makes signing gems overly difficult, uses non-standard methods, and nobody bothers to validate them anyway.

Whilst I believe strongly in Capistrano as a general purpose tool for all people right up to epic scale, where things start to get really bespoke for companies such as the Twitters and Google. I do think the future of software deployment is in small, containerised VMs and so-called PaaS, as what we're all doing right now has to end, some time. 

Non-deterministic deploys of code from (usually) un-tagged source control, with production environments needing all kinds of eccentric heavy weight tools to help with mutating assets to solve problems that are mostly a product of poor framework in the first place design can't go on forever.

Whilst I think Rails is a great framework (and by and large most users of Capistrano are Rails users), it's asset pipeline is a poorly thought out idea which causes a myriad of problems. Whilst I love Ruby as a language, it's failure to standardise on an interpreter has lead to horrible situations with people using rvm and rbenv and chruby in production where things really ought to be more specified.

These problems are problems of other tools, problems of poor design, and problems of poor education. People are often using rvm and rbenv in production environments because Ruby is pathologically difficult to install correctly on modern Linux distributions; and more often than not people choose LTS versions of their distribution, and then throw those guarantees out of the window by replacing system components with bleeding edge versions of turbo-GC hacked version of their required interpreters. This wouldn't be a problem, except that it's left up to Capistrano, and by extension to me to work out all the insane ways people might configure their repositories and production environments, and interpreter switchers and try to find a way to make it all work together. So far I've been holding it together, but I'm starting to fall apart at the seams, and I don't want Capistrano to fall apart with me.

To everyone who has contributed code to the 2.x branch, and everyone who has contributed code to the 3.x branch, in particular Tom Clements and Kir Shatrov I feel an immense debts of gratitude. To @torrancew on Freenode IRC (I don't want to use your real name here, and I couldn't reach you to ask for permission) thank you sincerely for years of patience, and support in the #capistrano channel.

For everyone else, I'd appreciate your input, how can I take the load off myself, how can I find a way to do this better, and build a better tool that is easier to maintain and easier to release, without the fear of breaking people's builds? How can I trust someone enough to hand over the baton if I can't learn to live with this level of stress?

To anyone who works at Github who reads this, why can't I add a file to tell people not to open issues on Github for user support, when there's a perfectly good mailing list here???

My next steps:

1. I'll release gems, correctly version bumbed (semver) for all unreleased code. If it's broken, sorry.
2. If people are using Capistrano without locking a known-good version in their Gemfile.
3. I won't be signing Gem releases, it's too painful, and so few people actually bother verifying Gem signatures, it's simply not worth it. And I hate that.
4. I'll be closing all open issues and pull requests at GH that relate to the v2 branch of the code.
5. I'll be yanking 2.5.15 as it's broken (something about Subversion flags that I have no way of testing, or verifying, and none of the proposed fixes include tests, so I'm rolling back to 2.5.14 which apparently didn't exhibit this problem.)
6. I won't be taking any more code for the v2 branch of the code. I haven't used it for more than a year, it's slow, was broken by some ill advised merges from someone who was trying to help, but really didn't make things better
7. I'll be looking for help with testing things before they are released. That means, following this release, no release until whoever contributed the code can verify that it works, and it's spent a while in beta.
8. I won't be seen in the IRC channel very often, I can't afford the time to maintain a presence there, and it's a wasteful medium for helping people as it's such an unstructured ephemeral stream of data.

I'll be encouraging everyone I meet to find a better way to deploy software, using containers, or switching to a language better suited to deployment, where Gem bundlers aren't required to get all the load paths fixed, and where concatenating a few css files, and minifying Javascript doesn't take 10 minutes.

I'll hold the reigns for now, but I need your help to keep going. I need to know that there are people for whom this project still matters, as the only people I ever hear feedback from are the disappointed, angry, vocal, entitled, minority.

Yours, an exhausted Lee Hambley

--

http://lee.hambley.name/

+49 (0) 170 298 5667

[Vipul Amler]

Hi Lee, 

On Sunday, October 6, 2013 10:41:47 PM UTC+5:30, Lee Hambley wrote:

Dear List Members,

I've been experiencing a rather overwhelming level of burnout lately, as I'm sure many of you regular readers on the list and on Github have noticed, my patience has grown short, and I'm not the passionate, helpful chap I once was. (Or at least, that I once tried to be)

I'm not even sure if it's a case of poisonous people sapping my energy, or a more general case of burnout since I run my own company, and have very little free time.

I'm considering stepping down from maintaining Capistrano at all, if I had to pick on a shortlist of reasons, it'd be:

• I don't use Rails all that much anymore, and many of the problems people report with Cap are really problems of Rails (i.e the entire manifest/asset pipeline disaster). When people have problems, I'm not equipped to diagnose what might be going wrong, as I simply don't deploy that way. My rails projects are all Rails 4 with no assets, or they are Rails 3 with the most standard 
• I've rewritten Capistrano and it's now a better tool, but I too cowardly to release it and make it mainstream, as Im afraid it'll destroy whatever good will for open source I have left when the flood of support questions inevitably comes in, followed by all the people who are unhappy with what I've built and feel obliged to tell me how bad I am at software.
• Ruby Gems is an awful platform. I feel crushed by the burden for making things secure, both making things difficult to use wrongly, and making them safe to use. Ruby Gems makes signing gems overly difficult, uses non-standard methods, and nobody bothers to validate them anyway.

Whilst I believe strongly in Capistrano as a general purpose tool for all people right up to epic scale, where things start to get really bespoke for companies such as the Twitters and Google. I do think the future of software deployment is in small, containerised VMs and so-called PaaS, as what we're all doing right now has to end, some time. 

Non-deterministic deploys of code from (usually) un-tagged source control, with production environments needing all kinds of eccentric heavy weight tools to help with mutating assets to solve problems that are mostly a product of poor framework in the first place design can't go on forever.

Whilst I think Rails is a great framework (and by and large most users of Capistrano are Rails users), it's asset pipeline is a poorly thought out idea which causes a myriad of problems. Whilst I love Ruby as a language, it's failure to standardise on an interpreter has lead to horrible situations with people using rvm and rbenv and chruby in production where things really ought to be more specified.

These problems are problems of other tools, problems of poor design, and problems of poor education. People are often using rvm and rbenv in production environments because Ruby is pathologically difficult to install correctly on modern Linux distributions; and more often than not people choose LTS versions of their distribution, and then throw those guarantees out of the window by replacing system components with bleeding edge versions of turbo-GC hacked version of their required interpreters. This wouldn't be a problem, except that it's left up to Capistrano, and by extension to me to work out all the insane ways people might configure their repositories and production environments, and interpreter switchers and try to find a way to make it all work together. So far I've been holding it together, but I'm starting to fall apart at the seams, and I don't want Capistrano to fall apart with me.

To everyone who has contributed code to the 2.x branch, and everyone who has contributed code to the 3.x branch, in particular Tom Clements and Kir Shatrov I feel an immense debts of gratitude. To @torrancew on Freenode IRC (I don't want to use your real name here, and I couldn't reach you to ask for permission) thank you sincerely for years of patience, and support in the #capistrano channel.

For everyone else, I'd appreciate your input, how can I take the load off myself, how can I find a way to do this better, and build a better tool that is easier to maintain and easier to release, without the fear of breaking people's builds? How can I trust someone enough to hand over the baton if I can't learn to live with this level of stress?

Decentralize. 

Give more people who have contributed frequently, access to Issues triaging, and Commit Access to offload the stress.

That way everyone chips in at regular intervals.

To anyone who works at Github who reads this, why can't I add a file to tell people not to open issues on Github for user support, when there's a perfectly good mailing list here???

My next steps:

1. I'll release gems, correctly version bumbed (semver) for all unreleased code. If it's broken, sorry.
2. If people are using Capistrano without locking a known-good version in their Gemfile.
3. I won't be signing Gem releases, it's too painful, and so few people actually bother verifying Gem signatures, it's simply not worth it. And I hate that.
4. I'll be closing all open issues and pull requests at GH that relate to the v2 branch of the code.
5. I'll be yanking 2.5.15 as it's broken (something about Subversion flags that I have no way of testing, or verifying, and none of the proposed fixes include tests, so I'm rolling back to 2.5.14 which apparently didn't exhibit this problem.)
6. I won't be taking any more code for the v2 branch of the code. I haven't used it for more than a year, it's slow, was broken by some ill advised merges from someone who was trying to help, but really didn't make things better
7. I'll be looking for help with testing things before they are released. That means, following this release, no release until whoever contributed the code can verify that it works, and it's spent a while in beta.
8. I won't be seen in the IRC channel very often, I can't afford the time to maintain a presence there, and it's a wasteful medium for helping people as it's such an unstructured ephemeral stream of data.

I'll be encouraging everyone I meet to find a better way to deploy software, using containers, or switching to a language better suited to deployment, where Gem bundlers aren't required to get all the load paths fixed, and where concatenating a few css files, and minifying Javascript doesn't take 10 minutes.

I'll hold the reigns for now, but I need your help to keep going. I need to know that there are people for whom this project still matters, as the only people I ever hear feedback from are the disappointed, angry, vocal, entitled, minority.

I am sorry to hear about this. As a user of capistrano I thank and appreciate all the work you and everyone has put into it.

Thankyou!

[TR NS]

On Sunday, October 6, 2013 1:11:47 PM UTC-4, Lee Hambley wrote:

I've rewritten Capistrano and it's now a better tool, but I too cowardly to release it and make it mainstream, as Im afraid it'll destroy whatever good will for open source I have left when the flood of support questions inevitably comes in, followed by all the people who are unhappy with what I've built and feel obliged to tell me how bad I am at software.

That's not an unreasonable fear.  You don't say so explicitly, but I assume this is because there are considerable "back-breaking" compatibility changes in the rewrite. I would suggest the answer then is to rename it and release as a new project. This allows the old project to live on harmoniously, and give the new project a fresh start. No fear necessary.

[Daniel Hengeveld]

To anyone who works at Github who reads this, why can't I add a file to tell people not to open issues on Github for user support?

Hi Lee - if you add a CONTRIBUTING file to the root of your repo, a message will appear atop the "new issue" form suggesting they read the contributing guidelines. Here's the documentation: https://github.com/blog/1184-contributing-guidelines  I hope this helps a bit.

~ @thedaniel

[Ralph von der Heyden]

Hey Lee,

I think we all take capistrano as a commodity, plus the 2.X version is "good enough" for most of us, so that very few people do chip in with the development. On the other hand, cap is such a widespread tool (4,737,971 downloads, holy crap!), so that bug reports like "OMG I don't know what I'm doing but this does not work LOL" are inevitable. This will be hard to tackle without a *team* of dedicated volunteers cleaning up the issue tracker, helping others out and stuff like that. But keeping up your current level of engagement is certainly not an option.

Nevertheless I thing cap3 should be released as cap. I'm using it in production with 2 projects, and it has been working like a charm until now. The work done around sshkit, the new dsl and countless other things are just too good to never see the light of day. Users that don't want to switch can easily bundle a 2.X version, cap is not the first gem to introduce breaking changes. Still, one needs to handle the support load. Maybe you can set up some FAQ about how to ask for support, file bug reports etc., then volunteers can go and point people that opened bug reports for user support to the FAQ and close the request (you can even create a bookmarklet for that, so it's a one click action). Some kind of policy or routine that prevents the issue tracker from running over, keeping the noise down to help you concentrate on the important parts.

Last but not least I want to thank you for maintaining cap for so long, pouring countless hours of your free time into the project. I hope you'll feel better soon. :)

Cheers,

Ralph

[Srđan Prodanović]

The way we move projects between environments is really an overly complex and flakey procedure these days.

We're very grateful to have Capistrano around to help us handle all the hassle of sophisticated deployment scenarios.

I also think you could release the rewrite with a new name. Thousands of your followers will greatly appreciate it (despite not being as vocal about it).

I hope decentralization, the prevalence of containers, and a CONTRIBUTING.md file ease your troubles with maintaining great open-source projects.

Hang in there,

Srđan

[Brad Landers]

Lee, I feel terrible for never having posted so much as a word of thanks for your efforts dedicated to Capistrano over the years. I know this is a horrible consolation, but you should consider that part of the reason you don't hear positive things about Capistrano is because it is the de facto standard for deploying Rails apps. For better or for worse, Capistrano gets taken for granted *because it works so well*. Capistrano is one of those tools that I've been using since Rails 0.7. Wow! I mean, stop to think about that for a moment. How many software authors can make that claim?

It sounds like you've become displeased with the Ruby ecosystem in general, and you're certainly justified in many of your complaints, but I want you to know that Capistrano has been a cornerstone of the trade that has paid my bills for the last 6 or 7 years. Capistrano is a piece of software that works every time I call on it. For that, I thank you. 

On Sunday, October 6, 2013 1:11:47 PM UTC-4, Lee Hambley wrote:

[Brad Gessler]

Hi Lee,

Thanks for all of your hard work on Cap. Its no wonder why you're feeling burnt out.

Most of the problems and frustrations you express have everything to do with there not being a clearly defined process of compiling apps, resolving dependencies, and execution. http://12factor.net/ is the most concise idea of seen to date, but the toolset around this is lacking. In my mind cap tries to do too much when its compiling Rails assets, running bundler, etc. Cap is best at running a series of tasks on multiple machines (love SSHKit btw) and rolling everything back if something goes wrong.

Perhaps some of the load could be taken off of Rails deployment code could be moved out of the Capistrano gem and moved into something like "capistrano-rails". You could continue focusing on the Capistrano gem without all of the Rails cruft and find other maintainers to deal with "capistrano-rails" complexity.

Brad

[John Knight]

Unsure if this has been mentioned, since I'm still reading through all the responses, but having used Cap almost exclusively for _non-Rails_ projects, and having switched from cap to ansible for a new set of projects, there's a pretty easy translation between the two, in a way that cap and chef/puppet/cfengine don't have ... One thought would be to work with the Ansible guys to develop a set of cap-style modules on top of ansible that would take advantage of all the great things cap can do, but without having to maintain the core of the crazy anymore.

Getting a native Ruby-based module base would both ease the port of cap-specific functionality _and_ make it easier for the cap community to build their own ansible modules.

Just a thought.

Bug

On Sunday, October 6, 2013 1:11:47 PM UTC-4, Lee Hambley wrote:

[Tamer Rizk]

Hi Lee,

"I don't use Rails all that much anymore"

So what is your framework of choice now?

Thanks,
Tamer

[Aaron Cruz]

I really appreciate the time and energy you put into a tool that I use very often. I also can't understand how you could find the energy to put into something you don't use. I am horrible at supporting anything I don't use. Absolutely horrible.

These problems are problems of other tools, problems of poor design, and problems of poor education. People are often using rvm and rbenv in production environments because Ruby is pathologically difficult to install correctly on modern Linux distributions; and more often than not people choose LTS versions of their distribution, and then throw those guarantees out of the window by replacing system components with bleeding edge versions of turbo-GC hacked version of their required interpreters. This wouldn't be a problem, except that it's left up to Capistrano, and by extension to me to work out all the insane ways people might configure their repositories and production environments, and interpreter switchers and try to find a way to make it all work together. So far I've been holding it together, but I'm starting to fall apart at the seams, and I don't want Capistrano to fall apart with me.

One of the big causes of this is that a large percentage of Ruby devs are not coming from an operations background. They are learning by doing. They are told to use a ruby version manager on our local machines and believe that they should follow that on the productions side. They are also told that GC hacks can give you X performance boost and not being told the trade-offs. Most of us just want to write code and then deploy it with a simple and useful tool like capistrano. Your opinions about these things make a lot of sense but this is not common knowledge in our community.

If you are able to pass the reigns or free up more time, I would love to hear more of your opinion about production environments in a blog post or more if this doesn't already exist. I would buy the e-book.

Good luck and thanks again,

Aaron

[Caleb Brown]

I use capistrano for all my projects (rails, django, wordpress, magento, etc.) I'm very grateful for it and agree that your concerns are major and should be addressed by the community at large.  

I'm sorry you've had to deal with poisonous people and the fragmentation and I hope that the community can step up and help out while still addressing your concerns.

-c

On Sunday, October 6, 2013 1:11:47 PM UTC-4, Lee Hambley wrote:

[jason.p...@acquitygroup.com]

Greets Lee,

 

I've spent the last 3 years of my career working almost exclusively with using Capistrano to deploy non-rails apps. I am also a founding member of a DevOps practice at my company. We make capistrano part of our job description for working on our team. The power in capistrano is it's parallel ability to deploy simply and sufficently. It also acts as a stepping stone to Puppet/Chef.

 

I haven't had a chance to introduce myself because I haven't had a need to bug you. The software works, quirks and all. If it doesn't, override it and move on...

 

That said, if you need project management help or project governance help or QA help so you can focus on what is right for -you-, tell us what you need done.

 

Personally, I think Brad's idea of creating a Capistrano "core" and subgems has a lot of merit.  That can be even extended to the various SCM implementations. I also think you should flood out most support requests to a site like stackoverflow and let that community deal with Ruby Developers. The people surrounding you should be DevOps Engineer/Architect types, not Ruby 101 folks. I have no idea why you have endured the fray so long.

 

-Jason Potkanski

Release Manager

 

ACQUITY GROUP  

Part of Accenture Interactive

 

 

 

[Cassiano Leal]

I’ve built a deployment framework around Capistrano and a few plugins. It deploys apps build using 3 different tech stacks — Java/JBoss, PHP and Rails — and it’s been in use for more than a year with little to no maintenance.

I thank you for all the hard work and for how beautifully simple and extensible cap is.

I also second the idea of a core tech-agnostic gem and plugins. Sounds like the way to go and each community could create/contribute plugins for their stacks.

--

Cassiano Leal

http://cassianoleal.com

http://twitter.com/cassianoleal

--
--
* You received this message because you are subscribed to the Google Groups "Capistrano" group.
* To post to this group, send email to capis...@googlegroups.com
* To unsubscribe from this group, send email to capistrano+...@googlegroups.com For more options, visit this group at http://groups.google.com/group/capistrano?hl=en
---
You received this message because you are subscribed to the Google Groups "Capistrano" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capistrano+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ryan Oblak]

Hi Lee,

You can add my name and that of my coworkers to the long list of folks indebted to you for your work on Capistrano.  I agree that this deployment style won't be around forever, but we've used it for the last couple of years with success.  We've found Capistrano to be reliable, logical, and easily extensible.

I might remind you that as far as Capistrano goes, you don't owe anyone anything.  You've put in a tremendous amount of work that has benefitted a large number of people.  You should feel really good about that.  If it's not fun for you any more, well, take care of yourself man.  I would hope that you enjoy writing this software and working on this project.  I'd even prefer not to have Capistrano updates if I knew it was making the guy maintaining it miserable.

From your post it's clear that there are parts of Capistrano that you enjoy working on and parts you don't enjoy.  Do you have to support all of the insane configurations that people put together?  No.  As others have suggested, maybe it's possible to refactor the code base such that the parts you enjoy working on are isolated from the parts you don't (e.g. refactor out a `capistrano-rails` gem).  That way you could let other people work on the stuff they care about and your code can remain pristine.  Or maybe there's some other way.  Whatever you decide, I hope you're able to regain the joy that IMHO is the reason for doing this open source stuff.

All the best,

Ryan

On Sunday, October 6, 2013 10:11:47 AM UTC-7, Lee Hambley wrote:

[Marc Lennox]

FWIW, we have been using capistrano for almost a year now and it's become a critical part of our process.  While I agree that PaaS in some cases is good way to go, I think it will be some time before PaaS will meet the requirements of the majority of applications.

After reading this thread, it triggered me to have a look at capistrano v3.  In short, very awesome improvements, especially for multi-server which we do a lot of.  I took the 1 day hit and converted all our stuff to v3, which actually reduced my cap code (I had a lot of "hacks" to handle multi-server stuff the way I wanted it to work).

Thanks Lee for all your efforts.

[Michał Papis]

Hi Lee,

I'm up to help with support (and possibly maintenance) as capistrano often overlaps with rvm which I'm maintaining right now, already joined #capistrano and started helping.

Thank you for your work on Capistrano!

Cheers,
Michal

[Lee Hambley]

Hi All,

I won't name names individually, but thanks to everyone who shared messages of good will and suggestions and tips for how to make things sustainable.

Here's how it's gone down:

Four days without a computer, worth every penny.

Capistrano 2 is as of now, officially unmaintained. It's been largely unmaintained for ~6 months anyway, and there'll be people who are stuck there, until they can move to v3. In the spirit of sustainable open source, if people want to send pull request with tests and docs for Cap 2 features that don't exist in v3 yet, we'll be happy to include them.

Cap 3 is a bunch of gems now, sshkit, capistrano itself, capistrano-bundler, capistrano-rails, capistrano-rvm (and rbenv, and chruby), and capistrano-maintenance. If anyone wants to host something generally applicable, contact the list, and suggest it, and we'll be happy to let you take the reigns, but host it under Github's Capistrano organisation. We think we're got pretty good coverage of the common cases with those gems that already exist, but I'm sure there'll be other things that turn out to be widely useful. Kir Shatrov and Tom Clements wrote the bulk of the Rails, rvm, rbenv, and other plugins, whilst I wrote sshkit, and most of Capistrano (Tom helped a lot with the core, too) - so the work is quite well split for now. I'm subscribed to all repositories at Github, and will continue to monitor all emails and try and respond in a timely fashion.

I'll be pushing http://harrow.io/ as a "pro" solution for Capistrano. It'll offer a lot more than what the old *webistrano* used to offer, one might think of Harrow as being for build/deployment as what Travis and Jenkins are to testing, and what Github and BitBucket have done for code hosting. Think ..... collaborative, versioned deployment and build environments. There'll be a follow up post to the list, and an attempt to use self-starter to raise the last bit of money we need to get Harrow launched announced shortly.

I'm also dropping a day's client work each week, the real cost to me is in the order of €2000 net/month, but I think it's a real way to continue to do what I want to do, and to push Harrow, which is where I want to be working in order to maintain Capistrano. I'm willing to invest 12-18 months to see if Capistrano, through Harrow can be made sustainable, before I'll pick the client work back up again, and get back to pushing my non-FOSS work.

I've never felt like I could really build up a community of happy, paying customers around a tool that has been free as long as it has existed, but the responses to the mailing list and HackerNews responses to my burnout, and the subsequent fall out have inspired me to give it a shot.

To you all, you have my thanks for your support, and my gratitude for standing my me, and by Capistrano. You have also my thanks for your patience, those of you who've had pull requests closed without answer, or who've caught the short end of the stick when my patience has run low.

Yours,

Lee Hambley

--

http://lee.hambley.name/

+49 (0) 170 298 5667

--