Re: Is anyone using chef to setup / deploy go web servers?

[Nate Finch]

Is there anything Go-specific that needs to be done? Once it's built, a go application is just an executable file.  Handle as you would any other executable (I don't know chef, so I don't know how it handles that stuff, but I presume there must be some handling for that).

On Tuesday, April 16, 2013 4:23:51 PM UTC-4, ma...@crowdmob.com wrote:

Hey there,

I've been starting to play around with Amazon OpsWorks, and Chef is the tool they use for setting up and deploying ec2 servers.

Before I go down the rabbit hole of writing my own Chef scripts to deploy my go web application, does anyone have any examples?  I only saw examples of Chef recipes to install the go binary, none for deployments.

Any help appreciated!

[Bobby Powers]

At my previous company, we had a webservice written in go that handled hundreds of thousands of (small) requests a day, and it was deployed & managed with chef.  Our process looked like this (and was the same process we used for our python code):

• build service as static binary & run tests via jenkins
• if tests pass, package binary & related files up int a deb & add it to our private debian repo
• to redeploy - run chef-client on all nodes with role[our_webservice]
• Chef client runs `apt-get update`
• chef-client sees there is a new version of our_webservice-$node_type (alpha, staging, prod, etc)
• chef-client apt-get installs the update
• chef-cleint then does a service our_webservice restart

Simple, and gets us 95% of the way there (the last 5% would be verifying the newly installed package seems to work right, and making sure no client requests are dropped.  but the startup time for a statically linked go binary is so insanely fast it wasn't a practical problem for us.  also we had the 2 web services behind an ELB, so there was some redundancy there).

So, to reiterate:

test & build a deb package of your go application on jenkins

have chef install stuff via deb packages

use upstart on ubuntu or systemd.

This makes your life much simpler compared to the usual chef 'compile everything on the web server itself' mentality I saw in a lot of the standard chef cookbooks

yours,

Bobby

On Tue, Apr 16, 2013 at 4:23 PM, <ma...@crowdmob.com> wrote:

Hey there,

I've been starting to play around with Amazon OpsWorks, and Chef is the tool they use for setting up and deploying ec2 servers.

Before I go down the rabbit hole of writing my own Chef scripts to deploy my go web application, does anyone have any examples?  I only saw examples of Chef recipes to install the go binary, none for deployments.

Any help appreciated!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Matthew Moore]

There is a related question, which I'm also asking!

Do you guys have a build/deploy process for go web apps?

I assume there is typically the following steps, though not necessarily in this order:

- clone the latest version of the app from some sort of SCM

- download dependencies

- build the app

- run tests

- shutdown previous app version

- start up new app version

It looks like Bobby just replied as I'm writing this, so let me take a look at that :P

Best,

Matt

--

Matthew Moore

Co-Founder & CTO, CrowdMob Inc.

Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!  ma...@crowdmob.com

[Matthew Moore]

Hey Bobby,

Awesome info!  Did you ever have open source versions of those cookbooks/recipes, or is there something you could share with me (or this group)?

Best,

Matt

--

Matthew Moore

Co-Founder & CTO, CrowdMob Inc.

Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!  ma...@crowdmob.com

[Bobby Powers]

We did not open source any of this work, but the cookbooks are trivial.  The recipe for the go web server is:

directory node['our_webservice']['config_dir'] do

  owner 'shared-user'

  group 'shared-user'

  mode 0755

  action :create

  recursive true

end

template node[:our_webservice][:config_file] do

  source "our_webservice.conf.erb"

  mode 0644

  notifies :restart, "service[our_webservice]", :delayed

end

package node['our_webservice']['package_name'] do

  action :upgrade

  notifies :restart, "service[our_webservice]", :delayed

end

service "our_webservice" do

  action :start

  provider Chef::Provider::Service::Upstart

end

All of the useful bits are in the packaging script.  We used a tool called FPM to do the actual deb package creation.  In sort:

1) create a tempdir and give it the following structure:

$TMPDIR/usr/bin/our_webservice

$TMPDIR/etc/init/our_webservice.conf     # this is the upstart job definition

$TMPDIR/etc/logrotate.d/our_webservice.conf

etc...

And then you can basically point FPM at that temp dir and say 'make me a deb that installs this'.  I've done real debian packaging, this is easier, especially under a deadline.  The TMPDIR creation & fpm invocation & some other sanity checks (like if go is actually creating static binaries) are all in a 'package.bash' file, for easy version controlling.  This means you can have your jenkins execute a shell script target that looks like:

go build

go test

./package.bash

or some such (jenkins halts the job at the first command that exits non-zero, so if the build or tests fail, it won't package things).

Hope this helps.

[Krzysztof Kowalczyk]

I do. I use Fabric, because I know Python and not Ruby, but there are equivalent Ruby tools.

In spirit it's what you describe, but details are slightly different.

I think the script https://github.com/kjk/apptranslator/blob/master/fabfile.py is pretty self-explanatory.

To deploy, I just type: fab deploy.

-- kjk

[Matthew Moore]

Hey guys, I just got a chef script working for us, with a lot of your feedback, which I wanted to make sure was re-shared for future searchers of this group.

I'm also happy to take feedback on it.  The cookbook is here, with a fleshed-out readme:

https://github.com/crowdmob/chef-golang-web-server-cookbook

I've confirmed it's working with Amazon OpsWorks, too :)

Thanks again for your thoughts & help!

Best,

Matt

--

Matthew Moore

Co-Founder & CTO, CrowdMob Inc.

Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!  ma...@crowdmob.com