no downtime deploys

[Matt Davies]

Hello everyone

We're running go binaries with supervisor on our linux boxes, some API's.  So far we've been scp'ing the binaries over and restarting supervisor, that worked.

Then the apps started to get busy, and scp was failing as the file was busy, sounds sensible.

So the solution was to stop the app, then copy the file over, then start the app.  Great.

apart from one thing

Another app was calling our API at the time and erred, pain.

Can anyone give me some ideas about how to do zero downtime deploys with go?

Any help, tips, links or general abuse is greatly welcomed.

Matt

[Shawn Milochik]

Check these out. Also, search this list. This has been discussed before.

https://github.com/facebookgo/grace

https://www.reddit.com/r/golang/comments/30nnq9/golang_graceful_restart_best_approach/

[Matt Davies]

Thanks Shawn, will do

[Shawn Milochik]

On Thu, Apr 28, 2016 at 11:43 AM, Matt Davies <ton...@gmail.com> wrote:

Thanks Shawn, will do

You're welcome. And although it isn't a perfect solution, you can always scp the binary to a different filename then pull a quick change once it's uploaded.

Maybe even have a script that looks for the presence of the "newVersion" file and have it "pkill currentVersion; mv newVersion currentVersion; ./currentVersion."

It's hacky, but a bit better than stopping the program before the upload.

 

[Zlatko Čalušić]

Here's what works well for me:

import         "gopkg.in/fsnotify.v1"

func exitOnChange() {
        watcher, err := fsnotify.NewWatcher()
        handle(err)
        defer watcher.Close()

        err = watcher.Add(os.Args[0])
        handle(err)

        <-watcher.Events

        log.Println("Executable changed, exiting...")
        os.Exit(0)
}

func main() {
        go exitOnChange()
...

and

deploy: dependency
        rsync -az $(GOPATH)/bin/binary username@remote:location

in the Makefile.

The final piece of the puzzle is systemd.service option, to restart the service even if it exits on its own:

[Service]
Type=simple
Restart=always
...

What happens there is that I run make deploy, the binary gets built (if needed) and rsynced to the remote host. As rsync uses temporary file during the transfer, you'll avoid "text file busy" errors. In the end, when the binary gets replaced, exitOnChange() goroutine will notice that and exit the service, systemd will notice that fact and immediately restart it. Of course, if you prefer supervisor, you can use it instead of systemd.

This simple setup makes development/deployment of system services a breeze. Edit, commit, make deploy, go have a cup of coffee. :)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Zlatko

[Konstantin Khomoutov]

On Thu, 28 Apr 2016 11:48:58 -0400
Shawn Milochik <sh...@milochik.com> wrote:

[...]

> You're welcome. And although it isn't a perfect solution, you can
> always scp the binary to a different filename then pull a quick
> change once it's uploaded.
>
> Maybe even have a script that looks for the presence of the
> "newVersion" file and have it "pkill currentVersion; mv newVersion
> currentVersion; ./currentVersion."
>
> It's hacky, but a bit better than stopping the program before the
> upload.

On a POSIX filesystem, when both files are on the same physical
filesystem the correct approach would be the reverse:

mv new current
pkill current

The `mv` call would result to atomic rename() which would nuke "current"
but make no damage to its running copy (which would still have a handle
on its on-disk data).

[Nick Craig-Wood]

On 28/04/16 16:37, Matt Davies wrote:
> We're running go binaries with supervisor on our linux boxes, some
> API's. So far we've been scp'ing the binaries over and restarting
> supervisor, that worked.
>
> Then the apps started to get busy, and scp was failing as the file was
> busy, sounds sensible.

Don't use scp to deploy binaries - a lesson I learnt the hard way!

scp overwrites the existing file which at best causes the "Text file
busy" error you've seen, or on old linuxes it used to crash your app
very badly if you did this to shared libraries.

Use rsync for deployment as by default it will write into a separate
file then use mv to move it over in an atomic operation after successful
transfer.

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick