Keep serving my Polymer app when building it

[David Notik]

Hi all!

It's time I address this issue: When deploying changes to production, I currently git pull and pub build on the server. (I also restart my server if any server stuff has changed, but that's irrelevant here.) The pub build kills everything in build/web and builds afresh. This means that while I'm building (a long-ish process), the assets that my server needs to serve the app are not there, and so new requests see a 404. Basically, my app is down while I pub build.

I've long thought pub build should by default build to a tmp directory, and then when done do a copy to the build/web folder. But that would still result in some downtime, as copying multiple files isn't atomic, right? If that were a good solution, I guess I could write a (grunt/gulp/pub/sh?) script that did similar?

Another option is an in-memory cache of the files – so when the server starts, read in the built assets and then if they're not found use the in-memory versions. This seems like trouble, given that there a bunch of assets (http://d.pr/i/1j2Xy) including some static assets (though I could get those into Cloud Storage, even though some things like generic.css and mustache templates are updated here and there and I'd need to update then in GCS every time).

Keeping in mind I'm no Google, I wonder what your suggestions are?

Thank you!

--D

cc: web@, cloud@

ᐧ

[Joel Trottier-Hébert]

I think you could just serve your app from another directory, so that you call pub build, and then have a script to move all the generated files to the served directory. That would fix all this.

--
You received this message because you are subscribed to the Google Groups "Dart Server-side and Cloud Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud+un...@dartlang.org.
Visit this group at https://groups.google.com/a/dartlang.org/group/cloud/.

[David Notik]

I noted that as a possible solution. But AFAIK even then there'd be some downtime (if we were getting lots of requests) as moving files isn't atomic.

ᐧ

[Joel Trottier-Hébert]

well then you need to have replication I think. There's nothing such as atomic file moves. That, or have 2 served directories, and when you know your new app version is ready, switch the directory to serve from your web server (to the new app).

[David Notik]

Thank you! What do you mean by this:

some way of updating the folder that your server is serving from on the fly

So, like versioned folder names, and store the latest version number in the database, and new requests look for that folder? 

On Friday, January 22, 2016 at 10:19:54 AM UTC-5, Jacob Macdonald wrote:

I don't think pub build is really meant to be used in quite this way, for this reason among others (what if you have a build failure? Now your app is down for good until you can fix it).

What I would do instead is deploy from some other folder, maybe deploy/web. That allows you to test the contents of web/build before actually deploying it.

To address the downtime problem, the easiest solution is to instead of copying those files directly to the deploy/web folder, copy them to some other folder (maybe deploy/web_new). Once the copy is done, simply rename deploy/web to deploy/web_old and then rename deploy/web_new to deploy/web. Renaming the folders in this way (using a script) should reduce the downtime you see to almost zero when swapping in the new app.

If you want to get fancier/safer you could have some way up updating the folder that your server is  serving from on the fly, which would result in zero downtime and allow  you to have versioned folder names so you could easily roll back. You can also do staged rollouts etc this way.

- Jake

[Jacob Macdonald]

You could do it any number of ways, but yes storing the latest version number (or just the name of the current folder to serve from) is one reasonable way. You could also just have some sort of config file which gets loaded when the server starts up, and some sort of endpoint that you can ping when you want it to reload its config.

[Hack Ster]

>>> new requests see  404. Basically, my app is down while I pub build

>> I think you could just serve your app from another directory, so that you call pub build, and then have a script to move all the generated files to the served directory. That would fix all this.

> I noted that as a possible solution. But AFAIK even then there'd be some downtime (if we were getting lots of requests) as moving files isn't atomic.

- a complete solution for this requires an implementation in the web server process

- there needs to be a time when both the old version of the application files and the new version of the application files are in place

- at that time the web server can stop serving all files from the old version and start serving all files from the new version

- you can come close to a complete solution without such web server functionality if you use load balancing

- instead of 1 web server process entirely responsible for requests , you have 2 and occasionally 3 web server process

### for this example

1. (1 load balancer) ; a web server process
2. (2 pub serve) ; a web server process
3. (3 pub serve) ; a web server process

- (1 load balancer) has the simple responsibility of handing off requests to some backend web server that it knows about

- (1 load balancer) knows about 2 backends servers, (2 pub serve) and (3 pub serve)

- (1 load balancer) will typically distribute requests to both (2 pub serve)-ACTIVATED and (3 pub serve)-ACTIVATED

- (1 load balancer) will only distribute requests to only (2 pub serve)-ACTIVATED if (3 pub serve)-NONACTIVATED

### deploying a new version of your web site

- the problem of deploying a new version of a website while maintaining uninterrupted service can be addressed if you have a configuration of servers similar to the example above

- when you want to deploy a new version of your application you follow this sequence :

  1. use [2 pub serve] make sure it is ON
    - status (2 pub serve)-ACTIVATED-OLD (3 pub serve)-ACTIVATED-OLD
  2. use [3 pub serve] use `kill server ;`
    - status (2 pub serve)-ACTIVATED-OLD (3 pub serve)-NONACTIVATED-OLD
  3. use [3 pub serve] use `git pull ;` `pub get ;` `pub build ;`
    - status (2 pub serve)-ACTIVATED-OLD (3 pub serve)-NONACTIVATED-UPGRADING
  4. use [3 pub serve] use `pub serve ;`
    - status (2 pub serve)-ACTIVATED-OLD (3 pub serve)-ACTIVATED-NEW
  5. use [2 pub serve] use `kill server ;`
    - status (2 pub serve)-NONACTIVATED-OLD (3 pub serve)-ACTIVATED-NEW
  6. use [2 pub serve] use `git pull ;` `pub get ;` `pub build ;`
    - status (2 pub serve)-NONACTIVATED-UPGRADING (3 pub serve)-ACTIVATED-NEW
  7. use [2 pub serve] use `pub serve ;`
    - status (2 pub serve)-ACTIVATED-NEW (3 pub serve)-ACTIVATED-NEW
  8. use [3 pub serve] use `kill server`
    - status (2 pub serve)-ACTIVATED-NEW (3 pub serve)-NONACTIVATED-NEW
    - do this only if you do not want to have a (3 pub serve)-ACTIVATED on an ongoing basis

- note that you should minimize the time spent in the state between step 4. and step 5.

- a danger in this scenario occurs during the state described between step 4. and step 5.
  - a given session (a session is a single client spanning multiple requests) might get routed to the new version , _and then back to the old version_ on a subsequent request

- if the load balancer uses sticky sessions (requests belonging to a given session are preferentially routed to the backend server used for the previous request) that helps to mitigate and almost eliminate the risk

-- - - hackst8r -- - -

[Warren Strange]

Another lateral thought....

Have you considered delivering your app containerized using the Dart docker base image?

You can use Kubernetes on GKE (container engine) to perform rolling upgrades. It will spin down one version of your service and spin up the new version. 

[David Notik]

Thanks for your recommendations, everyone! Some great ideas here.

I went w/ a simple script that swaps directories as Jake had suggested (ideas for improvements welcome):

https://gist.github.com/davenotik/d8e0914146d9d33f4046

Later, maybe Docker.

--D