auto-conscript

[Paul Phillips]

It's a little dream of mine to have reliable infrastructure which does
what conscript and the other elements of the repo below are doing. In
practice I always spend more time (a lot more time) trying to get all
the ducks in the same row with all the different versions and moving
targets and still-evolving software than I ever save relative to
slapping together yet another one-off. In case others share my dream, I
offer a small running start on today's ducks.

https://github.com/paulp/xsbtscript

[Jon-Anders Teigen]

I am speachless... the ducks, the awesomeness...

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
>

[Ray Racine]

In a very similar vein, one can use sbt-launcher as a deployment system.

We've been using the sbt-launcher for application deployment for a long time and it has worked out really well.

Process outline:

App Builds -

• SBT builds and publishes to an internal Nexus / Maven repository with 2 repos: snapshots and releases.
• For third party libs and the scala compiler / lib et al. we have a small ivy/ant script which fetches things down and loads them into the repo.
• All developer boxes point to the repo.
• As release are fully "locked" concerning dependencies.

App Deploys -

• On the target deployment server all you need is Java installed, sbt-launcher.jar, a small shell script, and the sbt launcher config file which points _only_ to the Nexus Release repo for PROD releases.
• We use "Java Service Wrapper" (very nice) as the "small shell script".
• Service Wrapper gives you ./bin/myapp start|stop|restart|status out of the box.
• This is perfect to put into your rc.d "services" for auto starting your app on failure etc..
• Service Wrapper give you a "ping" nanny monitor process for free as well.  It will auto restart a dead/hung app.
• To manually deploy a new version of the app:
• Edit sbt-launch config file to new version.
• ./bin/myapp restart
• Since the full history of releases are in the Nexus repo to rollback to any previous version, just change the version number and restart.

Of course manually editing a config file is sometimes a bridge too far.  You enterprise people know what I'm talking about.

Fool proof and simple app deploys:

• We gave our "ops" guys a script.
• That script uses a ssh key with a "deployer" login.  No private key you can't deploy to prod.  The ssh session for that key, is locked to a command (tiny shell script) which sets the version in the sbt-launcher config.
• Example:  To deploy a new release acros all nodes: $ deploy all 1.3.

Simple, effective and fairly bullet-proof.

[Jason Zaugg]

Am Mittwoch, 29. Juni 2011 23:26:51 UTC+2 schrieb Ray Racine:

In a very similar vein, one can use sbt-launcher as a deployment system.

We've been using the sbt-launcher for application deployment for a long time and it has worked out really well.

Can you elaborate? What command line and config file(s) do you use to get sbt-launch.jar to run your application?

-jason

[Ray Racine]

A requested loooooong elaboration.

Assumes one has looked at Java Service Wrapper and somewhat familiar with it's basic workings.

As indicated SBT has already published my app into a Maven / Nexus internal repository.  This is the binary artifact repo which also contains imported third party libraries.

Say on my target PROD deploy node I have roughly the following directory tree with the "bootstrap" files itemized below which we provision them onto a virgin box "automagically" (details elided).

/myapp

        /bin

       /conf

       /bin

       /lib

In my /bin directory are 2 files.  wrapper and myapp

• wrapper: is a exe used by Java Service Wrapper.  Window/Linux/Mac 32/64 bit native.
• myapp: Is a renamed out of the box bash script again from Java Service Wrapper.

So both of these are "generic", application agnotic and are never touched (nor is sbt-launch.jar).

In my conf directory: myapp.conf, myapp.launch and wrapper.conf

• myapp.conf is a standard "configgy-like" configuration file for your app.
• myapp.launch is the sbt-launcher config file.                    
• wrapper.conf the Java Service Wrapper configuration file.  

./lib has two jars

• sbt-launcher.jar  #generic
• wrapper.jar         #generic

myapp.launch contents are a pretty obvious sbt-launch configuration.

[scala]

  version: 2.9.0-1

[app]

  org: odp

  name: myapp

  version: 2.0.86

  class: com.myapp.MainXsbti

  components: myapp

  cross-versioned: true

  resources: ./conf

[repositories]

   # URL to my internal Nexus repo which contains EVERYTHING necessary.

[boot]

 directory: ./lib

[log]

 level: info

Mark H added the resources: property for me a while back.  This is appended to the classpath of the class loader minted by sbt-launch lto run/aunch myapp's main().  This resources myapp to find "resources" such as logback.xml and myapp.conf the app's configgy file.

wrapper.conf is Java Service Wrappers standard conf file. In general all your JVM command line arg stuff is in there, heap sizes, ping monitoring etc.

So that is a base bootstrap.  One could imagine say a naive ssh/scp + script to create the dir tree and copy over the few files.  That would reduce the bootstrap to Linux box with Java installed, a generic "deploy" account + ssh key pair + 1 bash script.   We do something a bit more sophisticated, but you get the idea.  Just get the above over 1-time to provision a virgin node.  Paul's conscript approach works as well.  Many cats with skins.

To manually start the app (again never manually start PROD but automate the invoking of the below).

• Edit myapp.launch to the desired application version to deploy, say 3.4.34
• ./bin/myapp start

Whew... pauses to take a breather.

What happens...

• generic Service Wrapper bash shell "myapp" is run with "start" arg.
• this runs the wrapper binary.
• which reads all the ./conf/wrapper.conf file for JVM config stuff etc.
• which launches a "nanny" jvm and a sbt-launcher jvm which ultimately runs myapp
• sbt-launcher now ...
• pulls down all necessary jars, compiler lib, 3rd party, the whole 9 yards from the Nexus repo into ./lib
• and finally sbt-launch puts together a classloader with all the newly downloaded app jars and deps and runs myapp's main ().
• since its all local copied once a version is deployed, even if you "lose" the Nexus repository any local cached myapp version can be run.

All of this is out-of-box functionality driven by config files.

Before this gets tooooo looooong.  Your main is something like this.

Main () is a nested doll of Service Wrapper's main() wrapping, SBT launchers main() wrapping MyApp's main().

object Main {

---

 // allows for standard command line launch 

  def main (args: Array[String]): Unit =  {  

    Init.initialize (args)

  }

}

class MainXsbti extends xsbti.AppMain with Logging {

 ....

  def run (launchConfig: xsbti.AppConfiguration) = {

    class Exit (val code: Int) extends xsbti.Exit

   

    ... 

    Main.main (launchConfig.arguments)

    

   ...

    new Exit (0)

  }

}

Java Service Wrapper offers various levels of main() wrappers as well which offer tighter coupling to the monitoring nanny etc.  So you could get a nested doll of MainWapper ( MainXsbti (MainMyApp))).  Depends how fancy your tap dance is.

Ok that's the quick and dirty.  We use paxos to ensure we don't get split-brain deployments, a guaranteed oracle of truth on what the "true" version a node should be running (if you are not running the correct version you don't join the cluster).  Somewhat along the lines of say using ZooKeeper to hold the correct app version and other config values.  One could eliminate most of (all) the config files.

Ray

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/4_JYgFa3ByIJ.

[eugene yokota]

Also, [conscript][1] can automate much of this process for github-hosted tools.

  $ cs eed3si9n/scalaxb

And it will download launchconfig, and create a shell script in ~/bin that invokes the sbt-launcher.

  [1]: https://github.com/n8han/conscript/

-eugene

On Wed, Jun 29, 2011 at 6:17 PM, Jason Zaugg <jza...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/4_JYgFa3ByIJ.

[Ray Racine]

I'll put in a quick plug for "Java Service Wrapper" again here.  Super well done project.  Full commercial support.

Key wins are:

• Standardized script(s).
• Cross platform support.
• Soft link the script right into your sysv init rc.d for auto app starting on node failure (power loss etc)
• Nanny process which will kill and restart a hung JVM
• Tons more.

Thanks for the conscript github link.  I'll take a look at it.

[Ray Racine]

And to actually answer JZ's question!!

Java Service Wrappers standard wrapper.conf file is where you tell it what java program to launch.  In this case that is sbt-launch. Not too dissimilar what sbt-launch does at the next level done.

$ grep sbt conf/wrapper.conf 

wrapper.java.classpath.3=./lib/sbt-launch.jar

wrapper.java.additional.1=-Dsbt.boot.properties=./conf/myapp.launch

wrapper.app.parameter.1=xsbt.boot.Boot

On Wed, Jun 29, 2011 at 6:17 PM, Jason Zaugg <jza...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/4_JYgFa3ByIJ.

[Ray Racine]

Sigh .. to plant one more seed ... one more cool thing that people may not know is that both Service Wrapper and sbt-launcher can do a self bootstrap.  i.e. They can download new versions of themselves and then reboot into their new versions.  Which means you can do this at the app level.  This was a cool and and key feature we leveraged. Self bootstrapping turtles all the way down....

So roughly, all app nodes open an HTTP admin port.  

Naively

PUT - curl  http://myapp.com/admin/deploy/version/1.2.3

POST - curl htpp://myapp.com/admin/restart

• App cluster updates the sbt-launch version config with the first PUT after a cluster paxos consensus agreement.  You can PUT the new version to any node,  if and only if the version was consensus agreed to by the cluster as a whole does a node "learn" the new version to record in the config file.
•  The second POST causes the app to do a shutdown and then uses sbt's "reboot" api call to reboot itself and come up on the new version.

[Mark Harrah]

On Wed, 29 Jun 2011 21:01:21 -0400
Ray Racine <ray.r...@gmail.com> wrote:

> Sigh .. to plant one more seed ...

Although I vaguely knew what you were up to, it is always interesting to hear the details. Thanks!

-Mark

[Jason Zaugg]

Am Donnerstag, 30. Juni 2011 03:01:21 UTC+2 schrieb Ray Racine:

Sigh .. to plant one more seed ... one more cool thing that people may not know is that both Service Wrapper and sbt-launcher can do a self bootstrap.  i.e. They can download new versions of themselves and then reboot into their new versions.  Which means you can do this at the app level.  This was a cool and and key feature we leveraged. Self bootstrapping turtles all the way down....

So roughly, all app nodes open an HTTP admin port.  

Naively

PUT - curl  http://myapp.com/admin/deploy/version/1.2.3

POST - curl htpp://myapp.com/admin/restart

• App cluster updates the sbt-launch version config with the first PUT after a cluster paxos consensus agreement.  You can PUT the new version to any node,  if and only if the version was consensus agreed to by the cluster as a whole does a node "learn" the new version to record in the config file.
•  The second POST causes the app to do a shutdown and then uses sbt's "reboot" api call to reboot itself and come up on the new version.

Thanks for the detailed explanation, Ray. I've used JSW in the past and found it to be very powerful; although it doesn't appeal to the ops guys I work with at the moment. But the choice of the wrapper is a separate decision from the choice to use the launcher, which is really appealing. 

Currently we deploy apps as zip artifacts to nexus, with all the deps bundled and have scripts to deploy that, which is already a big step forward from the old method of manually updating a repo of the binary layout of the app in perforce. But it's inspiring to see it done better.

-jason