Creating Release Distributions

[Eric Daugherty]

First I want to thank Tim and Mark for reaching out to me via my Blog
post about Maven and Ant (http://www.ericdaugherty.com/blog/2009/04/
maven-versus-ant.html). I was not aware of sbt before their posts and
not that I've played with it a bit I'm interested in exploring it
further.

As I mentioned in the above posts/comments, the only block I have from
using sbt exclusively to build my project is the packaging of the
releases. I've used Ant for a long time to build my releases and like
them a 'certain way'. You can browse my project structure here:
http://code.google.com/p/itunesexport-scala/source/browse/#svn/trunk

Essentially my src directory conforms to the maven default, and I have
a lib and doc directory for dependencies and various text files
(Release Notes, License, etc). I realize sbt uses the lib_managed
directory for dependencies as well.

Anyway, my Ant task for releases looks like this (there are some
property references here but I think they are self-describing):

<target name="release" depends="jar, docs">
<mkdir dir="${build.release.dir}" />
<zip destfile="${build.release.dir}\iTunesExportScala-$
{version}.zip">
<fileset dir="${build.jar.dir}" includes="**/*" />
<fileset dir="docs" />
</zip>
<zip destfile="${build.release.dir}\iTunesExportScala-src-$
{version}.zip">
<fileset dir=".">
<include name="docs/**/*" />
<include name="lib/**/*" />
<include name="src/**/*" />
<include name="*.pom" />
<include name="*.xml" />
</fileset>
</zip>
</target>

You can see the output zips here: http://www.ericdaugherty.com/dev/itunesexport/scala/

This creates two zip files. One is the binary release that contains
the executable jar and the supporting jar(s) and the contents of the
docs dir. The second jar is the entire source directory (or the stuff
actually version controlled).

I've never been able to figure out how to get Maven (or sbt) to easily
create these types of releases. Any hints on how I can do this in
sbt?

I also have a secondary question on the best practices for handling
the sbt build tool in scm. I have a bat file in my project root as
well as the sbt jar to run the build. What is the best practice,
should the jar be installed on the machine (ie viewed as common) or
contained in the project structure?

Eric

[Mark Harrah]

Hi Eric,

On 5/8/09, Eric Daugherty <edaug...@gmail.com> wrote:
>
> First I want to thank Tim and Mark for reaching out to me via my Blog
> post about Maven and Ant (http://www.ericdaugherty.com/blog/2009/04/
> maven-versus-ant.html). I was not aware of sbt before their posts and
> not that I've played with it a bit I'm interested in exploring it
> further.

Thanks for following up!

Thanks for the link and the build snippet. They make it clear what you
are trying to do.

> This creates two zip files. One is the binary release that contains
> the executable jar and the supporting jar(s) and the contents of the
> docs dir. The second jar is the entire source directory (or the stuff
> actually version controlled).
>
> I've never been able to figure out how to get Maven (or sbt) to easily
> create these types of releases. Any hints on how I can do this in
> sbt?

You can create the source zip using the `package-project` action. I
think Maven has something similar with the assembly plugin.
Currently, there are two directories that `package-project` includes
that it should not (project/boot and project/build/target), but I will
fix this.

You can use `zipTask` for the second zip. For example, in your
project definition:

def distName = "iTunesExportScala-" + version + ".zip"
def distPaths= ((outputPath ##) / defaultJarName) +++ ( ("docs" ##) ** "*" )
lazy val dist = zipTask(distPaths, outputPath, distName) dependsOn(`package`)

The first line constructs the name of the zip. The second selects
what goes in the zip. `path ##` is like <fileset dir="path"/>, ** is
descendent-or-self, and +++ is the union of the paths. (See the Paths
page on the wiki for details).

The third line calls `zipTask` to create the task and makes it depend
on `package`, which will first create the binary jar.

> I also have a secondary question on the best practices for handling
> the sbt build tool in scm. I have a bat file in my project root as
> well as the sbt jar to run the build. What is the best practice,
> should the jar be installed on the machine (ie viewed as common) or
> contained in the project structure?

I'll take this question to a separate thread for visibility...

Thanks,
Mark

[Mark Harrah]

Hi Eric,

As I mentioned in my other reply, I brought this question to a
separate thread for other opinions on this.

> I also have a secondary question on the best practices for handling
> the sbt build tool in scm. I have a bat file in my project root as
> well as the sbt jar to run the build. What is the best practice,
> should the jar be installed on the machine (ie viewed as common) or
> contained in the project structure?

There are four directories I would definitely exclude from scm:
project/boot, project/build/target, target, and lib_managed. I'd
exclude these because they are generated from the rest of the project.

As for the sbt launcher jar, I could see it going either way. Perhaps
some others have opinions on this?

-Mark

[Eric Daugherty]

Your instructions on creating a dist task that created the binary
release was very helpful. I made a small change to exclude the .svn
directory in docs:

def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
** "*" ).descendentsExcept("*", ".svn")

I also needed to include scala-library.jar in the distribution as a
runtime dependency. It is not in lib-managed (I assume it is an
implied dependency) so I grabbed it from the project dir. It seems
like a hack. Is there a 'better' way?

def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
** "*" ).descendentsExcept("*", ".svn") +++ ("project" / "boot" /
"scala-2.7.4" / "lib" ##) ** "scala-library.jar"

Pending a better solution, this successfully solved my release zip
issue, thanks!

For the source issue, the package-project task by default includes a
bunch of stuff I don't want (output directory from my IDE, IDE project
files, other non-source controlled files, etc). I wasn't sure how to
configure it so I brute forced it. The new config combined with what
you suggested is:

def srcDistName = "iTunesExportScala-src-" + version + ".zip"
def srcDistPaths = (("src") ** "*" +++ (("docs") ** "*" ) +++
("project") ** ("*.properties") +++ ("project" / "build" / "src") **
("*")).descendentsExcept("*", ".svn")
lazy val distSrc = zipTask(srcDistPaths, outputPath, srcDistName)
dependsOn(`package`)

def distName = "iTunesExportScala-" + version + ".zip"

def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
** "*" ).descendentsExcept("*", ".svn") +++ ("project" / "boot" /
"scala-2.7.4" / "lib" ##) ** "scala-library.jar"

lazy val dist = zipTask(distPaths, outputPath, distName) dependsOn

(distSrc)

It meets my needs. I may add code to include the sbt jar in the root
and a bat file or something depending on how I decide to handle that.
I'm inclined to include the sbt jar to make it stand-alone but I'm not
sure yet.

I may have a few other issues but they are only semi-related so I'll
create new threads for those once I verify them. Thanks for your
input, and I would appreciate any further input on how to improve what
I have.

Eric

On May 8, 6:41 pm, Mark Harrah <dmhar...@gmail.com> wrote:
> Hi Eric,
>

[Mark Harrah]

Hi Eric,

On Friday 08 May 2009, Eric Daugherty wrote:
> Your instructions on creating a dist task that created the binary
> release was very helpful. I made a small change to exclude the .svn
> directory in docs:
>
> def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
> ** "*" ).descendentsExcept("*", ".svn")

Yes, sorry about that. In fact there are two related methods that are useful
here: descendents and defaultExcludes. defaultExcludes is the default filter
used to exclude hidden files like .svn directories. It can be overridden to
add things to exclude. descendents(path, include) is the same as
path.descendentsExcept(include, defaultExcludes). So the docs part becomes:

descendents("docs" ##, "*")

> I also needed to include scala-library.jar in the distribution as a
> runtime dependency. It is not in lib-managed (I assume it is an
> implied dependency) so I grabbed it from the project dir. It seems
> like a hack. Is there a 'better' way?

There are two ways to do this 'better':
* The scalaJars method returns the locations of the scala-compiler.jar and
scala-library.jar that your code depends on (scala-compiler.jar isn't
included if your code doesn't depend on it).
* mainCompileConditional.analysis.allExternals provides Files for all jars
that your main classes depend on, including the scala jars above. It also
includes rt.jar/classes.jar, which can be filtered by checking if a jar is in
the project. Something like:

externals.filter(jar => Path.relativize(rootProject.info.projectPath,
jar).isDefined)

The above methods are only valid after `compile` has run because the
information is extracted from compilation. This is fine in your case, of
course.

>
> def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
> ** "*" ).descendentsExcept("*", ".svn") +++ ("project" / "boot" /
> "scala-2.7.4" / "lib" ##) ** "scala-library.jar"

(same comment about descendentsExcept)

> Pending a better solution, this successfully solved my release zip
> issue, thanks!
>
> For the source issue, the package-project task by default includes a
> bunch of stuff I don't want (output directory from my IDE, IDE project
> files, other non-source controlled files, etc). I wasn't sure how to
> configure it so I brute forced it. The new config combined with what
> you suggested is:

There is a way to do this now, but needs to be fixed, so it will have to wait
for the next release. (There is a method that defines the excluded paths for
package-project).

> def srcDistName = "iTunesExportScala-src-" + version + ".zip"
> def srcDistPaths = (("src") ** "*" +++ (("docs") ** "*" ) +++
> ("project") ** ("*.properties") +++ ("project" / "build" / "src") **
> ("*")).descendentsExcept("*", ".svn")
> lazy val distSrc = zipTask(srcDistPaths, outputPath, srcDistName)
> dependsOn(`package`)

You can do ** on any PathFinder, so you can coalesce some of the above:

("src" +++ "docs") ** "*"

although you probably want to use descendents here too, so then maybe:

descendents("src" +++ "docs" +++ ("project" / "build" / "src"), "*")

> def distName = "iTunesExportScala-" + version + ".zip"
> def distPaths = ((outputPath ##) / defaultJarName) +++ ( ("docs" ##)
> ** "*" ).descendentsExcept("*", ".svn") +++ ("project" / "boot" /
> "scala-2.7.4" / "lib" ##) ** "scala-library.jar"
> lazy val dist = zipTask(distPaths, outputPath, distName) dependsOn
> (distSrc)
>
> It meets my needs. I may add code to include the sbt jar in the root
> and a bat file or something depending on how I decide to handle that.
> I'm inclined to include the sbt jar to make it stand-alone but I'm not
> sure yet.
>
> I may have a few other issues but they are only semi-related so I'll
> create new threads for those once I verify them. Thanks for your
> input, and I would appreciate any further input on how to improve what
> I have.

Thanks for your feedback!

-Mark

[Harshad RJ]

On Sat, May 9, 2009 at 5:31 AM, Mark Harrah <dmha...@gmail.com> wrote:

As for the sbt launcher jar, I could see it going either way.  Perhaps
some others have opinions on this?

I like to check in the core version of sbt into the SCM, since other team members are not scala/sbt enabled and most distributions haven't started packaging sbt yet.

The core version is more deterministic than the launcher and IMO doesn't scare scala n00bs away.

--
Harshad RJ
http://hrj.wikidot.com

[Eric Daugherty]

Thanks I mostly understood everything. I struggled with how to
include the scala-library.jar still. I understand the concept but am
not sure how to get it into practice.

On May 8, 10:10 pm, Mark Harrah <dmhar...@gmail.com> wrote:
...

> > I also needed to include scala-library.jar in the distribution as a
> > runtime dependency.  It is not in lib-managed (I assume it is an
> > implied dependency) so I grabbed it from the project dir.  It seems
> > like a hack.  Is there a 'better' way?
>
> There are two ways to do this 'better':
>  * The scalaJars method returns the locations of the scala-compiler.jar and
> scala-library.jar that your code depends on (scala-compiler.jar isn't
> included if your code doesn't depend on it).
>   * mainCompileConditional.analysis.allExternals provides Files for all jars
> that your main classes depend on, including the scala jars above. It also
> includes rt.jar/classes.jar, which can be filtered by checking if a jar is in
> the project.  Something like:
>
> externals.filter(jar => Path.relativize(rootProject.info.projectPath,
> jar).isDefined)
>
> The above methods are only valid after `compile` has run because the
> information is extracted from compilation.  This is fine in your case, of
> course.
>

I'm not sure how to use scalaJars in the config. It is an Iterable
[File], how do I get that into the Path?

[Mark Harrah]

You had the right idea:

Path.lazyPathFinder( scalaJars.map(Path.fromFile) )

PathFinders are re-evaluated each call to `get`. To convert an Iterable to a
PathFinder, the argument to lazyPathFinder is call-by-name. Path.fromFile
converts a File to a Path.

Thanks,
Mark

[Mark Harrah]

Sorry, I just tried this and it didn't work right. I've fixed it in trunk.

-Mark