zip files as "dependencies"? help w/ non-standard artifacts
JustinC
I'm trying to convert into sbt an existing multi-project Ant build
that deals with a bunch of xml schemas and zip files and uses the
xmlbeans code generator to compile them into java. I haven't made a
lot of headway yet; I'm stuck on how best to reference an output zip
file from one project in another.
Here's a simplified idea of what the sub projects are:
OurSchema:
Produces a zip of schema files that we author and that
live under the project dir.
TheirSchema:
Takes checked-in zips of schema files authored by other teams
and reorganizes them (not import how). Output is more zip files.
CompileSchema:
Takes the zip files from above, unzips them into staging areas,
and calls xmlbeans compiler. Ultimately produces multiple jars
of java classes.
There are actually more parts to it, but that's the gist. The overall
build needs to be able to publish all the produced zip and jar files.
Now, I'm able to create a zip file easily enough:
class OurSchema(info: ProjectInfo) extends DefaultProject(info)
{
def zipName = "our-schema.zip"
def xsdAndWsdl = ("schema" ##) ** ("*.xsd" | "*.wsdl")
// this zip needs to be published
lazy val ourzip = zipTask(xsdAndWsdl, outputPath, zipName)
}
But, how should I reference this zip in other projects for which the
zip is essentially a dependency (but not in the "classpath" sense).
E.g. something like:
class CompileSchema(info: ProjectInfo) extends DefaultProject(info)
{
// Needs to unzip <root>/OurSchema/target/our-schema.zip into
// <root>/CompileSchema/target/schema/ours and point xmlbeans
// compiler at that directory
}
My hunch is that I could use the local repo, and maybe something to do
with "Artifact", but it's not coming to me. Any tips appreciated!
Thanks
-j
Mark Harrah
> Hi,
>
> I'm trying to convert into sbt an existing multi-project Ant build
> that deals with a bunch of xml schemas and zip files and uses the
> xmlbeans code generator to compile them into java. I haven't made a
> lot of headway yet; I'm stuck on how best to reference an output zip
> file from one project in another.
>
> Here's a simplified idea of what the sub projects are:
>
> OurSchema:
> Produces a zip of schema files that we author and that
> live under the project dir.
> TheirSchema:
> Takes checked-in zips of schema files authored by other teams
> and reorganizes them (not import how). Output is more zip files.
> CompileSchema:
> Takes the zip files from above, unzips them into staging areas,
> and calls xmlbeans compiler. Ultimately produces multiple jars
> of java classes.
Are the other schema file projects (similar to your OurSchema) subprojects of
the same build? In that case you wouldn't need to publish the zips. You
could just reference them directly. But if you want to use dependency
management...
> There are actually more parts to it, but that's the gist. The overall
> build needs to be able to publish all the produced zip and jar files.
I've needed to document artifacts anyway, so I added a new page on them [1].
To add additional artifacts you use Artifact as you mention. Basically, you
add an artifact to be published by doing:
lazy val art = Artifact("name", "type", "extension")
lazy val art2 = Artifact("name2", "type2", "extension2")
...
To select artifacts from a dependency with custom artifacts:
lazy val dep =
"org" % "module" % "rev" artifacts(Artifact(...), Artifact(...), ...)
You can put dependencies in configurations [2] to keep them together:
// define a configuration called "schema"
lazy val schemaConf = conf("schema")
// define a dependency in the "schema" configuration
lazy val dep =
"org" % "module" % "rev" % "schema" artifacts(Artifact(...), ...)
Then, after an 'update', access them like:
configurationPath(schemaConf) ** "*.zip"
'configurationPath' returns the location to which artifacts in the 'schemaConf'
configuration are retrieved.
> Now, I'm able to create a zip file easily enough:
>
> class OurSchema(info: ProjectInfo) extends DefaultProject(info)
> {
> def zipName = "our-schema.zip"
> def xsdAndWsdl = ("schema" ##) ** ("*.xsd" | "*.wsdl")
> // this zip needs to be published
> lazy val ourzip = zipTask(xsdAndWsdl, outputPath, zipName)
> }
>
> But, how should I reference this zip in other projects for which the
> zip is essentially a dependency (but not in the "classpath" sense).
> E.g. something like:
>
> class CompileSchema(info: ProjectInfo) extends DefaultProject(info)
> {
> // Needs to unzip <root>/OurSchema/target/our-schema.zip into
> // <root>/CompileSchema/target/schema/ours and point xmlbeans
> // compiler at that directory
> }
There isn't an unzipTask, but there is a FileUtilities.unzip method [3].
Other than that, does the above artifacts information provide what you were
looking for?
> My hunch is that I could use the local repo, and maybe something to do
> with "Artifact", but it's not coming to me. Any tips appreciated!
-Mark
[1] http://code.google.com/p/simple-build-tool/wiki/Artifacts
[2] http://code.google.com/p/simple-build-tool/wiki/ManagedConfigurations
[3] http://simple-build-
tool.googlecode.com/svn/artifacts/latest/api/sbt/FileUtilities$object.html
JustinC
On Jan 16, 6:49 am, Mark Harrah <dmhar...@gmail.com> wrote:
> On Saturday 16 January 2010, JustinC wrote:
> > Here's a simplified idea of what the sub projects are:
>
> > OurSchema:
> > Produces a zip of schema files that we author and that
> > live under the project dir.
> > TheirSchema:
> > Takes checked-in zips of schema files authored by other teams
> > and reorganizes them (not import how). Output is more zip files.
> > CompileSchema:
> > Takes the zip files from above, unzips them into staging areas,
> > and calls xmlbeans compiler. Ultimately produces multiple jars
> > of java classes.
>
> Are the other schema file projects (similar to your OurSchema) subprojects of
> the same build? In that case you wouldn't need to publish the zips. You
> could just reference them directly. But if you want to use dependency
> management...
Yes, I want to have everything under one parent build. Two things
worried me about direct reference though:
1) It seemed like it might be bad style to reference artifacts in the
"target" directory of one subproject from another subproject.
2) I want to be able to build just a subset of the schema files, even
after a clean, without having to process them all. (I have a hunch
parameterized tasks will help to specify the subset.) Referencing
previously built stuff as dependencies via the local repo seems like
the right way to avoid extra building. I want this because there are
a ton of schema files and xmlbeans is slow to compile all of them, and
I don't want to mess with change detection for this.
> To add additional artifacts you use Artifact as you mention. Basically, you
> add an artifact to be published by doing:
>
> lazy val art = Artifact("name", "type", "extension")
> lazy val art2 = Artifact("name2", "type2", "extension2")
> ...
Ok. I've tried a few things along the lines of:
class Sandbox(info: ProjectInfo) extends ParentProject(info)
{
lazy val ourSchema = project("our-schema", "our-schema", new
OurSchema(_))
class OurSchema(info: ProjectInfo) extends DefaultProject(info)
{
def zipName = "ourschema.zip"
def xsdAndWsdl = ("schema" ##) ** ("*.xsd" | "*.wsdl")
lazy val ourZip = zipTask(xsdAndWsdl, outputPath, zipName)
lazy val z = Artifact("ours", "zip", "zip")
// override def artifacts = Set(Artifact("ourschema", "zip",
"zip"))
override def packageAction = super.packageAction dependsOn
(ourZip)
}
}
My zip file gets built in the target dir, but isn't published; the
default jar (which I actually don't need) is instead:
[info] == our-schema / publish-local ==
[info] :: publishing :: org#our-schema
[info] published our-schema to /Users/j/.ivy2/local/org/our-schema/
1.0/jars/our-schema.jar
[info] published ivy to /Users/j/.ivy2/local/org/our-schema/1.0/ivys/
ivy.xml
[info] == our-schema / publish-local ==
I also get an error from the parent project:
[info] == sandbox / deliver-local ==
[error] Error running deliver-local: download failed: org#our-schema;
1.0!ourschema.zip
I'm sure I'm missing something obvious.
> To select artifacts from a dependency with custom artifacts:
>
> lazy val dep =
> "org" % "module" % "rev" artifacts(Artifact(...), Artifact(...), ...)
>
> You can put dependencies in configurations [2] to keep them together:
Ok, I think I get the dependencies bit. The configuration(s) would go
in the parent project?
> > class CompileSchema(info: ProjectInfo) extends DefaultProject(info)
> > {
> > // Needs to unzip <root>/OurSchema/target/our-schema.zip into
> > // <root>/CompileSchema/target/schema/ours and point xmlbeans
> > // compiler at that directory
> > }
>
> There isn't an unzipTask, but there is a FileUtilities.unzip method [3].
> Other than that, does the above artifacts information provide what you were
> looking for?
I'm sure all the stuff I need is here, it's just going to be a matter
of me getting up the learning curve.
Thanks again for the response and for the great tool.
-j
Mark Harrah
> Awesome, thanks! More below...
>
> On Jan 16, 6:49 am, Mark Harrah <dmhar...@gmail.com> wrote:
> > On Saturday 16 January 2010, JustinC wrote:
> > > Here's a simplified idea of what the sub projects are:
> > >
> > > OurSchema:
> > > Produces a zip of schema files that we author and that
> > > live under the project dir.
> > > TheirSchema:
> > > Takes checked-in zips of schema files authored by other teams
> > > and reorganizes them (not import how). Output is more zip files.
> > > CompileSchema:
> > > Takes the zip files from above, unzips them into staging areas,
> > > and calls xmlbeans compiler. Ultimately produces multiple jars
> > > of java classes.
> >
> > Are the other schema file projects (similar to your OurSchema)
> > subprojects of the same build? In that case you wouldn't need to publish
> > the zips. You could just reference them directly. But if you want to
> > use dependency management...
>
> Yes, I want to have everything under one parent build. Two things
> worried me about direct reference though:
>
> 1) It seemed like it might be bad style to reference artifacts in the
> "target" directory of one subproject from another subproject.
As long as you set up the proper task dependencies, it should be fine. It'd
look something like:
class RootProject ... {
lazy val a = project("a", "A", new ProjectA(_))
lazy val b = project("b", "B", new ProjectB(_))
class ProjectA ... {
lazy val useZipFromB =
task { doSomething( b.zipPath ) } dependsOn( b.makeZip )
}
class ProjectB ... {
def zipPath = outputPath / "my.zip"
lazy val makeZip = task { create(zipPath) }
}
}
Best practices in sbt don't really exist at this point, but I use this style.
I don't ever delete products (like "my.zip") except in the defining task
('makeZip') or in a 'clean' task, so I know that if I depend on 'makeZip', I
can use "my.zip" safely.
> 2) I want to be able to build just a subset of the schema files, even
> after a clean, without having to process them all. (I have a hunch
> parameterized tasks will help to specify the subset.) Referencing
> previously built stuff as dependencies via the local repo seems like
> the right way to avoid extra building. I want this because there are
> a ton of schema files and xmlbeans is slow to compile all of them, and
> I don't want to mess with change detection for this.
Yes I think that parameterized tasks would probably work. I'd think something
like:
class CompileSchema ... {
lazy val compileSchema =
task { args =>
val projects: Iterable[OurSchema] =
dependencies.filter(d => args contains d.name)
val zipTasks = projects.map(_.ourZip)
task { /*process zips in projects */ } dependsOn( zipTasks : _*)
}
}
with the detail that dependencies is of type Iterable[Project] and so the
you'd have to check that the project is of type OurSchema or whatever.
Probably not. Dependency management can be a pain. What version of sbt are
you using? I'll try your sample projects on that version and see.
> > To select artifacts from a dependency with custom artifacts:
> >
> > lazy val dep =
> > "org" % "module" % "rev" artifacts(Artifact(...), Artifact(...), ...)
> >
> > You can put dependencies in configurations [2] to keep them together:
>
> Ok, I think I get the dependencies bit. The configuration(s) would go
> in the parent project?
Configurations are specified per project so you can easily have different
configurations for each project. You don't have to repeat the declarations
though. You can put them in a common trait and mix it into each project:
trait WithConfs extends BasicManagedProject {
val a = conf("a")
}
class OurSchema(...) extends DefaultProject(info) with WithConfs
or declare them in a method in the parent and reference them in subprojects:
def confs = conf("a") :: conf("b") :: Nil
class OurSchema(...) extends DefaultProject(info) {
override def ivyConfigurations = super.ivyConfigurations ++ confs
}
> > > class CompileSchema(info: ProjectInfo) extends DefaultProject(info)
> > > {
> > > // Needs to unzip <root>/OurSchema/target/our-schema.zip into
> > > // <root>/CompileSchema/target/schema/ours and point xmlbeans
> > > // compiler at that directory
> > > }
> >
> > There isn't an unzipTask, but there is a FileUtilities.unzip method [3].
> > Other than that, does the above artifacts information provide what you
> > were looking for?
>
> I'm sure all the stuff I need is here, it's just going to be a matter
> of me getting up the learning curve.
>
> Thanks again for the response and for the great tool.
> -j
Sure, let me know if you have any feedback on the documentation- structure or
content- that would make it easier or clearer.
-Mark
JustinC
> > I'm sure I'm missing something obvious.
>
> Probably not. Dependency management can be a pain. What version of sbt are
> you using? I'll try your sample projects on that version and see.
I'm starting to work through your last post, but just wanted to post
my version info:
[info] Building project sandbox 1.0 using Sandbox
[info] with sbt 0.5.6 and Scala 2.7.7
I posted my play build file earlier. The only other thing missing is
that, in my sandbox, I have an empty file:
<projectRoot>/OurSchema/schema/foo.xsd
So that the zip task has something to do.
> Sure, let me know if you have any feedback on the documentation- structure or
> content- that would make it easier or clearer.
I will think about this as I'm plodding along. Not only am I new to
sbt, but I haven't done much scala or ivy either, though I've been
following all three for some time. Suffice it to say, it's a nice
challenge :)
I know when I'm learning something new, I like to have both a cookbook-
style set of examples, as well as a thorough, organized reference with
all the minutiae. The scaladoc perhaps could serve well as the
latter, though I notice there isn't much text in the those docs
themselves.
One thing I liked about Ant was that all the tasks have very complete
docs[1] with examples. Though sbt seems a much more flexible/open
tool, and also has a lot more conventions, so I'd guess it's
relatively harder to circumscribe with docs.
Mark Harrah
> > structure or content- that would make it easier or clearer.
>
> I will think about this as I'm plodding along. Not only am I new to
> sbt, but I haven't done much scala or ivy either, though I've been
> following all three for some time. Suffice it to say, it's a nice
> challenge :)
>
> I know when I'm learning something new, I like to have both a cookbook-
> style set of examples, as well as a thorough, organized reference with
> all the minutiae. The scaladoc perhaps could serve well as the
> latter, though I notice there isn't much text in the those docs
> themselves.
I intend the documentation on the site to be mostly reference but definitely
not devoid of examples: full ones like [1,2,3] or code examples throughout.
This is the type of documentation that I can produce without much guidance
from users.
I don't know what people want tutorials on though or that I'm even the best
person to write them. For example, I've seen several very nice writeups on
getting started with sbt and X, where X might be android, testing, or just
Scala in general (google 'scala sbt' to see a few).
I find Scala source code easier to read, so writing scaladoc is not naturally
appealing to me unless people ask for specific things to be documented (which
is welcome). I am definitely happy to write examples for the wiki, though.
I'm happy to let anyone edit the wiki as well (just ask). I'll check/edit
whatever people write, so there isn't a need to worry about it being perfect
or anything.
I do like to leverage effort put in by users. If a user comes along and wants
to see a basic example of doing X in sbt (obviously I'm not going to write
everyone's project definitions for them), writes the outline of a tutorial they
would like to see (I would write the tutorial), or proposes a structure for
the wiki that makes more sense as a user, I'm happy to fill in details. If
users want some way to vote on the topic of the month that I document, great,
I'm for that.
> One thing I liked about Ant was that all the tasks have very complete
> docs[1] with examples. Though sbt seems a much more flexible/open
> tool, and also has a lot more conventions, so I'd guess it's
> relatively harder to circumscribe with docs.
I certainly agree. Going forward, I'd like to make tasks a bit more self-
documenting at the code level and make the related inputs and outputs more
obvious. I think that external documentation will follow more easily with
this.
Thanks for your feedback!
-Mark
[1] http://code.google.com/p/simple-build-tool/wiki/ProjectDefinitionExamples
[2] http://code.google.com/p/simple-build-tool/wiki/WebstartExample
[3] http://code.google.com/p/simple-build-tool/wiki/WebApplicationExample
> -j
> [1] http://ant.apache.org/manual/index.html
>