aggregating all projects

[Paul Phillips]

If I fail to define a root project, sbt will define one for me which aggregates all projects. Okay, so what if I want to define a root project, but also want it to aggregate all subprojects automatically? It is not a small thing to have to manually and redundantly enumerate the names of 12 subprojects, and there appears to be nothing which will catch any error either (i.e. leave one out and there's no reason for it to be an error.)

Thanks to the project macro I've reduced my build.sbt to the "absolute" minimum assuming we take the 50% whitespace as a given, which means the aggregate boilerplate is both a huge percentage of the total file and an unpleasant maintenance burden.

As a final smack, there is what looks like a bug somewhere in the intersection of varargs, overloading, and implicit conversions such that I cannot define a list of projects and pass the list to the varargs aggregate method with _* notation. Only compiles if the individual names are given directly.

/scala/trunk/build.sbt:45: error: no `: _*' annotation allowed here

(such annotations are only allowed in arguments to *-parameters)

) aggregate (all: _*)

                ^

But writing that made me notice what would (and does) work.

aggregate (all map Project.projectToRef: _*)

[Paul Phillips]

Maybe this will work:

  aggregate(ReflectUtilities.allVals[Project](root).values.toSeq: _*)

Haven't tried it yet as I'm down another rabbit hole trying to figure out why everything is always recompiled, even when nothing has been touched.

[Paul Phillips]

On Wed, Aug 28, 2013 at 3:16 PM, Paul Phillips <pa...@improving.org> wrote:

Haven't tried it yet as I'm down another rabbit hole trying to figure out why everything is always recompiled, even when nothing has been touched.

 -      scalaSource in Compile := file("src") / id

 +      scalaSource in Compile := (baseDirectory in ThisBuild).value / "src" / id

Anyone happen to know what precisely is happening here such that they find the same source files, but the first one always recompiles everything and the second one doesn't? I would understand if the first form found nothing, but this behavior is harder to comprehend.

[Mark Harrah]

The first is relative to the current working directory instead of the project's base directory. At some point it gets made absolute but somehow both variants get saved in the Analysis object and some logic somewhere probably doesn't recognize them as the same in an uptodate check.

In 0.7, the file(...) method would resolve the path against the project base directory, but I haven't found a way to do that in 0.10+, which is unfortunate.

-Mark

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool.
> For more options, visit https://groups.google.com/groups/opt_out.

[Mark Harrah]

Yes, I don't have a good idea of the right way to fix this. The most straightforward way is to require .ref on Project instances where a ProjectRef is expected. That is pretty much everywhere you would currently be referring to a Project, but doesn't trip up scalac.

Or, perhaps Project could extend LocalProject. That's an ugly hack, but it doesn't trip up scalac.

I don't think I would use varargs as extensively again, but it is hard to throw away .dependsOn(x,y,z) for .dependsOn(Seq(x,y,z)). The behavior with implicits is probably reasonable considering the constraints, but I think situations like this (and not being mixed with by-name parameters) probably outweigh the benefits.

[Mark Harrah]

On Wed, 28 Aug 2013 14:27:09 -0700
Paul Phillips <pa...@improving.org> wrote:

> If I fail to define a root project, sbt will define one for me which
> aggregates all projects. Okay, so what if I want to define a root project,
> but also want it to aggregate all subprojects automatically? It is not a
> small thing to have to manually and redundantly enumerate the names of 12
> subprojects, and there appears to be nothing which will catch any error
> either (i.e. leave one out and there's no reason for it to be an error.)

There isn't a clean way better than the reflective approach you showed in a follow up, at least not without more work than you are looking for. Why do you need to define the root project yourself? Some reasons I can think of are to control its ID or add dependencies.

-Mark

[Paul Phillips]

On Thu, Aug 29, 2013 at 9:21 AM, Mark Harrah <dmha...@gmail.com> wrote:

The first is relative to the current working directory instead of the project's base directory.

That seems like a dangerous thing to have at all. Does cwd change if I start sbt in the base directory and then say "project sub" which is in a subdirectory? Either way though it took me hours to pinpoint what was happening here, so it would be great to fix whatever it is that is comparing paths. Comparing paths for equality in general is fraught with traps, as I think you have already found, as has scalac.

[Paul Phillips]

On Thu, Aug 29, 2013 at 9:26 AM, Mark Harrah <dmha...@gmail.com> wrote:

I don't think I would use varargs as extensively again, but it is hard to throw away .dependsOn(x,y,z) for .dependsOn(Seq(x,y,z)).

It's aggravating having to choose between several suboptimal routes. The only occasion where one may not be able to distinguish a T from a Seq[T] is if T is not bounded by something which isn't in Seq. Much of the time it isn't even generic, we have Foo and Seq[Foo], and there is no reason other than die-roll-of-the-implementation that we can't have both at once. Think about this:

  def f(xs: String*) = 1

  def f(xs: Seq[String]) = 2

./a.scala:3: error: double definition:

  method f:(xs: Seq[String])Int and

  method f:(xs: String*)Int at line 2

  have same type after erasure: (xs: Seq)Int

    def f(xs: Seq[String]) = 2

        ^

  one error found

Why is the purely synthetic implementation wrapper around the varargs call using the public type to wrap it? Why is a collision here possible at all? Why doesn't varargs use a custom varargs container? It isn't as if "Seq" is so friendly to java interop, which require varargs bridges already.

[Paul Phillips]

On Thu, Aug 29, 2013 at 9:31 AM, Mark Harrah <dmha...@gmail.com> wrote:

Some reasons I can think of are to control its ID or add dependencies.

The immediate motivation was the first - I don't consider "what things are called" to be a minor detail. The second seems likely to come up as well. More than either of those, I don't want at some point in the future to have to spin up on these details or otherwise identify the full list of what I can't do in order to do something which is now necessary.

https://github.com/paulp/scala/tree/sbt

[Mark Harrah]

On Thu, 29 Aug 2013 10:52:16 -0700
Paul Phillips <pa...@improving.org> wrote:

> On Thu, Aug 29, 2013 at 9:21 AM, Mark Harrah <dmha...@gmail.com> wrote:
>
> > The first is relative to the current working directory instead of the
> > project's base directory.
>
>
> That seems like a dangerous thing to have at all. Does cwd change if I
> start sbt in the base directory and then say "project sub" which is in a
> subdirectory?

Right, you should not depend on the cwd. That you can ever end up with relative paths is not desirable- I don't know how to prevent it. If `file` didn't exist, people would use new File, same problem. Sometimes you want to construct absolute paths.

In Project construction (either Project.apply or Project.in(File)), you can pass a relative path because it will resolve it against the base directory of the build. This is before settings, so baseDirectory.value / "sub" doesn't make sense yet. (From a consistency standpoint, this isn't great.)

Every task that accepts Files could also pull the baseDirectory and resolve everything against it. This isn't great because you check and/or convert to absolute paths everywhere you might possibly want to use a file. Plugin authors would have to do this consistently as well.

`file` could be removed and `absFile` added, which checks isAbsolute. I'm not sure if there are use cases not covered, but at least Project.in(File) would have to change somehow.

> Either way though it took me hours to pinpoint what was
> happening here, so it would be great to fix whatever it is that is
> comparing paths. Comparing paths for equality in general is fraught with
> traps, as I think you have already found, as has scalac.

The fix is very likely to be catching all relative paths and making them absolute because there are lookups of Files in maps everywhere in the incremental compiler. These files come from lots of places, so it would be ideal to catch them at the source (the `file` method). I'm pretty sure the hooks sbt puts into the compiler convert things to absolute there at that source.

-Mark

[Paul Phillips]

On Thu, Aug 29, 2013 at 4:05 PM, Mark Harrah <dmha...@gmail.com> wrote:

That you can ever end up with relative paths is not desirable- I don't know how to prevent it.

Don't have to prevent it, only have to warn about it at choke points.

[Paul Phillips]

On Thu, Aug 29, 2013 at 4:05 PM, Mark Harrah <dmha...@gmail.com> wrote:

`file` could be removed and `absFile` added, which checks isAbsolute.

I would maybe do it at the type level: class AbsoluteFile ; implicit def liftRelativeFile(f: File)(implicit baseDirectory): AbsoluteFile. You're forever having to play defense when the fundamental data type is java.io.File rather than something which can place tighter requirements around it at construction. I'm sure it would be a big painful thing to pursue. I'd be happy with the warning (here, and lots of other places where it would be obvious to you that I'm doing something insensible but is not so obvious to me.)

[Mark Harrah]

On Fri, 30 Aug 2013 08:33:23 -0700
Paul Phillips <pa...@improving.org> wrote:

> On Thu, Aug 29, 2013 at 4:05 PM, Mark Harrah <dmha...@gmail.com> wrote:
>
> > `file` could be removed and `absFile` added, which checks isAbsolute.
>
>
> I would maybe do it at the type level: class AbsoluteFile ; implicit def
> liftRelativeFile(f: File)(implicit baseDirectory): AbsoluteFile.

A problem is that a baseDirectory isn't available at that point. If it were, `file` would just resolve everything against it.

> You're
> forever having to play defense when the fundamental data type is
> java.io.File rather than something which can place tighter requirements
> around it at construction. I'm sure it would be a big painful thing to
> pursue.

Yes, I did the type level thing for 0.7 and earlier with Path, which guaranteed absolute files. Users had to convert between Path and File all the time, which wasn't a great experience either.

> I'd be happy with the warning (here, and lots of other places where
> it would be obvious to you that I'm doing something insensible but is not
> so obvious to me.)

I agree about a warning/error being probably the most practical approach, but there are many places this warning would have to be inserted. Say you set sourceDirectory := file("not-src"). This spreads to many settings and tasks, which would all have to check for relative files. There are several entry points like this, since pretty much everything is open to be modified.

So, it is obvious in the sense that there is a straightforward check, isAbsolute, but not easy to put everywhere. I think an absFile is a reasonable tradeoff because it covers the common use cases, like yours.

[Paul Phillips]

On Fri, Aug 30, 2013 at 9:09 AM, Mark Harrah <dmha...@gmail.com> wrote:

A problem is that a baseDirectory isn't available at that point.  If it were, `file` would just resolve everything against it.

Right, that was intentional because it makes it easy to issue the error at exactly those places where baseDirectory isn't available. The arrival of a default or fallback implicit is your sign that somebody is getting themselves into trouble.

[Paul Phillips]

On Fri, Aug 30, 2013 at 9:09 AM, Mark Harrah <dmha...@gmail.com> wrote:

Yes, I did the type level thing for 0.7 and earlier with Path, which guaranteed absolute files.  Users had to convert between Path and File all the time, which wasn't a great experience either.

Here is how I see it, assuming a "Path" encodes a File with a known-to-be-absolute path:

  a Path can always be implicitly converted to a File

  a File can always be implicitly converted to a Path if an implicit BaseDirectory is available

  if no such implicit is available, it is an error to pass a File where a Path is expected

Where does this break down?

[Mark Harrah]

Just where implicit conversions of this kind usually do. Files are often manipulated as Seq[File] or Map[File, ...] or whatever. So, you need Seq[Path] <=> Seq[File] and such things.

There was a PathFinder as well, but now this is mainly an implicit class for Seq[File] with file building methods and at the end a call to .get to get the Seq[File] back.

As a side note, I'd ideally like richer file-like types for other reasons, like distinguishing between "use this File for the output directory location" and "this File has been written to, you can read from it all you want, but no more writing". I've not bothered so far because it seems like it would be a mess.

-Mark

[Mark Harrah]

I see. baseDirectory is a setting, so you'd get back a setting:

implicit def liftRelativeFile(f: File): Initialize[AbsoluteFile] =
// essentially
Def.setting { baseDirectory.value / f }

-Mark

[Paul Phillips]

On Fri, Aug 30, 2013 at 9:45 AM, Mark Harrah <dmha...@gmail.com> wrote:

I see.  baseDirectory is a setting, so you'd get back a setting:

 implicit def liftRelativeFile(f: File): Initialize[AbsoluteFile] =
   // essentially
   Def.setting { baseDirectory.value / f }

Right, sbtify to taste.