Scala files compiled twice

[Udayakumar Rayala]

Hi,

We have different test configurations in our sbt build file(Unit,
Shared, Integration, Functional). All the setup is working fine. But
on some machines, it seems that certain configurations are compiled
twice.

Following is a sample output of it:

[info] Compiling 1 Scala source to /project/target/scala-2.9.1.final/
shared-classes...
[info] Compiling 26 Scala sources to /project/target/scala-2.9.1.final/
shared-classes...
[info] Compiling 45 Scala sources to /project/target/scala-2.9.1.final/
functional-classes...
[info] Compiling 44 Scala sources to /project/target/scala-2.9.1.final/
functional-classes...

Is it actually compiling different things and i am not able to see it
because of the ellipses? Or did anyone else notice this before?

I can post the sbt build file is this information is not sufficient.

Thanks,
Uday.

[Mark Harrah]

Hi Uday,

On Mon, 19 Dec 2011 11:03:39 -0800 (PST)
Udayakumar Rayala <uday....@gmail.com> wrote:

> Hi,
>
> We have different test configurations in our sbt build file(Unit,
> Shared, Integration, Functional). All the setup is working fine. But
> on some machines, it seems that certain configurations are compiled
> twice.
>
> Following is a sample output of it:
>
> [info] Compiling 1 Scala source to /project/target/scala-2.9.1.final/
> shared-classes...
> [info] Compiling 26 Scala sources to /project/target/scala-2.9.1.final/
> shared-classes...
> [info] Compiling 45 Scala sources to /project/target/scala-2.9.1.final/
> functional-classes...
> [info] Compiling 44 Scala sources to /project/target/scala-2.9.1.final/
> functional-classes...
>
> Is it actually compiling different things and i am not able to see it
> because of the ellipses? Or did anyone else notice this before?

This is (probably) standard incremental recompilation. sbt first compiles any changed files. If the public API has changed in some of those files, a second step compiles the transitive dependencies of those files with a changed public API to ensure correctness. The message could be better, but that's what is happening.

-Mark

[Udayakumar Rayala]

That makes sense.

And also, Is it possible to compile multiple configurations(Main, Unit, Integration, Functional) in parellel so that the overall compilation is faster? Or does it not make any difference ?

The reason is, in our current codebase it takes more time to compile than to run the tests.

Thanks,

Uday.

[Mark Harrah]

On Tue, 20 Dec 2011 10:26:09 -0800 (PST)
Udayakumar Rayala <uday....@gmail.com> wrote:

> That makes sense.
>
> And also, Is it possible to compile multiple configurations(Main, Unit,
> Integration, Functional) in parellel so that the overall compilation is
> faster? Or does it not make any difference ?

Yes, but dependencies must be satisfied. For example, Unit, Integration, and Functional probably depend on Main, so Main must be compiled first. If the test configurations don't have inter-dependencies, they can (and will) all be compiled in parallel. If Integration is defined as extending Unit, this implies a dependency on Unit, so Unit must be compiled first.

Also, the maximum number of concurrently executing tasks is limited to the number of cores on the machine.

-Mark

[Udayakumar Rayala]

Ok. But is there a sbt way of running tasks in parellel and waiting for it. Or i could run these compiles in background and wait for the processes to finish. Having a sbt way would be much cleaner.

Thanks,

Uday.

[Udayakumar Rayala]

> Ok. That works for me. But is there a sbt way of running multiple tasks in
> parellel and waiting for them.

sbt already runs tasks in parallel by default and waits for them to complete by continuing.

> Or I could run these commands in background and wait for these processes to
> complete. But having an sbt way of doing would be clean.

I don't understand how this relates to your question about compiling configuations in parallel.  What commands are running and what are they running in the background of?

    - Mark 

 

Hi Mark,

Lets say, i have 185 files in functional configuration, 160 files in integration configuration and 50 files in shared configuration on which both functional and integration depend on. I was thinking if i can compile 50 shared files first, then i can compile 185 functional and 160 integration files in parallel.

But SBT thinks that functional:compile and integration:compile have dependency on shared:compile. So it wont run these tasks in parallel. (Please correct me if my understanding of this is incorrect)

I have written a shell script which runs "./sbt shared:compile" first and then runs "./sbt functional:compile" and "./sbt integration:compile" commands in background so that they can run in parallel. But the problem is, i was not able to run two sbt instances in parallel in the same folder.

This is just one of the idea i am thinking so that i can speed up the compilation. If this is not the right approach then probably we have to live with this long compilation.

- Uday.

[Mark Harrah]

On Sun, 1 Jan 2012 13:47:08 -0800 (PST)
Udayakumar Rayala <uday....@gmail.com> wrote:

>
> >
> > > Ok. That works for me. But is there a sbt way of running multiple tasks
> > in
> > > parellel and waiting for them.
> >
> > sbt already runs tasks in parallel by default and waits for them to
> > complete by continuing.
> >
> > > Or I could run these commands in background and wait for these processes
> > to
> > > complete. But having an sbt way of doing would be clean.
> >
> > I don't understand how this relates to your question about compiling
> > configuations in parallel. What commands are running and what are they
> > running in the background of?
> >
> - Mark
>
> Hi Mark,
>
> Lets say, i have 185 files in functional configuration, 160 files in
> integration configuration and 50 files in shared configuration on which
> both functional and integration depend on. I was thinking if i can compile
> 50 shared files first, then i can compile 185 functional and 160
> integration files in parallel.
>
> But SBT thinks that functional:compile and integration:compile have
> dependency on shared:compile.

Yes. I don't see how this is inconsistent with your description in the first paragraph, though.

functional -> shared
integration -> shared
No dependencies between functional and integration.

> So it wont run these tasks in parallel.
> (Please correct me if my understanding of this is incorrect)

Why not? shared must be compiled first and then functional and integration can be compiled in parallel because there are no dependencies between them.

-Mark