incremental compilation + memory leek with macros

[Simon Schäfer]

I have two issues working on Eugenes newest macro stuff:

1)
Incremental compilation is not working. Whenever I do a save, SBT
recompiles the whole project. Is this because I'm working with macros or
do I have some configuration problem? Config at [1]

2)
There is a memory leek. After some dozens recompiles the compilation
crashes with a OutOfMemoryError. Sometimes even SBT crashes due to this
error. Something I can do against this?

I tried this with 0.12.1 and 0.13.0-SNAPSHOT and macro branch
topic/introduce-member [2].

[1]:
https://github.com/sschaef/sbt-example-paradise/blob/master/project/Build.scala
[2]: https://github.com/scalamacros/kepler/commits/topic/introduce-member

[Mark Harrah]

On Fri, 08 Mar 2013 23:57:36 +0100
Simon Schäfer <ma...@antoras.de> wrote:

> I have two issues working on Eugenes newest macro stuff:
>
> 1)
> Incremental compilation is not working. Whenever I do a save, SBT
> recompiles the whole project. Is this because I'm working with macros or
> do I have some configuration problem? Config at [1]

Macros do tend to require a lot of recompilation, but it is hard to say anything without more information. What file is being saved, what gets recompiled, etc...

> 2)
> There is a memory leek. After some dozens recompiles the compilation
> crashes with a OutOfMemoryError. Sometimes even SBT crashes due to this
> error. Something I can do against this?

This is likely to be a bug in the macro implementation in the compiler. If your project can be compiled with a standard Scala version, I'd check that and if it doesn't leak, file a bug against the introduce-member branch.

-Mark

> I tried this with 0.12.1 and 0.13.0-SNAPSHOT and macro branch
> topic/introduce-member [2].
>
> [1]:
> https://github.com/sschaef/sbt-example-paradise/blob/master/project/Build.scala
> [2]: https://github.com/scalamacros/kepler/commits/topic/introduce-member
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Simon Schäfer]

Hi Mark,

Macros do tend to require a lot of recompilation, but it is hard to say anything without more information.  What file is being saved, what gets recompiled, etc...

I have two projects - one for the macros and one for the user code. It doesn't matter in which project the file is - on a save each source file is recompiled.

Even files that are not referenced are recompiled (and recompilation of everything is done when such files are changed). For example I have a single main class that executes every module[1]. It doesn't matter if calls to the test-methods are uncommented or not.

> 2)
> There is a memory leek. After some dozens recompiles the compilation
> crashes with a OutOfMemoryError. Sometimes even SBT crashes due to this
> error. Something I can do against this?

This is likely to be a bug in the macro implementation in the compiler.  If your project can be compiled with a standard Scala version, I'd check that and if it doesn't leak, file a bug against the introduce-member branch.

Yes, I will do that.


[1]: https://github.com/sschaef/scala-macro-playground/blob/master/core/src/main/scala/playground/Main.scala

[Simon Schäfer]

Hi Eugene,

are you aware of the memory leak mentioned in this topic?

> 2)
> There is a memory leek. After some dozens recompiles the compilation
> crashes with a OutOfMemoryError. Sometimes even SBT crashes due to this
> error. Something I can do against this?

This is likely to be a bug in the macro implementation in the compiler.  If your project can be compiled with a standard Scala version, I'd check that and if it doesn't leak, file a bug against the introduce-member branch.

Yes, I will do that.

After switching back to the macro paradise master branch the memory leak seems gone, furthermore compile time has decreased dramatically. Thus, it is a bug in the introduce-member branch. Incremental compilation still doesn't seem to work, but it is not problematic anymore due to reduced compile times.

[Eugene Burmako]

Yes indeed that's a side-effect of the prototype I put in place to implement introduce-member. I took measures to fight potential leaks, but apparently they weren't terribly successful. Could you please create a JIRA issue and let me know how important this scenario is to you?

[Mark Harrah]

On Thu, 14 Mar 2013 11:05:17 -0700 (PDT)
Simon Schäfer <ma...@antoras.de> wrote:

> Hi Mark,
>
>
> > Macros do tend to require a lot of recompilation, but it is hard to say
> > anything without more information. What file is being saved, what gets
> > recompiled, etc...
> >
>
> I have two projects - one for the macros and one for the user code. It
> doesn't matter in which project the file is - on a save each source file is
> recompiled.
>
> Even files that are not referenced are recompiled (and recompilation of
> everything is done when such files are changed). For example I have a
> single main class that executes every module[1]. It doesn't matter if calls
> to the test-methods are uncommented or not.

You'll want to look at 'last compile'. It is likely that sbt sees that many (> 50%) sources need to be recompiled and therefore it just recompiles everything. That output should show if that is the case as well as what was actually invalidated.

-Mark

> [1]:
> https://github.com/sschaef/scala-macro-playground/blob/master/core/src/main/scala/playground/Main.scala

[Simon Schäfer]

Hi Mark, hi Eugene,

Please Eugene, could you take another look on this?

It is true, more than 50% of the source files get invalidated. The
interesting thing is that this is also the case when nothing changes,
i.e when I execute "run" two times without changing something between
the calls ("initial invalidated sources" are correctly empty but
"modified external sources" are non empty).

The "modified external sources" are the macros, maybe the compiler wants
to recompile them always (in macro-paradise master). Is this a bug or
expected behavior?

[Mark Harrah]

On Mon, 18 Mar 2013 18:10:21 +0100
Simon Schäfer <ma...@antoras.de> wrote:

> Hi Mark, hi Eugene,
>
> Please Eugene, could you take another look on this?
>
> It is true, more than 50% of the source files get invalidated. The
> interesting thing is that this is also the case when nothing changes,
> i.e when I execute "run" two times without changing something between
> the calls ("initial invalidated sources" are correctly empty but
> "modified external sources" are non empty).
>
> The "modified external sources" are the macros, maybe the compiler wants
> to recompile them always (in macro-paradise master). Is this a bug or
> expected behavior?

There are too many variables here to guess. It would be best if you could provide a minimal, reproducible test case.

-Mark

[Simon Schäfer]

> There are too many variables here to guess. It would be best if you could provide a minimal, reproducible test case.
>
> -Mark

I did it: https://github.com/sschaef/sbttest

Sample output:

> run
[info] Compiling 2 Scala sources to
/home/antoras/test/sbttest/macros/target/scala-2.11/classes...
[info] Compiling 1 Scala source to
/home/antoras/test/sbttest/core/target/scala-2.11/classes...
[info] Running Main
0
0
[success] Total time: 8 s, completed Mar 21, 2013 6:17:37 PM
> run
[info] Compiling 2 Scala sources to
/home/antoras/test/sbttest/macros/target/scala-2.11/classes...
[info] Compiling 1 Scala source to
/home/antoras/test/sbttest/core/target/scala-2.11/classes...
[info] Running Main
0
0
[success] Total time: 3 s, completed Mar 21, 2013 6:17:43 PM

Between the two runs there is no source code change, thus the second
time nothing should be recompiled.

[nafg]

Can you elaborate -- does sbt always recompile everything when more than 50% of files need to be recompiled? Is this a recent change? Why?

[Mark Harrah]

On Thu, 4 Apr 2013 22:13:31 -0700 (PDT)
nafg <nafto...@gmail.com> wrote:

> Can you elaborate -- does sbt always recompile everything when more than
> 50% of files need to be recompiled? Is this a recent change? Why?

Yes, yes, and because it avoids recompiling most of the project multiple times. It is especially for larger projects. The amount will be configurable in 0.13. Is there a situation where it is making things worse for you?

-Mark

[Naftoli Gugenheim]

I think it might be helpful in a certain project but I didn't see that happen. In what version was this introduced?

--
You received this message because you are subscribed to a topic in the Google Groups "simple-build-tool" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/simple-build-tool/e82Yg6jfrGE/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to simple-build-t...@googlegroups.com.

[Mark Harrah]

On Mon, 8 Apr 2013 09:35:31 -0400
Naftoli Gugenheim <nafto...@gmail.com> wrote:

> I think it might be helpful in a certain project but I didn't see that
> happen. In what version was this introduced?

0.12.2

> > simple-build-t...@googlegroups.com <javascript:;>.
> > To post to this group, send email to simple-b...@googlegroups.com<javascript:;>
> > .

> > Visit this group at http://groups.google.com/group/simple-build-tool?hl=en
> > .
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
> >
>

> --

> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.