Re: [sbt] Compiler plugin + not detecting files to compile after a clean

[Mark Harrah]

On Fri, 31 Aug 2012 23:50:46 -0700 (PDT)
Renato Garcia <renatao...@gmail.com> wrote:

> Hi,
>
> I have a compiler plugin that for a given class MyClass it will generate a
> source file QMyClass.scala at the "generated" dir and then compile it. The
> compilation is done in the plugin using the following code:
>
> new global.Run().compile(sourceFileName :: Nil)
>
>
> The initial build works fine, and the incremental also, however when I
> "clean" and then try to compile again, it won't compile MyClass anymore. It
> doesn't fail with, it just seems that sbt is not picking it to compile. The
> logs for the initial build and for the build after the clean are below.
>
> If I exit the console and enter again it will work, or if I issue "sbt
> clean compile" from the command line for the full build it will also work.
>
> Sbt seems to get lost after the first compilation. Does anybody have an
> idea of what's going on and how I could try to avoid this issue?

Generated sources need to be present at the start of a compilation. There is no support for compiling new sources produced during a compilation, like I think javac does for annotations. If you see it work at all, it is because a second 'compile' will include the source generated by the first 'compile', but it will always be one step out of date.

To compile the generated sources, explicitly add another compilation by making it a subproject or a configuration.

-Mark

> Thanks!
>
> === Initial build log (compile) ====
> [debug]
> [debug] Initial source changes:
> [debug] removed:Set()
> [debug] added:
> Set(/home/rgarcia/tmp/SimpleSbt/src/main/scala/simple/MyClass.scala)
> [debug] modified: Set()
> [debug] Removed products: Set()
> [debug] Modified external sources: Set()
> [debug] Modified binary dependencies: Set()
> [debug] Initial directly invalidated sources:
> Set(/home/rgarcia/tmp/SimpleSbt/src/main/scala/simple/MyClass.scala)
> [debug]
> [debug] Sources indirectly invalidated by:
> [debug] product: Set()
> [debug] binary dep: Set()
> [debug] external source: Set()
> [debug] Initially invalidated:
> Set(/home/rgarcia/tmp/SimpleSbt/src/main/scala/simple/MyClass.scala)
> [info] Compiling 1 Scala source to
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes...
> [debug] Getting compiler-interface from component compiler for Scala
> 2.10.0-M7
> [debug] Other repositories:
> [debug] Default repositories:
> [debug]
> FileRepository(local,FileConfiguration(true,None),sbt.Patterns@41dc7621)
> [debug] Getting compiler-interface from component compiler for Scala
> 2.10.0-M7
> [debug] Other repositories:
> [debug] Default repositories:
> [debug]
> FileRepository(local,FileConfiguration(true,None),sbt.Patterns@62ea0720)
> [debug] Running cached compiler 5f61dd83, interfacing (CompilerInterface)
> with Scala compiler version 2.10.0-M7
> [debug] Calling Scala compiler with arguments (CompilerInterface):
> [debug]
> -Xplugin:/dvlp/workspaces/rennie/querydsl-scala-compiler-plugin/target/scala-compiler-plugin-0.0.1-SNAPSHOT.jar
> [debug] -d
> [debug] /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes
> [debug] -bootclasspath
> [debug]
> /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/netx.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/plugin.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes:/home/rgarcia/.sbt/boot/scala-2.10.0-M7/lib/scala-library.jar
> [debug] -classpath
> [debug]
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes:/home/rgarcia/tmp/SimpleSbt/lib/scala-compiler-plugin-0.0.1-SNAPSHOT.jar
> Testing
> Testing
> [debug] Scala compilation took 3.591696541 s
> [debug] Invalidated direct:
> Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [debug] Invalidated direct: Set()
> [debug] Incrementally invalidated:
> Set(/home/rgarcia/tmp/SimpleSbt/src/main/scala/simple/MyClass.scala,
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [info] Compiling 1 Scala source to
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes...
> [debug] Getting compiler-interface from component compiler for Scala
> 2.10.0-M7
> [debug] Other repositories:
> [debug] Default repositories:
> [debug]
> FileRepository(local,FileConfiguration(true,None),sbt.Patterns@1daf2af7)
> [debug] Getting compiler-interface from component compiler for Scala
> 2.10.0-M7
> [debug] Other repositories:
> [debug] Default repositories:
> [debug]
> FileRepository(local,FileConfiguration(true,None),sbt.Patterns@3a7bbea4)
> [debug] Running cached compiler 5cd81082, interfacing (CompilerInterface)
> with Scala compiler version 2.10.0-M7
> [debug] Calling Scala compiler with arguments (CompilerInterface):
> [debug]
> -Xplugin:/dvlp/workspaces/rennie/querydsl-scala-compiler-plugin/target/scala-compiler-plugin-0.0.1-SNAPSHOT.jar
> [debug] -d
> [debug] /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes
> [debug] -bootclasspath
> [debug]
> /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/netx.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/plugin.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes:/home/rgarcia/.sbt/boot/scala-2.10.0-M7/lib/scala-library.jar
> [debug] -classpath
> [debug]
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes:/home/rgarcia/tmp/SimpleSbt/lib/scala-compiler-plugin-0.0.1-SNAPSHOT.jar
> Testing
> Testing
> [debug] Scala compilation took 1.163614745 s
> [debug] Invalidated direct: Set()
> [debug] Incrementally invalidated: Set()
> [success] Total time: 5 s, completed Sep 1, 2012 4:32:17 PM
>
>
> === (compile) after (clean) ===
>
> [debug]
> [debug] Initial source changes:
> [debug]
> removed:Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [debug] added: Set()
> [debug] modified: Set()
> [debug] Removed products:
> Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes/simple/QMyClass$.class,
> /home/rgarcia/tmp/SimpleSbt/target/scala-2.10/classes/simple/QMyClass.class)
> [debug] Modified external sources: Set()
> [debug] Modified binary dependencies: Set()
> [debug] Initial directly invalidated sources:
> Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [debug]
> [debug] Sources indirectly invalidated by:
> [debug] product:
> Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [debug] binary dep: Set()
> [debug] external source: Set()
> [debug] Initially invalidated:
> Set(/home/rgarcia/tmp/SimpleSbt/target/scala-2.10/generated/simple/QMyClass.scala)
> [debug] Invalidated direct: Set()
> [debug] Incrementally invalidated: Set()
> [success] Total time: 0 s, completed Sep 1, 2012 4:33:49 PM
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/zmMvqoeOPi0J.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
>

[Renato Garcia]

Hi Mark,

Thanks for your reply!

The compiler is actually compiling the generated sources. Everything is working as expected in this aspect. The issue is that sbt seems to get lost in this scenario when tracking what needs to be recompiled in the interactive mode.

So, if I issue a "sbt clean compile", it will always work. Same if I enter the console and do a ";clean ;compile", exit and repeat it those steps. However, if I use interactive mode, only the first ";clean ;compile" will have effect and the subsequent "clean" and "compile" will not compile anything. It will just run and not pick anything to compile.

I'm new to sbt, but from what I could understand, the first "clean/compile" will simply recompile ALL the sources, then sbt will start tracking what is the product generated for each source; later if that product is removed, it means that the respective source needs to be recompiled. Therefore, the subsequent "clean" will simply remove all the class files, which are the products, and this will trigger the compilation of ALL the sources. It will achieve the same desired result, which is to recompile ALL sources, however it will use a different approach -- and that is why it works only in the first time.

Assuming that this is somewhat correct, I believe that sbt is having trouble to detect the generated product. Usually it will be something like MyClass.scala -> MyClass.class, but in this new scenario it is MyClass.scala -> QMyClass.class and later QMyClass.scala -> QMyClass.class, which could be overriding the the first mapping and then MyClass.scala will never get recompiled. This is just a guess, I'll need to take a look at the code, but does it make sense? I'm in the right path?

Thanks again!
Renato

[Mark Harrah]

On Sat, 1 Sep 2012 20:32:00 -0700 (PDT)
Renato Garcia <renatao...@gmail.com> wrote:

> Hi Mark,
>
> Thanks for your reply!
>
> The compiler is actually compiling the generated sources. Everything is
> working as expected in this aspect. The issue is that sbt seems to get lost
> in this scenario when tracking what needs to be recompiled in the
> interactive mode.

I see that you described this the first time, sorry. I'm surprised that the compiler allows simultaneous Runs as you show here:

new global.Run().compile(sourceFileName :: Nil)

What phase does your plugin run after and/or how are you keeping the compiler from getting confused by the multiple Runs?

(Right now, I don't see how sbt can handle this without specific scalac support for generating/compiling sources, but perhaps it will work out!)

-Mark

> > > To post to this group, send email to simple-b...@googlegroups.com<javascript:>.

> >
> > > To unsubscribe from this group, send email to

> > simple-build-t...@googlegroups.com <javascript:>.

> > > For more options, visit this group at
> > http://groups.google.com/group/simple-build-tool?hl=en.
> > >
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/-g8HD3DUiY0J.

[Renato Garcia]

The plugin is run at "jvm" phase. I don't think that there is anything special, just creating a new run seem to work, even in the scenario where I'm compiling classes that dependeng on the generated one.