Zinc incompatibility with SBT

[Grzegorz Slowikowski]

Hi guys

Since SBT 0.13.8 'AggressiveCompier' is deprecated. SBT doesn't use it, but Zinc still does.
I don't know, what problems this could case, but I found one minor, but irritating.

Zinc 0.3.9 (only this version, using Incremental Compiler 0.13.9) emits misleading warning:

"Pruning sources from previous analysis, due to incompatible CompileSetup."

even when compiling clean project. This warning comes from

https://github.com/sbt/sbt/blob/v0.13.9/compile/integration/src/main/scala/sbt/compiler/AggressiveCompile.scala#L156

SBT does not emit it in similar context because it doesn't use AggressiveCompiler at all. Thsi class is deprecated since 0.13.8

I think Zinc compilation should be identical to SBT. I would like to fix it (but I don't know how) and release new Zinc versions 0.3.8.2 and 0.3.9. WDYT?
I could help with PRs, but need some pointers.

Regards
Grzegorz Slowikowski

[eugene yokota]

The new sbt 1.0 codebase splits up sbt's incremental compiler into sbt/incrementalcompiler.

Martin Duhem and I have been working on a branch that uses it instead of sbt:

https://github.com/typesafehub/zinc/tree/topic/incrementalcompiler

That might be portable to 0.13.9 if you're interested.

-eugene

--
You received this message because you are subscribed to the Google Groups "sbt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/4f1c6c6c-6046-4ab3-ac6e-63009b6bc183%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grzegorz Słowikowski]

Hi Eugene

Thank you for quick response. I've already seen "incrementalcompiler"
repository, but didn't see this Zinc branch using it. I will investigate it.

I don't know your plans for SBT, but I changed my mind regading new,
fixed releases of Zinc (0.3.8.2 and 0.3.9.1). If 0.3.8.x and 0.3.9
versions of Zinc use AggressiveCompiler, it should not be changed.

I can suppress this misleading warning message in my code because I have
SBT Logger wrapper in my code delegating logs to Maven logging system.
This will be an ugly hack, but perhaps best in this situation.

BTW, shouldn't 'case _' be changed to 'case Some(_)' in this block:
https://github.com/sbt/sbt/blob/v0.13.9/compile/integration/src/main/scala/sbt/compiler/AggressiveCompile.scala#L146-L158

My project using Zinc is here:
https://github.com/sbt-compiler-maven-plugin/sbt-compiler-maven-plugin

Regards
Grzegorz

> <mailto:sbt-dev+u...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sbt-dev/4f1c6c6c-6046-4ab3-ac6e-63009b6bc183%40googlegroups.com

> <https://groups.google.com/d/msgid/sbt-dev/4f1c6c6c-6046-4ab3-ac6e-63009b6bc183%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "sbt-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sbt-dev/WffCi755QhM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sbt-dev+u...@googlegroups.com
> <mailto:sbt-dev+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sbt-dev/CA%2BCq6KRWrcTcxp9nUaCiVcheLRvxhubN1X7xkimE0aLtvUu1Vw%40mail.gmail.com
> <https://groups.google.com/d/msgid/sbt-dev/CA%2BCq6KRWrcTcxp9nUaCiVcheLRvxhubN1X7xkimE0aLtvUu1Vw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Grzegorz Slowikowski]

Hi

I tried to use this version with my Maven plugin. Unfortunately it's not possible now. My list of problems and questions is quite long. Should I write it all here?

Regards
Grzegorz

[eugene yokota]

Note that incrementalcompiler is still work in progress, but sure if you have questions about it you can write it all here.

-eugene

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/e6b600df-d37d-4c58-b0e6-cd64b3d35494%40googlegroups.com.

[Grzegorz Slowikowski]

Hi Eugene

I know, it's work in progress. I'm providing early feedback :)

Blocking issue is unavailability of `sbinary` (version 0.4.2) dependency in Maven central repository (or any other Maven-style repository). I had to temporarily install it locally to move forward. This is new dependency. Is it really required?

In the meantime I found `incrementalcompiler` version `0.1.0-M` published to Maven central repository. I tried to make Zinc compatible with this version, but don't know, what to do with https://github.com/typesafehub/zinc/blob/0af231df47371c70b59c1db2d7735a9a9ffffd42/src/main/scala/com/typesafe/zinc/Compiler.scala#L63
Give me some pointers or fix Zinc (`topic/incrementalcompiler` branch), please.

I have more problems.questions, but would like to be able to build with `incrementalcompiler` version `0.1.0-M1` first.

Regards
Grzegorz

[Martin Duhem]

Hi Grzegorz,

There’s a branch [1] in Zinc’s repository that already works with the incremental compiler version 0.1.0-M1.

Please note that because of an issue in the incremental compiler v 0.1.0-M1, incremental recompilation will most likely not work (the previous analysis gets always pruned).

[1]: https://github.com/typesafehub/zinc/tree/wip/integrate-latest

Cheers,

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/8b587fc6-b97e-44bc-867c-5b1ec8515bdd%40googlegroups.com.

[Grzegorz Slowikowski]

Hi

I used this branch and it works for me (I know that incremental compilation issue will be fixed, so it's not a problem ATM)

First, I would change Zinc version from "0.3.10-SNAPSHOT" to eg. "0.4.0-SNAPSHOT" (or "0.4-SNAPSHOT") because of huge internal changes.

As I wrote before, I have problems with dependencies:
1. `sbinary`
Sbinary is not available in Maven central repository. I had to install it locally. It's used by `incrementalcompiler-persist` module. I wonder where. Analysis cache file format hasn't changed since SBT 0.13.9, `sbinary` is used in classes handling analysis cache persistence, but it wasn't used in 0.13.9. Anyway, I cannot exclude it.
2. zinc's runtime dependency tree is very large comparing it to previous version.

Zinc 0.3.9 deps:
+- com.typesafe.zinc:zinc:jar:0.3.9:compile
   +- org.scala-lang:scala-library:jar:2.10.5:compile
   +- com.typesafe.sbt:incremental-compiler:jar:0.13.9:compile
   |  +- org.scala-lang:scala-compiler:jar:2.10.5:compile
   |  |  \- org.scala-lang:scala-reflect:jar:2.10.5:compile
   |  \- com.typesafe.sbt:sbt-interface:jar:0.13.9:compile
   \- com.typesafe.sbt:compiler-interface:jar:sources:0.13.9:compile

Zinc 0.3.10-SNAPSHOT deps:
+- com.typesafe.zinc:zinc:jar:0.3.10-SNAPSHOT:compile
   +- org.scala-lang:scala-library:jar:2.10.5:compile
   +- org.scala-sbt:incrementalcompiler_2.10:jar:0.1.0-M1:compile
   |  +- org.scala-sbt:incrementalcompiler-core_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:incrementalcompiler-apiinfo_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:incrementalcompiler-classpath_2.10:jar:0.1.0-M1:compile
   |  |  \- org.scala-sbt:util-relation_2.10:jar:0.1.0-M6:compile
   |  +- org.scala-sbt:incrementalcompiler-persist_2.10:jar:0.1.0-M1:compile
   |  |  \- org.scala-tools.sbinary:sbinary_2.10:jar:0.4.2:compile
   |  +- org.scala-sbt:incrementalcompiler-compile-core_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:launcher-interface:jar:1.0.0-M1:compile
   |  |  +- org.scala-sbt:util-logging_2.10:jar:tests:0.1.0-M6:compile
   |  |  \- org.scala-sbt:util-control_2.10:jar:0.1.0-M6:compile
   |  +- org.scala-sbt:incrementalcompiler-classfile_2.10:jar:0.1.0-M1:compile
   |  \- org.scala-sbt:incrementalcompiler-ivy-integration_2.10:jar:0.1.0-M1:compile
   |     \- org.scala-sbt:librarymanagement_2.10:jar:0.1.0-M2:compile
   |        +- org.scala-sbt:io_2.10:jar:tests:1.0.0-M3:compile
   |        +- org.scala-sbt:util-collection_2.10:jar:0.1.0-M3:compile
   |        +- org.scala-sbt.ivy:ivy:jar:2.3.0-sbt-927bc9ded7f8fba63297cddd0d5a3d01d6ad5d8d:compile
   |        +- com.jcraft:jsch:jar:0.1.46:compile
   |        \- org.scala-sbt:serialization_2.10:jar:0.1.2:compile
   |           +- org.scala-lang.modules:scala-pickling_2.10:jar:0.10.1:compile
   |           |  \- org.scalamacros:quasiquotes_2.10:jar:2.0.1:compile
   |           +- org.json4s:json4s-core_2.10:jar:3.2.10:compile
   |           |  +- org.json4s:json4s-ast_2.10:jar:3.2.10:compile
   |           |  \- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
   |           +- org.spire-math:jawn-parser_2.10:jar:0.6.0:compile
   |           \- org.spire-math:json4s-support_2.10:jar:0.6.0:compile
   +- org.scala-sbt:compiler-interface:jar:0.1.0-M1:compile
   |  \- org.scala-sbt:util-interface:jar:0.1.0-M6:compile
   +- org.scala-sbt:compiler-bridge_2.10:jar:sources:0.1.0-M1:compile
   |  +- org.scala-lang:scala-compiler:jar:2.10.5:compile
   |  |  \- org.scala-lang:scala-reflect:jar:2.10.5:compile
   |  +- org.scala-sbt:io_2.10:jar:1.0.0-M3:compile
   |  \- org.scala-sbt:util-logging_2.10:jar:0.1.0-M6:compile
   |     \- jline:jline:jar:2.11:compile
   \- org.scala-sbt:compiler-bridge_2.11:jar:sources:0.1.0-M1:compile
      +- org.scala-sbt:io_2.11:jar:1.0.0-M3:compile
      \- org.scala-sbt:util-logging_2.11:jar:0.1.0-M6:compile

I tried to reduce this tree by excluding subtrees in Maven:
a) org.scala-sbt:incrementalcompiler-ivy-integration_2.10
b) org.scala-sbt:compiler-bridge_2.10
c) org.scala-sbt:compiler-bridge_2.11
and it works, reduced dependency tree:
+- com.typesafe.zinc:zinc:jar:0.3.10-SNAPSHOT:compile
   +- org.scala-lang:scala-library:jar:2.10.5:compile
   +- org.scala-sbt:incrementalcompiler_2.10:jar:0.1.0-M1:compile
   |  +- org.scala-sbt:incrementalcompiler-core_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:incrementalcompiler-apiinfo_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:incrementalcompiler-classpath_2.10:jar:0.1.0-M1:compile
   |  |  |  \- org.scala-lang:scala-compiler:jar:2.10.5:compile
   |  |  |     \- org.scala-lang:scala-reflect:jar:2.10.5:compile
   |  |  +- org.scala-sbt:io_2.10:jar:1.0.0-M3:compile
   |  |  +- org.scala-sbt:util-logging_2.10:jar:0.1.0-M6:compile
   |  |  |  \- jline:jline:jar:2.11:compile
   |  |  \- org.scala-sbt:util-relation_2.10:jar:0.1.0-M6:compile
   |  +- org.scala-sbt:incrementalcompiler-persist_2.10:jar:0.1.0-M1:compile
   |  |  \- org.scala-tools.sbinary:sbinary_2.10:jar:0.4.2:compile
   |  +- org.scala-sbt:incrementalcompiler-compile-core_2.10:jar:0.1.0-M1:compile
   |  |  +- org.scala-sbt:launcher-interface:jar:1.0.0-M1:compile
   |  |  +- org.scala-sbt:util-logging_2.10:jar:tests:0.1.0-M6:compile
   |  |  \- org.scala-sbt:util-control_2.10:jar:0.1.0-M6:compile
   |  \- org.scala-sbt:incrementalcompiler-classfile_2.10:jar:0.1.0-M1:compile
   \- org.scala-sbt:compiler-interface:jar:0.1.0-M1:compile
      \- org.scala-sbt:util-interface:jar:0.1.0-M6:compile

- excluding `incrementalcompiler-ivy-integration` reduced deps tree largely. Is it really needed in runtime?
- what about `compiler-bridge` dependencies, why are they present? They should be resolved dynamically (at least I do so on Maven side)
- what about `jline` (I didn't try to remove it yet)?
- what is `launcher-interface` for?
- `org.scala-sbt:util-logging_2.10:jar:tests:0.1.0-M6` looks weird, this should be test-scoped dependency, but has default scope in pom file (http://repo1.maven.org/maven2/org/scala-sbt/incrementalcompiler-compile-core_2.10/0.1.0-M1/incrementalcompiler-compile-core_2.10-0.1.0-M1.pom)

Comment my findings, please, if you have time.

Cheers
Grzegorz

[Grzegorz Slowikowski]

Hi

I see https://github.com/typesafehub/zinc/tree/topic/incrementalcompiler branch again, using `incrementalcompiler` 0.1.0-M2, but this version is not available in Maven central repo and there is no corresponding tag in GitHub

Regards

Grzegorz

On Monday, November 16, 2015 at 2:21:28 PM UTC+1, Martin Duhem wrote:

[Dale Wijnand]

It's in jCenter: https://jcenter.bintray.com/org/scala-sbt/incrementalcompiler_2.10/0.1.0-M2/

But you're right about the tag in GitHub. Ping Eugene.

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/573fae7a-9ff9-4d73-a4e7-4a6f1b913022%40googlegroups.com.

[Grzegorz Slowikowski]

Hi Dale

I don't want to add additional repositories. Why didn't you push it to Maven central, just like `0.1.0-M1`?

Regards
Grzegorz

[Dale Wijnand]

To be fair, it's a milestone, and the intent is for the stable release to be in Maven Central, so the additional repository would only be temporary. But at the same time, like the git tag, it might have just been forgetfulness, we can just sync it to Maven Central from jCenter.

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/25bbf9bf-1ab1-4adc-b4c6-8736d844fa8a%40googlegroups.com.

[Grzegorz Slowikowski]

Hi Dale

I just wanted to be sure, the final version will be published to Maven central before I will publish my plugin using it. Anyway, I see 0.1.0-M2 already published, thank you.

Grzegorz

[eugene yokota]

Thanks for the detailed report. Not sure if we can address all of the points soon, but pointing these out now is very helpful.

One thing to note is that sbt itself has always consisted of many internal modules.

Zinc uses the fat JAR version of sbt through us republishing the artifact, so now you're seeing more guts.

On Tue, Nov 17, 2015 at 3:36 AM, Grzegorz Slowikowski <gslowi...@gmail.com> wrote:

Hi

I used this branch and it works for me (I know that incremental compilation issue will be fixed, so it's not a problem ATM)

First, I would change Zinc version from "0.3.10-SNAPSHOT" to eg. "0.4.0-SNAPSHOT" (or "0.4-SNAPSHOT") because of huge internal changes.

Yes, we should bump up.

 

As I wrote before, I have problems with dependencies:
1. `sbinary`
Sbinary is not available in Maven central repository. I had to install it locally. It's used by `incrementalcompiler-persist` module. I wonder where. Analysis cache file format hasn't changed since SBT 0.13.9, `sbinary` is used in classes handling analysis cache persistence, but it wasn't used in 0.13.9. Anyway, I cannot exclude it.

This likely means that Zinc is not self-contained within Maven Central until we port some stuff over.

 

2. zinc's runtime dependency tree is very large comparing it to previous version.

This is to be expected. As noted above.

This is an interesting finding. This may have been caused by the way I set up the build.

We should look into what's needed during compile time vs runtime.

There may also be some tension between all-batteries-included (serves both sbt, Zinc, etc) vs minimalist.

My currently inclination is to provide some API point that's easy to understand, and treat all sub modules to be internal details.

 

- excluding `incrementalcompiler-ivy-integration` reduced deps tree largely. Is it really needed in runtime?

Ivy along with other tricks allows sbt to use future versions of Scala compiler. Not sure if it's needed by Zinc.

 

- what about `compiler-bridge` dependencies, why are they present? They should be resolved dynamically (at least I do so on Maven side)

Might not be needed during runtime.

 

- what about `jline` (I didn't try to remove it yet)?

It's used to check for ANSI support:

https://github.com/sbt/util/blob/30c4bc104367817ba0fc8016506f38767fdf78fc/internal/util-logging/src/main/scala/sbt/internal/util/ConsoleLogger.scala#L122-L136

 

- what is `launcher-interface` for?

This describes the interface between the application hosted by the sbt launcher and the launcher.

It defines some of the Java classes used by the incrementalcompiler like ScalaProvider.

 

- `org.scala-sbt:util-logging_2.10:jar:tests:0.1.0-M6` looks weird, this should be test-scoped dependency, but has default scope in pom file (http://repo1.maven.org/maven2/org/scala-sbt/incrementalcompiler-compile-core_2.10/0.1.0-M1/incrementalcompiler-compile-core_2.10-0.1.0-M1.pom)

Could you file a Github issue for it?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/492938c9-5561-4a09-a6c1-2884960bed9c%40googlegroups.com.

[eugene yokota]

Pushed the tag and synced to Maven Central.

There's some delay after I publish artifacts to Bintray and when it becomes available.

And I normally wait until the artifacts become available to sync to Maven Central from Bintray, which is a manual process.

-eugene

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/573fae7a-9ff9-4d73-a4e7-4a6f1b913022%40googlegroups.com.