Writing plugins that leverages dependnecy resulotion to retrieve (& run) 3rd party components

[hochgi]

Hi,

I'm trying to retrieve artifacts for 3rd party components via sbt's dependency resulotion,
in order to launch during tests (but not only), and test against.

My first go at this was with my sbt-cassandra plugin (which I plan to update soon),
and I managed to get by with just getting cassandra on the classpath, since the artifact is a tar.gz,
and thus won't cause trouble at runtime.

But the pattern I used was repeated, and things broke.
Please take a look at the following line:
https://github.com/thomsonreuters/CM-Well/blob/aed540727f775fd5bfbab58e6d710bb3d2d9f506/server/project/sbt-zookeeper-plugin/src/main/scala/SbtZookeeperPlugin.scala#L110-L113

ZooKeeper was first added as a regular dependency, which caused a mess in our logging, among other things...
I tried following what Eugene did here: https://stackoverflow.com/questions/18611316/multiple-executable-jar-files-with-different-external-dependencies-from-a-single
But no luck. I get errors like:

Reference to undefined setting:

  bg/zk:managedClasspath from bg/zk:startZookeeper ((com.github.israel.sbt.zookeeper.SbtZookeeperPlugin) SbtZookeeperPlugin.scala:110)
 
    at sbt.Init$class.Uninitialized(Settings.scala:270)
    at sbt.Def$.Uninitialized(Def.scala:10)
    at sbt.Init$class.delegate(Settings.scala:191)
    at sbt.Def$.delegate(Def.scala:10)
    at sbt.Init$class.compiled(Settings.scala:138)
    at sbt.Def$.compiled(Def.scala:10)
    at sbt.Init$class.make(Settings.scala:144)
    at sbt.Def$.make(Def.scala:10)
    at sbt.Load$$anonfun$8.apply(Load.scala:161)
    at sbt.Load$$anonfun$8.apply(Load.scala:156)
    at sbt.Load$.timed(Load.scala:1025)
    at sbt.Load$.apply(Load.scala:156)
    at sbt.Load$.defaultLoad(Load.scala:39)
    at sbt.BuiltinCommands$.liftedTree1$1(Main.scala:548)
    at sbt.BuiltinCommands$.doLoadProject(Main.scala:548)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:540)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:540)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:59)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:59)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
    at sbt.Command$.process(Command.scala:93)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
    at sbt.State$$anon$1.doX$1(State.scala:183)
    at sbt.State$$anon$1.process(State.scala:190)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.MainLoop$.next(MainLoop.scala:96)
    at sbt.MainLoop$.run(MainLoop.scala:89)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:68)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:63)
    at sbt.Using.apply(Using.scala:24)
    at sbt.MainLoop$.runWithNewLog(MainLoop.scala:63)
    at sbt.MainLoop$.runAndClearLast(MainLoop.scala:46)
    at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:30)
    at sbt.MainLoop$.runLogged(MainLoop.scala:22)
    at sbt.StandardMain$.runManaged(Main.scala:109)
    at sbt.xMain.run(Main.scala:38)
    at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
    at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
    at xsbt.boot.Launch$.run(Launch.scala:109)
    at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
    at xsbt.boot.Launch$.launch(Launch.scala:117)
    at xsbt.boot.Launch$.apply(Launch.scala:18)
    at xsbt.boot.Boot$.runImpl(Boot.scala:41)
    at xsbt.boot.Boot$.main(Boot.scala:17)
    at xsbt.boot.Boot.main(Boot.scala)
[error] Reference to undefined setting:
[error]
[error]   bg/zk:managedClasspath from bg/zk:startZookeeper ((com.github.israel.sbt.zookeeper.SbtZookeeperPlugin) SbtZookeeperPlugin.scala:110)
[error] 
[error] Use 'last' for the full log.

Also read explanations from traveling through the 4th dimension with sbt 0.13 but couldn't figure out how to do it.
The code I tried (actually I tried many things, this is just one example) was:

val ZK = config("zk").describedAs("ZooKeeper Dependencies required for the plugin.")

override def projectSettings = Seq(
    ...
    zookeeperVersion := "3.4.9",
    libraryDependencies += "org.apache.zookeeper" % "zookeeper" % zookeeperVersion.value % ZK,
    ...
  ) +: inConfig(ZK)(Seq(
    ...
    startZookeeper := {
      ...
      val classpath = Attributed.data(managedClasspath.value)
      val cp = classpath.map(_.getAbsolutePath).mkString(System.getProperty("path.separator"))
      val javaExec = Seq(System.getProperty("java.home"),"bin","java").mkString(System.getProperty("file.separator"))
      val mainClass = "org.apache.zookeeper.server.quorum.QuorumPeerMain"
      val pb = new java.lang.ProcessBuilder(javaExec, "-classpath", cp, mainClass, configFile).inheritIO().directory(baseDir)
      zookeeperProcess = pb.start()
      ...
    },
    ...
  ))

Which yields the above error.
Any hints on how to do it the right way?

Thanks,
Gilad.

[Mirco Dotta]

Hey,

I believe you might just be missing `ivyConfigurations += ZK` in the project settings. By the way, the Lagom sbt-plugin may be a good source of inspiration as they do similar things https://github.com/lagom/lagom/blob/master/dev/sbt-plugin/src/main/scala/com/lightbend/lagom/sbt/LagomPlugin.scala

Cheers,

Mirco