Listing transitive dependencies, with version info

Michael Slinn

unread,

Jul 12, 2012, 4:56:53 PM7/12/12

to simple-b...@googlegroups.com

I want to list out all dependencies for an SBT project, with version information for each dependency. Discovering direct dependencies is easy; the first part of the script below does that well. Seems that 'show all-dependencies' does not show transitive dependencies. Is there an SBT task I can run that returns the same information for transitive dependencies as all-dependencies does for direct dependencies? Right now I just dump out the managed-classpath, which is not as useful.

#!/bin/bash

echo "Direct dependencies"
sbt 'show all-dependencies' | gawk 'match($0, /List$(.*)$/, a) {print a[1]}' | tr -d ' ' | tr ',' '\n' | sort -t ':' | \
tr ':' '\t' | expand -t 30

echo -e "\nAll dependencies, including transitive dependencies"
sbt 'show managed-classpath' | tr -d ' ' | tr ',' '\n' | gawk 'match($0, /Attributed$(.*)$/, a) {print a[1]}' | \
tr -d '()' | sed "s^$HOME/.ivy2/cache/^^g" | sed "s^/jars^^" | \
gawk -F / '{print $1, $3}' | sort | tr ' ' '\t' | expand -t 30

Sample output:

Direct dependencies
ch.qos.logback                logback-classic               1.0.0
com.amazonaws                 aws-java-sdk                  1.1.8
com.google.protobuf           protobuf-java                 2.4.1
com.novus                     salat-core                    1.9.0
com.typesafe.akka             akka-actor                    2.0.1
org.apache.solr               solr-solrj                    3.6.0
org.scala-lang                scala-library                 2.9.1

All dependencies, including transitive dependencies
ch.qos.logback                logback-classic-1.0.0.jar
ch.qos.logback                logback-core-1.0.0.jar
com.amazonaws                 aws-java-sdk-1.1.8.jar
com.google.protobuf           protobuf-java-2.4.1.jar
commons-codec                 commons-codec-1.6.jar
commons-httpclient            commons-httpclient-3.1.jar
commons-io                    commons-io-2.1.jar
commons-logging               commons-logging-1.1.1.jar
com.novus                     salat-core_2.9.1-1.9.0.jar
com.novus                     salat-util_2.9.1-1.9.0.jar
com.thoughtworks.paranamer    paranamer-2.4.1.jar
com.typesafe.akka             akka-actor-2.0.1.jar
javax.activation              activation-1.1.jar
javax.mail                    mail-1.4.5.jar
joda-time                     joda-time-1.6.2.jar
net.liftweb                   lift-json_2.9.1-2.5-SNAPSHOT.jar
org.apache.solr               solr-solrj-3.6.0.jar
org.codehaus.jackson          jackson-core-asl-1.9.8.jar
org.codehaus.woodstox         wstx-asl-3.2.7.jar
org.mongodb                   casbah-commons_2.9.1-2.4.1.jar
org.mongodb                   casbah-core_2.9.1-2.4.1.jar
org.mongodb                   casbah-gridfs_2.9.1-2.4.1.jar
org.mongodb                   casbah-query_2.9.1-2.4.1.jar
org.mongodb                   casbah-util_2.9.1-2.4.1.jar
org.mongodb                   mongo-java-driver-2.8.0.jar
org.scalaj                    scalaj-collection_2.9.1-1.2.jar
org.scala-lang                scalap-2.9.1.jar
org.scala-tools.time          time_2.9.1-0.5.jar
org.slf4j                     jcl-over-slf4j-1.6.1.jar
org.slf4j                     slf4j-api-1.6.4.jar
org.specs2                    specs2_2.9.1-1.7.1.jar
org.specs2                    specs2-scalaz-core_2.9.1-6.0.1.jar
stax                          stax-api-1.0.1.jar

Mike

Jason Zaugg

unread,

Jul 12, 2012, 5:36:03 PM7/12/12

to simple-b...@googlegroups.com

On Thursday, July 12, 2012 10:56:53 PM UTC+2, Michael Slinn wrote:

I want to list out all dependencies for an SBT project, with version information for each dependency. Discovering direct dependencies is easy; the first part of the script below does that well. Seems that 'show all-dependencies' does not show transitive dependencies. Is there an SBT task I can run that returns the same information for transitive dependencies as all-dependencies does for direct dependencies? Right now I just dump out the managed-classpath, which is not as useful.

Have you see sbt-dependency-graph? [1]

-jason

[1] https://github.com/jrudolph/sbt-dependency-graph/

Mike Slinn

unread,

Jul 12, 2012, 6:44:53 PM7/12/12

to simple-b...@googlegroups.com

No, I have not, thank you. I tried to use it, but failed. I posted questions. BTW, it really pulls in a lot of dependencies!

Mike

nafg

unread,

Jul 12, 2012, 9:26:00 PM7/12/12

to simple-b...@googlegroups.com

It shouldn't be necessary to install a plugin and examine graphs to just get a complete, hierarchical listing of all dependencies used.

Josh Suereth

unread,

Jul 12, 2012, 9:44:00 PM7/12/12

to simple-b...@googlegroups.com

Why not? That's the maven way (maven dependency plugin)

You could read the ivy update report yourself. Let me check my build for code and add that here....

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/RB8BqyGGlzkJ.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

Mike Slinn

unread,

Jul 12, 2012, 10:02:22 PM7/12/12

to simple-b...@googlegroups.com

Lest we get off-topic:

Is there an SBT task I can run that returns the same information for transitive dependencies as all-dependencies does for direct dependencies?

Mike

Jason Zaugg

unread,

Jul 13, 2012, 2:13:46 AM7/13/12

to simple-b...@googlegroups.com

On Friday, July 13, 2012 4:02:22 AM UTC+2, Michael Slinn wrote:

Lest we get off-topic:

Is there an SBT task I can run that returns the same information for transitive dependencies as all-dependencies does for direct dependencies?

I think all the info you need is in external-dependency-classpath.

> show scalacheck-binding/compile:external-dependency-classpath

[info] ArrayBuffer(Attributed(dummy), Attributed(/Users/jason/.sbt/0.12.0-RC2/boot/scala-2.9.2/lib/scala-library.jar), Attributed(/Users/jason/.ivy2/cache/org.scala-tools.testing/scalacheck_2.9.1/jars/scalacheck_2.9.1-1.9.jar), Attributed(/Users/jason/.ivy2/cache/org.scala-tools.testing/test-interface/jars/test-interface-0.5.jar))

Here's an earlier thread that describes how to extract values from the entries of type Attributed[File].

https://groups.google.com/d/msg/simple-build-tool/tQWOKpLYElM/WJY485poFaMJ

-jason

Mike Slinn

unread,

Jul 13, 2012, 3:26:52 AM7/13/12

to simple-b...@googlegroups.com

Jason,

external-dependency-classpath is like managed-classpath, in that it does not provide decoded versioning information. all-dependencies is special that way. Is there an expanded version of all-dependencies, that truly does show all dependencies?

Mike

Michael Slinn

unread,

Jul 13, 2012, 3:42:13 AM7/13/12

to simple-b...@googlegroups.com

Not sure if my last statement is correct, because I'm unsure if console-project is a necessary plugin to do what you seem to be implying. This research project is growing in scope, instead of converging to a solution. Would you be kind enough to break down 'for dummies' exactly what one would need to do in order to list all dependencies, with versioning? I'm looking for a result like:

ch.qos.logback                logback-classic               1.0.0
com.amazonaws                 aws-java-sdk                  1.1.8
com.google.protobuf           protobuf-java                 2.4.1
com.novus                     salat-core                    1.9.0
com.typesafe.akka             akka-actor                    2.0.1
org.apache.solr               solr-solrj                    3.6.0    compile
org.scala-lang                scala-library                 2.9.1

I think that the solution would be of general interest.

Thanks,

Mike

Jason Zaugg

unread,

Jul 13, 2012, 3:46:40 AM7/13/12

to simple-b...@googlegroups.com

On Friday, July 13, 2012 9:26:52 AM UTC+2, Michael Slinn wrote:

Jason,

external-dependency-classpath is like managed-classpath, in that it does not provide decoded versioning information. all-dependencies is special that way. Is there an expanded version of all-dependencies, that truly does show all dependencies?

Attributed[File] is a (File, Map[Key, Value]). For managed dependencies, you can extract the module ID of each entry.

> set TaskKey[Seq[ModuleID]]("ex") <<= (externalDependencyClasspath in Compile) map (cp => cp.flatMap(_.get(Keys.moduleID.key)))

[info] Reapplying settings...

[info] Set current project to analytics-utils (in build file:/C:/code/analytics/)

> show ex

[info] ArrayBuffer(org.scala-lang:scala-library:2.9.2, org.scalaz:scalaz-core_2.9.1:6.0.3, com.google.guava:guava:11.0.2, com.google.code.findbugs:jsr305:1.3.9, fastutil:fastutil:5.1.5, joda-time:joda-time:1.6, org.slf4j:slf4j-api:1.6.4, ch.qos.logback:logback-classic:1.0.2, ch.qos.logback:logback-core:1.0.2, org.sat4j:org.sat4j.core:2.3.1, com.efgfp.commons:spring-wrapper:0.18, com.efgfp.commons:efg-commons-lang:0.5, org.springframework:spring-core:2.5.6, commons-logging:commons-logging:1.1.1, commons-lang:commons-lang:2.4, commons-io:commons-io:1.4)

> set TaskKey[Seq[String]]("ex-versions") <<= TaskKey[Seq[ModuleID]]("ex") map ((ms: Seq[ModuleID]) => ms.map(_.revision))

[info] Reapplying settings...

[info] Set current project to analytics-utils (in build file:/C:/code/analytics/)

> show ex-versions

[info] ArrayBuffer(2.9.2, 6.0.3, 11.0.2, 1.3.9, 5.1.5, 1.6, 1.6.4, 1.0.2, 1.0.2, 2.3.1, 0.18, 0.5, 2.5.6, 1.1.1, 2.4, 1.4)

-jason

Jason Zaugg

unread,

Jul 13, 2012, 4:38:10 AM7/13/12

to simple-b...@googlegroups.com

On Friday, July 13, 2012 9:42:13 AM UTC+2, Michael Slinn wrote:

Not sure if my last statement is correct, because I'm unsure if console-project is a necessary plugin to do what you seem to be implying.

console-project is a built-in SBT command. It allows you to experiment with your project's settings in a REPL.

This research project is growing in scope, instead of converging to a solution. Would you be kind enough to break down 'for dummies' exactly what one would need to do in order to list all dependencies, with versioning? I'm looking for a result like:
ch.qos.logback                logback-classic               1.0.0
com.amazonaws                 aws-java-sdk                  1.1.8
com.google.protobuf           protobuf-java                 2.4.1
com.novus                     salat-core                    1.9.0
com.typesafe.akka             akka-actor                    2.0.1
org.apache.solr               solr-solrj                    3.6.0    compile
org.scala-lang                scala-library                 2.9.1

lazy val versionReport = TaskKey[String]("version-report")

// Add this setting to your project.

versionReport <<= (externalDependencyClasspath in Compile, streams) map {

(cp: Seq[Attributed[File]], streams) =>

val report = cp.map {

attributed =>

attributed.get(Keys.moduleID.key) match {

case Some(moduleId) => "%40s %20s %10s %10s".format(

moduleId.organization,

moduleId.name,

moduleId.revision,

moduleId.configurations.getOrElse("")

)

case None =>

// unmanaged JAR, just

attributed.data.getAbsolutePath

}

}.mkString("\n")

streams.log.info(report)

report

}

> analytics-utils/version-report

[info] org.scala-lang scala-library 2.9.2

[info] org.scalaz scalaz-core_2.9.1 6.0.3

[info] com.google.guava guava 11.0.2

[info] com.google.code.findbugs jsr305 1.3.9

[info] fastutil fastutil 5.1.5

[info] joda-time joda-time 1.6

[info] org.slf4j slf4j-api 1.6.4

[info] ch.qos.logback logback-classic 1.0.2

[info] ch.qos.logback logback-core 1.0.2

[info] org.sat4j org.sat4j.core 2.3.1

[info] com.efgfp.commons spring-wrapper 0.18

[info] com.efgfp.commons efg-commons-lang 0.5

[info] org.springframework spring-core 2.5.6

[info] commons-logging commons-logging 1.1.1

[info] commons-lang commons-lang 2.4

[info] commons-io commons-io 1.4

Mike Slinn

unread,

Jul 13, 2012, 12:24:21 PM7/13/12

to simple-b...@googlegroups.com

Jason,

Awesome! Many thanks for taking the time to write this out. It would be really good it this was turned into a plugin that was available from a standard location. It would be great if it was incorporated into SBT as a standard command.

Suggestions:

add a column that displays the repo name that each dependency came from
add an extra column for scope (blank, compile, test, etc)
make the columns auto-size their width

I've got to fight fires for a few days, will revisit this once the fires are extinguished.

Thanks again,

Mike

Michael Slinn

unread,

Jul 13, 2012, 3:22:43 PM7/13/12

to simple-b...@googlegroups.com

Oops, I see that my first request is actually in the code you wrote (moduleId.configurations).

Mike

Naftoli Gugenheim

unread,

Jul 13, 2012, 3:29:11 PM7/13/12

to simple-b...@googlegroups.com

On Thu, Jul 12, 2012 at 9:44 PM, Josh Suereth <joshua....@gmail.com> wrote:

Why not? That's the maven way (maven dependency plugin)

Because it seems to me that the tool whose job it is to hold the list of dependencies, could trivially expose that information as text, instead of having (a) the hassle of installing a plugin, (b) the waste of fetching that plugin's dependencies, and (c) the complexity of generating and opening a graphic.