Unable to use download avro extension

[Jagadeesh M]

I see the following extension while downloading avro extention. And why is the process forcing me to delete the existing extensions and hadoop-extensions directory? Should it not simply override if they already exist ?

(eng):/apps/home/eng/eagle-eyes/druid-0.9.0>clear
eng):/apps/home/eng/eagle-eyes/druid-0.9.0>java -classpath "lib/*" io.druid.cli.Main tools pull-deps -c io.druid.extensions:druid-avro-extensions:0.9.0
May 02, 2016 3:56:01 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2016-05-02T15:56:02,036 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', loadList=null}]
2016-05-02T15:56:02,042 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-avro-extensions] for class [io.druid.cli.CliCommandCreator]
Exception in thread "main" com.metamx.common.ISE: Unable to create extensions directory at [/apps/home/eng/eagle-eyes/druid-0.9.0/extensions]
        at io.druid.cli.PullDependencies.createRootExtensionsDirectory(PullDependencies.java:453)
        at io.druid.cli.PullDependencies.run(PullDependencies.java:223)
        at io.druid.cli.Main.main(Main.java:105)
(eng):/apps/home/eng/eagle-eyes/druid-0.9.0>rm -r hadoop-dependencies/
(eng):/apps/home/eng/eagle-eyes/druid-0.9.0>rm -r extensions/
(eng):/apps/home/eng/eagle-eyes/druid-0.9.0>java -classpath "lib/*" io.druid.cli.Main tools pull-deps -c io.druid.extensions:druid-avro-extensions:0.9.0
May 02, 2016 3:56:18 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2016-05-02T15:56:18,705 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', loadList=null}]
2016-05-02T15:56:18,927 INFO [main] io.druid.cli.PullDependencies - Start pull-deps with local repository [/apps/home/eng/.m2/repository] and remote repositories [[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]]
2016-05-02T15:56:18,927 INFO [main] io.druid.cli.PullDependencies - Start downloading dependencies for extension coordinates: [[io.druid.extensions:druid-avro-extensions:0.9.0]]
2016-05-02T15:56:18,933 INFO [main] io.druid.cli.PullDependencies - Start downloading extension [io.druid.extensions:druid-avro-extensions:jar:0.9.0]
2016-05-02T15:56:19,164 ERROR [main] io.druid.cli.PullDependencies - Unable to resolve artifacts for [io.druid.extensions:druid-avro-extensions:jar:0.9.0 (runtime) -> [] < [ (https://repo1.maven.org/maven2/, releases+snapshots),  (https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local, releases+snapshots)]].
java.lang.NullPointerException
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:361) ~[aether-impl-0.9.0.M2.jar:?]
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289) ~[tesla-aether-0.0.5.jar:0.0.5]
        at io.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:341) [druid-services-0.9.0.jar:0.9.0]
        at io.druid.cli.PullDependencies.run(PullDependencies.java:240) [druid-services-0.9.0.jar:0.9.0]
        at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.0.jar:0.9.0]
Exception in thread "main" java.lang.NullPointerException
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:361)
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)
        at io.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:341)
        at io.druid.cli.PullDependencies.run(PullDependencies.java:240)
        at io.druid.cli.Main.main(Main.java:105)

Thanks!

[Jakub Liska]

Hey,

I have the exact same problem and question. In order to install graphite extension I do this : 

# Install graphite extension

WORKDIR /root/imply-$implyversion

RUN java -classpath "/root/imply-$implyversion/dist/druid/lib/*" io.druid.cli.Main tools pull-deps --defaultVersion $druidversion -c io.druid.extensions.contrib:graphite-emitter

ADD extensions/graphite-emitter /root/imply-$implyversion/dist/druid/extensions/graphite-emitter

RUN rm -rf extensions hadoop-dependencies

Since pull-deps script cannot "add" an extension to an existing extension directory I have to download it somewhere else and then copy it over there.

[Jagadeesh M]

I did the same. Was looking at the code, and they somehow throw an extension if the directory already exist. I believe they should change it.

[Fangjin Yang]

This is a known problem with the current extensions system.

The best workaround and how to use extensions are documented here: http://imply.io/docs/latest/extensions

[Jagadeesh M]

Thanks Fangjin! Honestly, I just figured this few hours ago. Was planning to deploy imply to our test AWS instances and noticed all the extensions are already packaged with imply. But thanks for the update!