Re: Druid Master Not Loading Postgresql Extension

[transient1]

I tried combining the loadList into one statement as I have seen in other places with no change—s3 extensions load, postgresql extensions don't. I also tried removing the loadList all together and found that all of the extensions under the dist/extensions directory will get loaded (including postgresql), but then I get a different error:

2017-03-27T17:56:58,963 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:


1) A binding to com.google.common.base.Supplier<io.druid.server.audit.SQLAuditManagerConfig> was already configured at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.metadata.storage.mysql.MySQLMetadataStorageModule)


2) A binding to io.druid.server.audit.SQLAuditManagerConfig was already configured at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:132) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:132) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.metadata.storage.mysql.MySQLMetadataStorageModule)


2 errors
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at com.google.inject.Guice.createInjector(Guice.java:73)
        at com.google.inject.Guice.createInjector(Guice.java:62)
        at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:366)
        at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:62)
        at io.druid.cli.ServerRunnable.run(ServerRunnable.java:39)
        at io.druid.cli.Main.main(Main.java:106)

On Monday, March 27, 2017 at 1:49:30 PM UTC-4, transient1 wrote:

I just deployed a Druid cluster (using Imply). The cluster elements are:

- 3x Zookeeper version 3.4.9

- 2x Kafka version 0.10

- 1x Postgresql server version 9.5.4

- Centos 7

- Java 8

- Imply 2.0.0

  - 2x Masters

  - 2x Data Servers

  - 1x Query Server

I modified common_runtime.properties to look like this:

#
# Extensions
#


druid.extensions.directory=dist/druid/extensions
druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies
#
# Logging
#


# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true


#
# Zookeeper
#


druid.zk.service.host=1.1.1.1,1.1.1.2,1.1.1.3
druid.zk.paths.base=/druid


# For PostgreSQL:
druid.extensions.loadList=["postgresql-metadata-storage"]
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://druid.databaseserver.com:5432/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=password

#

# Deep storage

#

druid.extensions.loadList=["druid-s3-extensions"]

# For local disk (only viable in a cluster if this is a network mount):

#druid.storage.type=local

#druid.storage.storageDirectory=var/druid/segments

# For S3:
druid.storage.type=s3
druid.storage.bucket=value
druid.storage.baseKey=druid/segments
druid.s3.accessKey=value
druid.s3.secretKey=value


#
# Service discovery
#


druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator


#
# Monitoring
#


druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]
druid.emitter=logging
druid.emitter.logging.logLevel=debug

When I run "sudo bin/supervise -c conf/supervise/master-with-zk.conf" I get the following errors in coordinator.log:

2017-03-27T17:33:30,119 ERROR [main] io.druid.cli.CliCoordinator - Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:


1) Unknown provider[postgresql] of Key[type=io.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[derby]]
  at io.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:86) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.metadata.storage.derby.D
erbyMetadataStorageDruidModule)

There are actually 5 instances of similar error, all seeming to say it can't use postgresql. The logs prior to the error shows the extensions being loaded, and postgresql is not among them:

2017-03-27T17:33:26,435 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2017-03-27T17:33:26,439 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2017-03-27T17:33:26,486 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2017-03-27T17:33:27,098 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='d
ist/druid/extensions', hadoopDependenciesDir='dist/druid/hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=[druid-s3-extensions]}]
2017-03-27T17:33:27,100 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-s3-extensions] for class [io.druid.cli.CliCommandCreator]
2017-03-27T17:33:27,109 INFO [main] io.druid.initialization.Initialization - added URL[file:/opt/imply-2.0.0/dist/druid/extensions/druid-s3-extensions/druid-s3-extensions-0.9.2.jar]
2017-03-27T17:33:27,328 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-s3-extensions] for class [io.druid.initialization.DruidModule]
........
2017-03-27T17:33:29,627 INFO [main] io.druid.cli.CliCoordinator - * druid.extensions.directory: dist/druid/extensions
2017-03-27T17:33:29,627 INFO [main] io.druid.cli.CliCoordinator - * druid.extensions.hadoopDependenciesDir: dist/druid/hadoop-dependencies
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.extensions.loadList: ["druid-s3-extensions"]
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.indexer.logs.s3Bucket: druid.prod.choicestream.com
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.indexer.logs.s3Prefix: druid/indexing-logs
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.indexer.logs.type: s3
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.metadata.storage.connector.connectURI: jdbc:postgresql://druid.ceuv8jib5min.us-east-1.rds.amazonaws.com:5432/druid
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.metadata.storage.connector.password: <masked>
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.metadata.storage.connector.user: druid
2017-03-27T17:33:29,628 INFO [main] io.druid.cli.CliCoordinator - * druid.metadata.storage.type: postgresql
.......

It seems like Druid is loading the S3 extension just fine, but the postgresql extension is not being loaded. I've also verified that postgresql-metadata-storage is there under druid/dist/extenstions. 

Any idea what I'm doing wrong here? I can't see anything obvious like a typo.