Druid on GCS Missing Config Key

[th...@spotify.com]

Hi Folks,

I'm trying to use the GCS connector to link the Druid indexer with GS. 

I have specified the fs.gs.project.id in the core-site.xml file, but I'm still receiving the below error. Does anyone have some insight? 

Thanks so much!

2017-09-01T16:53:23,908 INFO [main] io.druid.storage.google.GoogleStorageDruidModule - Configuring GoogleStorageDruidModule...

Sep 01, 2017 4:53:23 PM com.google.inject.internal.MessageProcessor visit

INFO: An exception was caught and reported. Message: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

java.lang.RuntimeException: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

        at com.google.common.base.Throwables.propagate(Throwables.java:160)

        at io.druid.indexer.HadoopDruidIndexerConfig.fromDistributedFileSystem(HadoopDruidIndexerConfig.java:205)

        at io.druid.cli.CliInternalHadoopIndexer.getHadoopDruidIndexerConfig(CliInternalHadoopIndexer.java:164)

        at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:90)

        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

        at com.google.inject.spi.Elements.getElements(Elements.java:110)

        at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)

        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

        at com.google.inject.spi.Elements.getElements(Elements.java:110)

        at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)

        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

        at com.google.inject.spi.Elements.getElements(Elements.java:110)

        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)

        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)

        at com.google.inject.Guice.createInjector(Guice.java:99)

        at com.google.inject.Guice.createInjector(Guice.java:73)

        at com.google.inject.Guice.createInjector(Guice.java:62)

        at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:370)

        at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:63)

        at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:109)

        at io.druid.cli.Main.main(Main.java:108)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:117)

        at io.druid.cli.Main.main(Main.java:108)

Caused by: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

        at com.google.cloud.hadoop.util.ConfigurationUtil.getMandatoryConfig(ConfigurationUtil.java:39)

        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.createOptionsBuilderFromConfig(GoogleHadoopFileSystemBase.java:2175)

        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1822)

        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:1003)

        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:966)

        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2316)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)

        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)

        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)

        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

        at io.druid.indexer.HadoopDruidIndexerConfig.fromDistributedFileSystem(HadoopDruidIndexerConfig.java:193)

        ... 27 more

[Gian Merlino]

Hmm, I'm not sure why it wouldn't be picked up from there, but maybe try adding it to the jobProperties too?

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/641cb14a-3eb7-46cb-b735-dba57cd043a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Erik Dubbelboer]

We use the following core-site.xml which works fine for us: https://gist.github.com/erikdubbelboer/ef724d68b77c234a913058d903d2bcf9

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.