2015-01-31 18:37:34,607 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_rtb_auctions_2015-01-31T18:37:26.116-08:00, type=index_hadoop, dataSource=rtb_auctions}] java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:227) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:218) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:197) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2379) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:505) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPaths(FileInputFormat.java:470) at io.druid.indexer.path.StaticPathSpec.addInputPaths(StaticPathSpec.java:58) at io.druid.indexer.HadoopDruidIndexerConfig.addInputPaths(HadoopDruidIndexerConfig.java:312) at io.druid.indexer.JobHelper.ensurePaths(JobHelper.java:123) at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:55) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:324) ... 11 more Caused by: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 25 more 2015-01-31 18:37:34,612 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_hadoop_rtb_auctions_2015-01-31T18:37:26.116-08:00", "status" : "FAILED", "duration" : 2893 }
I am not sure of what is happening, any suggestion? Do I have to re compile Druid with Hadoop 2.4.0 as Druid comes with Hadoop 2.3? It seems Hadoop 2.3 is not supported by Amazon...
I am also not sure if I set the hadoopCoordinates properly. This hadoop coordinates refer to the regular Hadoop distribution, however I am using the Amazon Hadoop distribution...
I tried to find a maven repository for Hadoop EMR 2.4.0 but impossible to find it. Did anyone have successfully set up a batch pipeline with EMR/Druid and can help me with that?
Thanks!
Guillaume
--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e637dcfe-e95e-4eaf-8685-d11e80e6772a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Fangjin,
No I'm not including it. I specified the hadoop 2.4 maven repo for the hadoop task as specified in the documentation. Isn't it the way we tell the middle manager where to find hadoop classes it needs to launch the job?
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/la4XtqYuaLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAKyF60KQeWkt9%3DQZL25%3D7%2BnK3X6f3ThL2PhO8JeUkhWNMyq7aQ%40mail.gmail.com.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e637dcfe-e95e-4eaf-8685-d11e80e6772a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/la4XtqYuaLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/ade95d72-b18f-4223-86d4-c7a2d4076771%40googlegroups.com.To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e637dcfe-e95e-4eaf-8685-d11e80e6772a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/la4XtqYuaLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAKyF60KQeWkt9%3DQZL25%3D7%2BnK3X6f3ThL2PhO8JeUkhWNMyq7aQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/ade95d72-b18f-4223-86d4-c7a2d4076771%40googlegroups.com.--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/la4XtqYuaLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b79ea078-7f21-4f4c-8570-495a5b62c736%40googlegroups.com.
16/12/22 02:38:03 INFO config.ConfigurationObjectFactory: Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()]
16/12/22 02:38:03 INFO config.ConfigurationObjectFactory: Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServ
iceConfig#getFormatString()]
Dec 22, 2016 2:38:03 AM com.google.inject.internal.MessageProcessor visit
INFO: An exception was caught and reported. Message: java.lang.NullPointerException
java.lang.NullPointerException
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
at com.google.inject.spi.Elements.getElements(Elements.java:101)
at com.google.inject.spi.Elements.getElements(Elements.java:92)
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:172)
at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
at com.google.inject.spi.Elements.getElements(Elements.java:101)
at com.google.inject.spi.Elements.getElements(Elements.java:92)
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:152)
at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
at com.google.inject.spi.Elements.getElements(Elements.java:101)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367)
at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60)
at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108)
at io.druid.cli.Main.main(Main.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115)
at io.druid.cli.Main.main(Main.java:105)
16/12/22 02:38:03 ERROR cli.CliHadoopIndexer: failure!!!!
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115)
at io.druid.cli.Main.main(Main.java:105)
Caused by: com.google.inject.CreationException: Guice creation errors:
1) An exception was caught and reported. Message: null
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:172)
2) Binding to null instances is not allowed. Use toProvider(Providers.of(null)) if this is your intended behaviour.
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:93)
3) Binding to null instances is not allowed. Use toProvider(Providers.of(null)) if this is your intended behaviour.
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:94)
4) Could not find a suitable constructor in io.druid.metadata.MetadataStorageTablesConfig. Classes must have either one (and only one) constructor annotated with @Inject or a
zero-argument constructor that is not private.
at io.druid.metadata.MetadataStorageTablesConfig.class(MetadataStorageTablesConfig.java:34)
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)
4 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
at com.google.inject.Guice.createInjector(Guice.java:95)