Hi all,We are trying to upgrade our cluster from 0.7.x to 0.8.1. We are currently using the Hadoop indexer for our batch pipelines (we are using version 0.6.171).It looks like the indexing task spec has changed and I cannot use the specification I am using with the jar 0.6.171 of the Druid project.I have looked at the documentation and updated my indexing task to follow the new specs. However I get this error for the third job of the batch indexing process:2015-11-02 19:11:55,911 ERROR [main] cli.CliHadoopIndexer (Logger.java:error(98)) - failure!!!!java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:120)at io.druid.cli.Main.main(Main.java:91)Caused by: java.lang.RuntimeException: java.lang.RuntimeException: No buckets?? seems there is no data to index.at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:207)at io.druid.indexer.JobHelper.runJobs(JobHelper.java:182)at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96)at io.druid.indexer.JobHelper.runJobs(JobHelper.java:182)at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:132)at io.druid.cli.Main.main(Main.java:91)... 6 moreCaused by: java.lang.RuntimeException: No buckets?? seems there is no data to index.at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:159)1) First of all, can someone explain me what are the purpose of each job in the batch ingestion process? (determine_partitions_groupby, determine_partitions_dimselection ...).
2) What does this error mean? I am sure the interval in granularitySpec is correct because I am using the same interval and the same input path for my data when I ingest data with version 0.6.171.
3) Is it possible to continue ingesting my data with the Hadoop indexer version 0.6.171 whereas my cluster will be 0.8.1?
4) In the metadataUpdateSpec spec I tried to use "mysql" or "db" for the type but it did not work. I am now using derby. What does it stand for? Is it the right way to do it if my metadata storage is MySQL?
I have attached the complete MR logs. Thanks for your help!
SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0-amzn-7/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/hadoop/druid-services/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]2015-11-05 13:57:26,321 INFO [main] util.Version (Version.java:<clinit>(27)) - HV000001: Hibernate Validator 5.1.3.Final2015-11-05 13:57:27,307 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], defaultVersion='0.8.1', localRepository='/home/hadoop/.m2/repository', remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0-amzn-7/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/hadoop/druid-services/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]2015-11-05 13:57:28,117 INFO [main] util.Version (Version.java:<clinit>(27)) - HV000001: Hibernate Validator 5.1.3.Final2015-11-05 13:57:28,923 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], defaultVersion='0.8.1', localRepository='/home/hadoop/.m2/repository', remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]2015-11-05 13:57:29,882 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], defaultVersion='0.8.1', localRepository='/home/hadoop/.m2/repository', remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]2015-11-05 13:57:30,694 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@737361b8]2015-11-05 13:57:30,708 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]2015-11-05 13:57:31,044 INFO [main] config.ConfigurationObjectFactory (ConfigurationObjectFactory.java:buildSimple(162)) - Using method itself for [druid.computation.buffer.size, ${base_path}.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]2015-11-05 13:57:31,049 INFO [main] config.ConfigurationObjectFactory (ConfigurationObjectFactory.java:buildSimple(162)) - Using method itself for [${base_path}.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]2015-11-05 13:57:31,049 INFO [main] config.ConfigurationObjectFactory (ConfigurationObjectFactory.java:buildSimple(162)) - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]2015-11-05 13:57:31,050 INFO [main] config.ConfigurationObjectFactory (ConfigurationObjectFactory.java:buildSimple(151)) - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]2015-11-05 13:57:31,234 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[interface io.druid.segment.data.BitmapSerdeFactory] from props[druid.processing.bitmap.] as [ConciseBitmapSerdeFactory{}]Nov 05, 2015 1:57:31 PM com.google.inject.servlet.GuiceFilter setPipelineWARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.2015-11-05 13:57:31,391 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@77c143b0]2015-11-05 13:57:31,402 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]2015-11-05 13:57:31,422 INFO [main] guice.JsonConfigurator (Logger.java:info(70)) - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/internal-hadoop-indexer', host='ip-10-81-200-79.ec2.internal', port=0}]2015-11-05 13:57:31,428 ERROR [main] cli.CliHadoopIndexer (Logger.java:error(98)) - failure!!!!
java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:120)at io.druid.cli.Main.main(Main.java:91)
Caused by: com.google.inject.ProvisionException: Guice provision errors:1) Unknown provider[mysql] of Key[type=io.druid.metadata.SQLMetadataConnector, annotation=[none]], known options[[derby]]at io.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:67)while locating io.druid.metadata.SQLMetadataConnectorfor parameter 2 at io.druid.metadata.IndexerSQLMetadataStorageCoordinator.<init>(IndexerSQLMetadataStorageCoordinator.java:69)while locating io.druid.metadata.IndexerSQLMetadataStorageCoordinatorat io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:98)while locating io.druid.indexing.overlord.IndexerMetadataStorageCoordinator1 errorat com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:119)
at io.druid.cli.Main.main(Main.java:91)... 6 more
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/f8d4072e-0fd4-4648-a281-486db637c68c%40googlegroups.com.--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.