what is this error in english?

104 views
Skip to first unread message

David Montgomery

unread,
Jun 30, 2015, 4:45:47 AM6/30/15
to druid-de...@googlegroups.com

This is my realtime.spec.  Whats wrong in english?  I added redundancy to the realtime node. 



Thanks

[
    {
        "schema": {
            "shardSpec": {
                "type": "linear",
                "partitionNum": 0

            },
            "dataSource": "datasource-topic-pixel",
            "aggregators": [
                {
                    "type": "count",
                    "name": "pixel"
                }
            ],
            "indexGranularity": "minute"
        },
        "config": {
            "maxRowsInMemory": 500000,
            "intermediatePersistPeriod": "PT10m"
        },
        "firehose": {
            "type": "kafka-0.8",
            "consumerProps": {
                "zookeeper.connect": "1.zk.do.development.sf.test.com:2181",
                "zookeeper.connection.timeout.ms": "15000",
                "zookeeper.session.timeout.ms": "15000",
                "zookeeper.sync.time.ms": "5000",
                "group.id": "topic-pixel-development",
                "fetch.message.max.bytes": "1048586",
                "auto.offset.reset": "largest",
                "auto.commit.enable": "false"
            },
            "feed": "topic-pixel-development",
            "parser": {
                "timestampSpec": {
                    "column": "utcdt",
                    "format": "iso"
                },
                "data": {
                    "format": "json"
                },
                "dimensionExclusions": [
                    "wp"
                ]
            }
        },
        "plumber": {
            "type": "realtime",
            "windowPeriod": "PT10m",
            "segmentGranularity": "hour",
            "basePersistDirectory": "/tmp/realtime/basePersist",
            "rejectionPolicyFactory": {
                "type": "messageTime"
            }
        }
    },
    {
        "schema": {
            "shardSpec": {
                "type": "linear",
                "partitionNum": 0

            },
            "dataSource": "datasource-topic-impression",
            "aggregators": [
                {
                    "type": "count",
                    "name": "impressions"
                }
            ],
            "indexGranularity": "minute"
        },
        "config": {
            "maxRowsInMemory": 500000,
            "intermediatePersistPeriod": "PT10m"
        },
        "firehose": {
            "type": "kafka-0.8",
            "consumerProps": {
                "zookeeper.connect": "1.zk.do.development.sf.test.com:2181",
                "zookeeper.connection.timeout.ms": "15000",
                "zookeeper.session.timeout.ms": "15000",
                "zookeeper.sync.time.ms": "5000",
                "group.id": "topic-impression-development",
                "fetch.message.max.bytes": "1048586",
                "auto.offset.reset": "largest",
                "auto.commit.enable": "false"
            },
            "feed": "topic-impression-development",
            "parser": {
                "timestampSpec": {
                    "column": "utcdt",
                    "format": "iso"
                },
                "data": {
                    "format": "json"
                },
                "dimensionExclusions": [
                    "wp"
                ]
            }
        },
        "plumber": {
            "type": "realtime",
            "windowPeriod": "PT10m",
            "segmentGranularity": "hour",
            "basePersistDirectory": "/tmp/realtime/basePersist",
            "rejectionPolicyFactory": {
                "type": "messageTime"
            }
        }
    },
    {
        "schema": {
            "shardSpec": {
                "type": "linear",
                "partitionNum": 0

            },
            "dataSource": "datasource-topic-click",
            "aggregators": [
                {
                    "type": "count",
                    "name": "click"
                }
            ],
            "indexGranularity": "minute"
        },
        "config": {
            "maxRowsInMemory": 500000,
            "intermediatePersistPeriod": "PT10m"
        },
        "firehose": {
            "type": "kafka-0.8",
            "consumerProps": {
                "zookeeper.connect": "1.zk.do.development.sf.test.com:2181",
                "zookeeper.connection.timeout.ms": "15000",
                "zookeeper.session.timeout.ms": "15000",
                "zookeeper.sync.time.ms": "5000",
                "group.id": "topic-click-development",
                "fetch.message.max.bytes": "1048586",
                "auto.offset.reset": "largest",
                "auto.commit.enable": "false"
            },
            "feed": "topic-click-development",
            "parser": {
                "timestampSpec": {
                    "column": "utcdt",
                    "format": "iso"
                },
                "data": {
                    "format": "json"
                },
                "dimensionExclusions": [
                    "wp"
                ]
            }
        },
        "plumber": {
            "type": "realtime",
            "windowPeriod": "PT10m",
            "segmentGranularity": "hour",
            "basePersistDirectory": "/tmp/realtime/basePersist",
            "rejectionPolicyFactory": {
                "type": "messageTime"
            }
        }
    },
    {
        "schema": {
            "shardSpec": {
                "type": "linear",
                "partitionNum": 0

            },
            "dataSource": "datasource-topic-test",
            "aggregators": [
                {
                    "type": "count",
                    "name": "click"
                }
            ],
            "indexGranularity": "minute"
        },
        "config": {
            "maxRowsInMemory": 500000,
            "intermediatePersistPeriod": "PT10m"
        },
        "firehose": {
            "type": "kafka-0.8",
            "consumerProps": {
                "zookeeper.connect": "1.zk.do.development.sf.test.com:2181",
                "zookeeper.connection.timeout.ms": "15000",
                "zookeeper.session.timeout.ms": "15000",
                "zookeeper.sync.time.ms": "5000",
                "group.id": "topic-click-development",
                "fetch.message.max.bytes": "1048586",
                "auto.offset.reset": "largest",
                "auto.commit.enable": "false"
            },
            "feed": "topic-test-development",
            "parser": {
                "timestampSpec": {
                    "column": "utcdt",
                    "format": "iso"
                },
                "data": {
                    "format": "json"
                },
                "dimensionExclusions": [
                    "wp"
                ]
            }
        },
        "plumber": {
            "type": "realtime",
            "windowPeriod": "PT10m",
            "segmentGranularity": "hour",
            "basePersistDirectory": "/tmp/realtime/basePersist",
            "rejectionPolicyFactory": {
                "type": "messageTime"
            }
        }
    }
]









Jun 30, 2015 8:16:00 AM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.segment.realtime.FireDepartment] value failed: dataSchema (through reference chain: java.util.ArrayList[0])
  at io.druid.guice.FireDepartmentsProvider.<init>(FireDepartmentsProvider.java:41)
  while locating io.druid.guice.FireDepartmentsProvider
  at io.druid.guice.RealtimeModule.configure(RealtimeModule.java:79)
  while locating java.util.List<io.druid.segment.realtime.FireDepartment>
    for parameter 0 at io.druid.segment.realtime.RealtimeMetricsMonitor.<init>(RealtimeMetricsMonitor.java:41)
  while locating io.druid.segment.realtime.RealtimeMetricsMonitor
  at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:78)
  at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:78)
  while locating com.metamx.metrics.MonitorScheduler
  at io.druid.server.metrics.MetricsModule.configure(MetricsModule.java:63)
  while locating com.metamx.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at com.google.inject.Guice.createInjector(Guice.java:73)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:369)
    at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:55)
    at io.druid.cli.ServerRunnable.run(ServerRunnable.java:37)
    at io.druid.cli.Main.main(Main.java:88)
Caused by: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.segment.realtime.FireDepartment] value failed: dataSchema (through reference chain: java.util.ArrayList[0])
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at io.druid.guice.FireDepartmentsProvider.<init>(FireDepartmentsProvider.java:52)
    at io.druid.guice.FireDepartmentsProvider$$FastClassByGuice$$229da177.newInstance(<generated>)
    at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
    at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
    at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.Scopes$1$1.get(Scopes.java:65)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
    at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
    at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
    at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1040)
    at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:81)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:105)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.Scopes$1$1.get(Scopes.java:65)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
    ... 8 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.segment.realtime.FireDepartment] value failed: dataSchema (through reference chain: java.util.ArrayList[0])
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234)
    at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3066)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2122)
    at io.druid.guice.FireDepartmentsProvider.<init>(FireDepartmentsProvider.java:43)
    ... 55 more
Caused by: java.lang.NullPointerException: dataSchema
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229)
    at io.druid.segment.realtime.FireDepartment.<init>(FireDepartment.java:55)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230)
    ... 66 more
Jun 30, 2015 8:16:15 AM org.hibernate.validator.internal.util.Version <clinit>

Fangjin Yang

unread,
Jun 30, 2015, 12:59:33 PM6/30/15
to druid-de...@googlegroups.com, davidmo...@gmail.com
What version of Druid are you using? It looks like you are using a 0.6.x schema on a 0.7.x node.

David Montgomery

unread,
Jul 1, 2015, 12:47:24 AM7/1/15
to druid-de...@googlegroups.com
Then now what is this error I am seeing in my log file?  I updated the realtime spec to 0.7.3.


Jul 01, 2015 4:38:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.17.1 02/28/2013 12:47 PM'
Jul 01, 2015 4:38:26 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
Jul 01, 2015 4:38:28 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider
Jul 01, 2015 4:38:28 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider
Jul 01, 2015 4:38:28 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined"



Thanks



[
  {
    "dataSchema" : {
      "dataSource" : "datasource-topic-pixel",
      "parser" : {
        "type" : "string",
        "parseSpec" : {
          "format" : "json",

          "timestampSpec" : {
            "column": "utcdt",
            "format": "iso"
          },
          "dimensionsSpec" : {
            "dimensions": ["country_name","region","city"],
            "dimensionExclusions" : [],
            "spatialDimensions" : []
          }
        }
      },
      "metricsSpec" : [{

        "type" : "count",
        "name" : "pixel"
      }],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "minute",
        "queryGranularity" : "NONE"
      }
    },
    "ioConfig" : {
      "type" : "realtime",

      "firehose": {
        "type": "kafka-0.8",
        "consumerProps": {
          "zookeeper.connect": "<%=@zookeeper%>",

          "zookeeper.connection.timeout.ms" : "15000",
          "zookeeper.session.timeout.ms" : "15000",
          "zookeeper.sync.time.ms" : "5000",
          "group.id": "topic-pixel-<%=@environment%>",

          "fetch.message.max.bytes" : "1048586",
          "auto.offset.reset": "largest",
          "auto.commit.enable": "false"
        },
        "feed": "topic-pixel-<%=@environment%>"

      },
      "plumber": {
        "type": "realtime"
      }
    },
    "tuningConfig": {
      "shardSpec": {
                "type": "linear",
                "partitionNum": <%=@partitionNum%>
      },
      "type" : "realtime",

      "maxRowsInMemory": 500000,
      "intermediatePersistPeriod": "PT10m",
      "windowPeriod": "PT10m",
      "basePersistDirectory": "\/tmp\/realtime\/basePersist",
      "rejectionPolicy": {
        "type": "serverTime"
      }
    }
  }
]




--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/z_mfO7XUMCk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e5269652-042d-4fc1-ad35-b468a68fc01b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nishant Bangarwa

unread,
Jul 1, 2015, 10:21:58 AM7/1/15
to druid-de...@googlegroups.com
Hi david, 
the previous error was due using older schema for task druid was unable to parse it. 
are you seeing any exception after updating the schema also ? can you share stack trace for that ? 

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACF%2B8rp-azqQqWSJMmspoo5EoB%2B%3DG96tkFWWTrZE8pmD0_E%2BqA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

David Montgomery

unread,
Jul 1, 2015, 10:52:03 AM7/1/15
to druid-de...@googlegroups.com
Hi,

The error I am getting now is posted here.  https://groups.google.com/forum/#!topic/druid-development/CsVdfAipYoE

Seems I am having issues with s3 I guess.  It was working with 0.6.x.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACW6nteif9f97NfvZLDbT1VjsHzwHHDnR_RCHezp7N%3DpVivEoA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages