realtime node fails during startup

[Ram Ganesan]

Can anyone help on solving this problem.

--------------

2015-03-15T19:30:58,366 ERROR [main] io.druid.cli.CliRealtime - Error when starting up.  Failing.

com.google.inject.ProvisionException: Guice provision errors:

1) Error injecting constructor, java.lang.NullPointerException

  at io.druid.guice.FireDepartmentsProvider.<init>(FireDepartmentsProvider.java:41)

  while locating io.druid.guice.FireDepartmentsProvider

  at io.druid.guice.RealtimeModule.configure(RealtimeModule.java:78)

  while locating java.util.List<io.druid.segment.realtime.FireDepartment>

    for parameter 0 at io.druid.segment.realtime.RealtimeManager.<init>(RealtimeManager.java:84)

  while locating io.druid.segment.realtime.RealtimeManager

  at io.druid.guice.RealtimeModule.configure(RealtimeModule.java:82)

  while locating io.druid.query.QuerySegmentWalker

    for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:89)

  while locating io.druid.server.QueryResource

1 error

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134) ~[druid-api-0.3.4.jar:0.7.0]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.0.jar:0.7.0]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.0.jar:0.7.0]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.0.jar:0.7.0]

Caused by: java.lang.NullPointerException

at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.7.0_76]

at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:708) ~[jackson-core-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2122) ~[jackson-databind-2.4.4.jar:2.4.4]

at io.druid.guice.FireDepartmentsProvider.<init>(FireDepartmentsProvider.java:43) ~[druid-server-0.7.0.jar:0.7.0]

at io.druid.guice.FireDepartmentsProvider$$FastClassByGuice$$229da177.newInstance(<generated>) ~[guice-4.0-beta.jar:0.7.0]

at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49) ~[druid-api-0.3.4.jar:0.7.0]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

[Gian Merlino]

Hi Ram, have you set the property "druid.realtime.specFile"?

[Eric Zheng]

Hi, Ram. I have encountered this problem before and I suggest you check your shell of starting realtime node.

[Ram Ganesan]

Hi Gian,

Thanks for the pointer.

It works with spec file, we are currently using tranquility for realtime ingestion. Do you see a need for realtime node while using tranquility? 

a) If realtime nodes are not required, then how will they sync for real time queries and segment hand-offs. 

a) If required, what would be my spec file for realtime nodes? (any examples would greatly help)

b) Do you recommend a better architecture for the same use case. [we send events from multiple sources into kafka queue , then via storm + tranquility --> overlord via ZK]

Please see the basic bolt config for tranquility below.

---------------------- Tranquility bolt --------------

public class MyBeamFactory implements BeamFactory {

    @Override

    public Beam makeBeam(Map map, backtype.storm.task.IMetricsContext iMetricsContext) {

        CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient(map.get("tranquility.zk.connect").toString(), new BoundedExponentialBackoffRetry(100, 1000, 5));

        curatorFramework.start();

        String indexService = "druid:prod:overlord"; // Your overlord's druid.service, with slashes replaced by colons.

        String firehosePattern = "druid:prod:firehose:%s"; // Make up a service pattern, include %s somewhere in it.

        String discoveryPath = "/druid/prod/discovery"; // Your overlord's druid.discovery.curator.path.

        String dataSource = "wikipedia";

        List<String> dimensions = ImmutableList.of("page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city");

        List<AggregatorFactory> aggregators = ImmutableList.<AggregatorFactory> of(

                new CountAggregatorFactory("count"),

                new DoubleSumAggregatorFactory("added", "added"),

                new DoubleSumAggregatorFactory("deleted", "deleted"),

                new DoubleSumAggregatorFactory("delta", "delta")

        );

        return DruidBeams.builder(new Timestamper<String>() {

            public DateTime timestamp(String a) {

                System.out.println(a);

                //Long date = Long.parseLong(theMap.get("timestamp").toString());

                return new DateTime(System.currentTimeMillis());

            }

        }).curator(curatorFramework)

                .discoveryPath(discoveryPath)

                .location(DruidLocation.create(indexService, firehosePattern, dataSource))

                .rollup(DruidRollup.create(dimensions, aggregators, QueryGranularity.NONE))

                .tuning(ClusteredBeamTuning.builder().segmentGranularity(Granularity.DAY).windowPeriod(new Period("PT10M")).replicants(1).partitions(1).build())

                .buildBeam();

    }

}

-----------

[Nishant Bangarwa]

Hi Ram, 

see Inline

On Mon, Mar 16, 2015 at 4:29 PM, Ram Ganesan <r...@minewhat.com> wrote:

Hi Gian,

Thanks for the pointer.

It works with spec file, we are currently using tranquility for realtime ingestion. Do you see a need for realtime node while using tranquility? 

a) If realtime nodes are not required, then how will they sync for real time queries and segment hand-offs. 

tranquility submit realtime tasks to the overlord, A realtime index task runs on a peon and is similar to a realtime node, It ingests the data and serve the realtime queries as well. 

details on RealtimeIndexTask can be found here http://druid.io/docs/latest/Tasks.html  

a) If required, what would be my spec file for realtime nodes? (any examples would greatly help)

tranquility manages this for you, If you wish to see the spec file created by tranquility, it is printed in the peon/ task logs.  

b) Do you recommend a better architecture for the same use case. [we send events from multiple sources into kafka queue , then via storm + tranquility --> overlord via ZK]

this seems good to me. @Gian, might have some more comments/ thoughts on this. 

  

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/32795c86-1f87-4cff-87ad-654d3733bbc0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nishant Software Engineer | METAMARKETS m  +91-9729200044 nishant....@metamarkets.com

[Gian Merlino]

There's no need to use standalone realtime nodes if you're using tranquility, but you do need an indexing service (overlord + middle managers). In general we recommend using standalone realtime nodes if you want druid to read directly from kafka, and using the indexing service if you want to push events into druid using storm, samza, or your own custom processor.

[Ram Ganesan]

Thanks Nishant & Gian, we are able to ingest and query. 

Based on your inputs, we have come up with an architecture, i have attached the same. please let me know your feedback.

It will be helpful, if we can add an arch diagram in the production cluster config page. 

Thanks!

Ram

P.S: We can work on the creative, if our understanding is correct.

[Nishant Bangarwa]

Hi Ram, 

your diagram seems cool, adding a bit more to the communication between tranquility, overlord and peons, 

under the hood tranquility talks to the overlord to create realtime index tasks. 

overlord assigns the task to a middlemanager which creates a peon to run the realtime index task. 

realtime index task announces itself in zookeeper service discovery from where tranquility discovers the peon running the task.  

once the task is found tranquility sends the batched events directly to the peon (realtime index task) via HTTP POST requests. 

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/75f4cc52-bb5b-4b87-8187-3379686aaa2b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ram Ganesan]

Thanks Nishant, we have given a pull request for the docs update. Let me know, if you have any other suggestions.

https://github.com/druid-io/druid/pull/1220

--Ram

[Nishant Bangarwa]

Thanks for the contribution :)  

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/5e8fefc8-f144-47e4-ae2d-8e109329b5bf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.