Not able to run simple Tranquality example

[siddhartha dutta]

Hi

I am not able to achieve simple data ingestion using Tranquillity. Need help.
The code run fine without any error but no data gets ingested into Druid.
My Code:

package com.sid;

import com.google.common.collect.ImmutableList;
import com.metamx.common.Granularity;
import com.metamx.tranquility.beam.ClusteredBeamTuning;
import com.metamx.tranquility.druid.*;
import com.metamx.tranquility.typeclass.Timestamper;
import com.twitter.finagle.Service;
import com.twitter.util.Await;
import com.twitter.util.Future;
import io.druid.granularity.QueryGranularity;
import io.druid.query.aggregation.AggregatorFactory;
import io.druid.query.aggregation.CountAggregatorFactory;
import io.druid.query.aggregation.DoubleSumAggregatorFactory;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
import org.joda.time.DateTime;
import org.joda.time.Period;

import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;

public class MyTranquilityTest {
    final static String indexService = "druid:overlord"; // Your overlord's service name.
    final static String firehosePattern = "druid:firehose:%s"; // Make up a service pattern, include %s somewhere in it.
    final static String discoveryPath = "/discovery"; // Your overlord's druid.discovery.curator.path
    final static String dataSource = "tranq_test";
    final static List<String> dimensions = ImmutableList.of("campaignid", "region");
    final static List<AggregatorFactory> aggregators = ImmutableList.of(
            new CountAggregatorFactory("count"),
            new DoubleSumAggregatorFactory("impressions", "impressions"),
            new DoubleSumAggregatorFactory("clicks", "clicks")
    );

    // Tranquility needs to be able to extract timestamps from your object type (in this case, Map<String, Object>).
    final static Timestamper<Map<String, Object>> timestamper = new Timestamper<Map<String, Object>>() {
        @Override
        public DateTime timestamp(Map<String, Object> theMap) {
            return new DateTime(theMap.get("date"));
        }
    };

    public static void main(String[] args) {
        System.out.println("Start of Tranquility");
        try {
            // Tranquility uses ZooKeeper (through Curator) for coordination.
            final CuratorFramework curator = CuratorFrameworkFactory
                    .builder()
                    .connectString("localhost:2181")
                    .retryPolicy(new ExponentialBackoffRetry(1000, 20, 30000))
                    .build();
            curator.start();

            // Tranquility needs to be able to serialize your object type. By default this is done with Jackson. If you want to
            // provide an alternate serializer, you can provide your own via ```.objectWriter(...)```. In this case, we won't
            // provide one, so we're just using Jackson:
            final Service<List<Map<String, Object>>, Integer> druidService = DruidBeams
                    .builder(timestamper)
                    .curator(curator)
                    .discoveryPath(discoveryPath)
                    .location(
                            DruidLocation.create(
                                    indexService,
                                    firehosePattern,
                                    dataSource
                            )
                    )
                    .rollup(DruidRollup.create(DruidDimensions.specific(dimensions), aggregators, QueryGranularity.DAY))
                    .tuning(ClusteredBeamTuning.create(Granularity.DAY, new Period("PT0.5S"), new Period("P1D"), 1, 1))
                    .buildJavaService();
            //
            String json = "[{ \"campaignid\":\"1\", \"date\":\"2014-10-01\", \"region\":\"us\", \"impressions\":100, \"clicks\":10 }, { \"campaignid\":\"1\", \"date\":\"2014-10-02\", \"region\":\"us\", \"impressions\":100, \"clicks\":9 }, { \"campaignid\":\"2\", \"date\":\"2014-10-03\", \"region\":\"uk\", \"impressions\":110, \"clicks\":1 }, { \"campaignid\":\"3\", \"date\":\"2014-10-02\", \"region\":\"ca\", \"impressions\":100, \"clicks\":20 }] ";
            List<Map<String, Object>> value = convertAJsonToAListMap(json);
            System.out.println("value:"+value);
            // Send events to Druid:
            final Future<Integer> numSentFuture = druidService.apply(value);
            // Wait for confirmation:
            final Integer numSent = Await.result(numSentFuture);
            System.out.println("numSent:"+numSent+", numSentFuture:"+numSentFuture);
            // Close lifecycled objects:
            Await.result(druidService.close());
            curator.close();
            System.out.println("End of Tranquility");

        } catch (Exception e) {
            // Log
            System.out.println("Message:" + e.getMessage());
        }
    }

    public static List<Map<String, Object>> convertAJsonToAListMap(String json) {
        List<Map<String, Object>> list = new LinkedList<Map<String, Object>>();
        List<Map<String, Object>> listSample = new LinkedList<Map<String, Object>>();
        ObjectMapper mapper = new ObjectMapper();
        try {
            list = mapper.readValue(json, listSample.getClass());
        } catch (JsonMappingException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return list;
    }
}



Please let me know if anybody needs anything more.

Regards
Siddhartha Dutta

[Nishant Bangarwa]

Hi Siddhartha, 

you are running with windowPeriod of 1Day and the event you are ingesting seems to be older than the windowPeriod and that is probably the cause for it getting dropped. 

can you try ingesting an event whose timestamp is within windowPeriod ?  

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/7bbffc89-0e55-4f54-bea9-bfe801b13d52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nishant Software Engineer | METAMARKETS m  +91-9729200044 nishant....@metamarkets.com

[siddhartha dutta]

Hi Nishant,

Changed the window period to 3 days as per the data. Still no data ingested & code ran without any error.
Can you suggest something else.

Regards
Siddhartha Dutta

[Andres Gomez]

Can you see the metrics logs on the task's log ??? For default, I think is this directory /tmp/persistent/task/{task_id} and the log is some like this:

2014-12-17 22:45:49,371 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2014-12-17T22:45:49.371Z","service":"middleManager","host":"i0003fec8:7081","metric":"events/thrownAway","value":0,"user2":"rb_monitor"}]

2014-12-17 22:45:49,371 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2014-12-17T22:45:49.371Z","service":"middleManager","host":"i0003fec8:7081","metric":"events/unparseable","value":0,"user2":"rb_monitor"}]

2014-12-17 22:45:49,372 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2014-12-17T22:45:49.372Z","service":"middleManager","host":"i0003fec8:7081","metric":"events/processed","value":571,"user2":"rb_monitor"}]

2014-12-17 22:45:49,372 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2014-12-17T22:45:49.372Z","service":"middleManager","host":"i0003fec8:7081","metric":"rows/output","value":0,"user2":"rb_monitor"}]

Can you post this metrics?? thanks!!

Andrés 

[siddhartha dutta]

Hi Andres,

There are no logs at tmp/persisent/task for the data I am trying to ingest to Druid through Tranquillity.
If I am running a Indexing Job then log files are getting generated per job.

Regards
Siddhartha Dutta

[siddhartha dutta]

Hi Andres/Nishant

This is my configuration for overlord runtime.properties
druid.host=localhost
druid.port=8087
druid.service=overlord

druid.zk.service.host=localhost

druid.discovery.curator.path=/discovery
druid.extensions.coordinates=["io.druid.extensions:druid-kafka-seven:0.6.147"]

druid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid
druid.db.connector.user=reap
druid.db.connector.password=reap

druid.selectors.indexing.serviceName=overlord
druid.indexer.queue.startDelay=PT0M
druid.indexer.runner.javaOpts="-server -Xmx256m"
druid.indexer.runner.startPort=8088
druid.indexer.fork.property.druid.processing.numThreads=1
druid.indexer.fork.property.druid.computation.buffer.size=100000000

When running the code though I am not getting any errors I am getting the following warnings:
WARN [ConnectionStateManager-0] org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
WARN [main] finagle - Name resolution is pending

Please let me know if any of the above information is making any sense.

Regards
Siddhartha Dutta

[Andres Gomez]

Can you add this property druid.indexer.logs.directory, is the directory to task's log ... Do you know if the tasks are running??? 

Are you using middleManagers?? If you are using middleManager you need add this property: druid.indexer.runner.type=remote

Regards, 

Andrés

[siddhartha dutta]

Hi Andres,

I had added the property to overlord runtime.properties and restarted Nodes: Coordinator, Historical, Broker & Indexing Service which are running locally.
I had not used any middle managers.
The tasks are not getting submitted to Indexing Service.

Still no success.

Regards
Siddhartha Dutta

[Andres Gomez Ferrer]

but now, can you see the task’s log?? If you will see, can you post the log .. What is your zookeeper’s version??

Regards,

Andrés Gómez

Developer 

redborder.net / ago...@redborder.net

Phone: +34 955 60 11 60

  

Piénsalo antes de imprimir este mensaje

 

Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.

 

This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system. 

En 18 de diciembre de 2014 en 12:22:33, siddhartha dutta (siddhartha...@gmail.com) escrito:

You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/9xsaWm4JNbE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b6558501-df08-45fc-8493-c00d3e55a871%40googlegroups.com.

[siddhartha dutta]

Hi Andres,

There are no logs that corresponds to the Tranquillity code that I am running. Though I can see a lot of log files for the jobs that were previously submitted by me. Like wikipedia job example.

I am using Zookeeper 3.4.6.

Please let me know if you need any other information.

Regards
Siddhartha Dutta

[Andres Gomez Ferrer]

Hi Siddhartha,

I want see the log files for the jobs (task), to check his STATUS, Can you check?

Regards,

Andrés Gómez

Developer 

redborder.net / ago...@redborder.net

Phone: +34 955 60 11 60

  

Piénsalo antes de imprimir este mensaje

 

Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.

 

This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system. 

En 18 de diciembre de 2014 en 12:41:10, siddhartha dutta (siddhartha...@gmail.com) escrito:

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/35fc80bb-9c40-4272-a757-7a4ec96efee6%40googlegroups.com.

[siddhartha dutta]

Hi Andres,

When I am running the Tranquillity code it is not showing up on the console http://localhost:8087/console.html.
Nor I am able to see any logs in the log folder.

Regards
Siddhartha Dutta

...

[ago...@redborder.net]

Mmm ok.. Then the task is not create

Is the time of your machine, the same of your event?

Regards,

Andrés

jueves, 18 diciembre 2014, 01:02p.m. +01:00 de siddhartha dutta <siddhartha...@gmail.com>:

Hi Andres,

When I am running the Tranquillity code it is not showing up on the console  http://localhost:8087/console.html .
Nor I am able to see any logs in the log folder.

Regards
Siddhartha Dutta

On Thursday, 18 December 2014 17:13:29 UTC+5:30, Andres Gomez  wrote:
>Hi Siddhartha,
>
>I want see the log files for the jobs (task), to check his STATUS, Can you check?
>
>Regards,
>
>Andrés Gómez
>Developer  
>redborder.net  /   ago...@redborder.net
>Phone:  +34 955 60 11 60
>
>
>   
>

>Piénsalo antes de imprimir este mensaje
> 
>Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
> 

>This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. I f you have received this email in error, please notify the sender and delete it from your system. 
>
>
>En 18 de diciembre de 2014 en 12:41:10, siddhartha dutta ( siddhartha...@gmail.com ) escrito:

>>Hi Andres,
>>
>>There are no logs that corresponds to the Tranquillity code that I
am running. Though I can see a lot of log files for the jobs that
were previously submitted by me. Like wikipedia job example.
>>
>>I am using Zookeeper 3.4.6.
>>
>>Please let me know if you need any other information.
>>
>>Regards
>>Siddhartha Dutta
>>
>>On Thursday, 18 December 2014 16:55:44 UTC+5:30, Andres Gomez
wrote:
>>>but now, can you see the task’s log?? If you will see, can you post
the log .. What is your zookeeper’s version??
>>>
>>>Regards,
>>>
>>>Andrés Gómez
>>>Developer  
>>>redborder.net  /   ago...@redborder.net
>>>Phone: +34
955 60 11 60
>>>
>>>
>>>   
>>>

>>>Piénsalo
antes de imprimir este mensaje
>>> 
>>>Este correo
electrónico, incluidos sus anexos, se dirige exclusivamente a su
destinatario. Contiene información CONFIDENCIAL cuya divulgación
está prohibida por la ley o puede estar sometida a secreto
profesional. Si ha recibido este mensaje por error, le rogamos nos
lo comunique inmediatamente y proceda a su
destrucción.
>>> 
>>>This
email, including attachments, is intended exclusively for its
addressee. It contains information that is CONFIDENTIAL whose
disclosure is prohibited by law and may be covered by legal

privilege. I f you have received this email in error, please notify

[{"feed":"metrics","timestamp":"2014-12-17T22:45:49.372Z","service":"middleManager","host":"i0003fec8:7081","metric": "events/processed","value":571 ,"user2":"rb_monitor"}]

To view this discussion on the web visit  https://groups.google.com/d/msgid/druid-development/521f0a4e-a724-4776-9002-7f256c3eac58%40googlegroups.com .

[Gian Merlino]

The windowPeriod is relative to right now, based on your machines' system clocks, so to ingest data from early October you'd need a windowPeriod of at least 80 days. This isn't a recommended configuration for production (it will prevent indexing-to-historical handoff from proceeding in a timely manner) but it should work for testing.

[siddhartha dutta]

Hi Gian,

Did the change in windowPeriod to 80 days and got some error. It had failed to create the indexing task and there was a timeout exception.

The Stack trace is as follows:

Start of Tranquility
2014-12-19 17:24:24,217 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=sid-VirtualBox
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_72
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre
2014-12-19 17:24:24,230 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/lib/jvm/java-7-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-7-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfxrt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-7-oracle/jre/lib/ext/sunjce_provider.jar:/home/sid/GitProjects/Test/target/classes:/home/sid/.m2/repository/org/apache/poi/poi-ooxml/3.7/poi-ooxml-3.7.jar:/home/sid/.m2/repository/org/apache/poi/poi/3.7/poi-3.7.jar:/home/sid/.m2/repository/org/apache/poi/poi-ooxml-schemas/3.7/poi-ooxml-schemas-3.7.jar:/home/sid/.m2/repository/org/apache/xmlbeans/xmlbeans/2.3.0/xmlbeans-2.3.0.jar:/home/sid/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/sid/.m2/repository/org/apache/geronimo/specs/geronimo-stax-api_1.0_spec/1.0/geronimo-stax-api_1.0_spec-1.0.jar:/home/sid/.m2/repository/dom4j/dom4j/1.6.1/dom4j-1.6.1.jar:/home/sid/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar:/home/sid/.m2/repository/com/metamx/tranquility_2.10/0.2.16/tranquility_2.10-0.2.16.jar:/home/sid/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar:/home/sid/.m2/repository/com/metamx/scala-util_2.10/1.8.29/scala-util_2.10-1.8.29.jar:/home/sid/.m2/repository/org/eintr/loglady/loglady_2.10/1.1.0/loglady_2.10-1.1.0.jar:/home/sid/.m2/repository/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar:/home/sid/.m2/repository/com/metamx/java-util/0.26.6/java-util-0.26.6.jar:/home/sid/.m2/repository/org/skife/config/config-magic/0.9/config-magic-0.9.jar:/home/sid/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/home/sid/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.2.2/jackson-annotations-2.2.2.jar:/home/sid/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.2.2/jackson-core-2.2.2.jar:/home/sid/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.2.2/jackson-databind-2.2.2.jar:/home/sid/.m2/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/home/sid/.m2/repository/joda-time/joda-time/2.1/joda-time-2.1.jar:/home/sid/.m2/repository/com/metamx/http-client/0.9.6/http-client-0.9.6.jar:/home/sid/.m2/repository/io/netty/netty/3.9.0.Final/netty-3.9.0.Final.jar:/home/sid/.m2/repository/com/metamx/emitter/0.2.12/emitter-0.2.12.jar:/home/sid/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/home/sid/.m2/repository/com/metamx/server-metrics/0.0.9/server-metrics-0.0.9.jar:/home/sid/.m2/repository/org/hyperic/sigar/1.6.5.132/sigar-1.6.5.132.jar:/home/sid/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/sid/.m2/repository/org/joda/joda-convert/1.6/joda-convert-1.6.jar:/home/sid/.m2/repository/org/scala-tools/time/time_2.10/0.6-mmx1/time_2.10-0.6-mmx1.jar:/home/sid/.m2/repository/org/yaml/snakeyaml/1.11/snakeyaml-1.11.jar:/home/sid/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.2.2/jackson-dataformat-smile-2.2.2.jar:/home/sid/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-joda/2.2.2/jackson-datatype-joda-2.2.2.jar:/home/sid/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.10/2.2.2/jackson-module-scala_2.10-2.2.2.jar:/home/sid/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/home/sid/.m2/repository/com/google/code/findbugs/jsr305/2.0.1/jsr305-2.0.1.jar:/home/sid/.m2/repository/org/jdbi/jdbi/2.27/jdbi-2.27.jar:/home/sid/.m2/repository/mysql/mysql-connector-java/5.1.18/mysql-connector-java-5.1.18.jar:/home/sid/.m2/repository/com/h2database/h2/1.3.158/h2-1.3.158.jar:/home/sid/.m2/repository/c3p0/c3p0/0.9.1.2/c3p0-0.9.1.2.jar:/home/sid/.m2/repository/org/apache/curator/curator-framework/2.4.0/curator-framework-2.4.0.jar:/home/sid/.m2/repository/org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar:/home/sid/.m2/repository/org/apache/curator/curator-recipes/2.4.0/curator-recipes-2.4.0.jar:/home/sid/.m2/repository/org/apache/curator/curator-x-discovery/2.4.0/curator-x-discovery-2.4.0.jar:/home/sid/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/home/sid/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/home/sid/.m2/repository/com/twitter/util-core_2.10/6.18.0/util-core_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/finagle-core_2.10/6.18.0/finagle-core_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/util-app_2.10/6.18.0/util-app_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/util-collection_2.10/6.18.0/util-collection_2.10-6.18.0.jar:/home/sid/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/home/sid/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/sid/.m2/repository/com/twitter/util-hashing_2.10/6.18.0/util-hashing_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/util-jvm_2.10/6.18.0/util-jvm_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/util-logging_2.10/6.18.0/util-logging_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/finagle-http_2.10/6.18.0/finagle-http_2.10-6.18.0.jar:/home/sid/.m2/repository/com/twitter/util-codec_2.10/6.18.0/util-codec_2.10-6.18.0.jar:/home/sid/.m2/repository/commons-codec/commons-codec/1.7/commons-codec-1.7.jar:/home/sid/.m2/repository/org/slf4j/jul-to-slf4j/1.7.2/jul-to-slf4j-1.7.2.jar:/home/sid/.m2/repository/io/druid/druid-server/0.6.146/druid-server-0.6.146.jar:/home/sid/.m2/repository/io/druid/druid-processing/0.6.146/druid-processing-0.6.146.jar:/home/sid/.m2/repository/io/druid/druid-common/0.6.146/druid-common-0.6.146.jar:/home/sid/.m2/repository/io/druid/druid-api/0.2.7/druid-api-0.2.7.jar:/home/sid/.m2/repository/com/google/inject/guice/4.0-beta/guice-4.0-beta.jar:/home/sid/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/sid/.m2/repository/com/google/inject/extensions/guice-multibindings/4.0-beta/guice-multibindings-4.0-beta.jar:/home/sid/.m2/repository/io/airlift/airline/0.5/airline-0.5.jar:/home/sid/.m2/repository/org/hibernate/hibernate-validator/5.0.1.Final/hibernate-validator-5.0.1.Final.jar:/home/sid/.m2/repository/org/jboss/logging/jboss-logging/3.1.1.GA/jboss-logging-3.1.1.GA.jar:/home/sid/.m2/repository/com/fasterxml/classmate/0.8.0/classmate-0.8.0.jar:/home/sid/.m2/repository/commons-io/commons-io/2.0.1/commons-io-2.0.1.jar:/home/sid/.m2/repository/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar:/home/sid/.m2/repository/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/home/sid/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-guava/2.2.3/jackson-datatype-guava-2.2.3.jar:/home/sid/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/home/sid/.m2/repository/com/metamx/bytebuffer-collections/0.0.2/bytebuffer-collections-0.0.2.jar:/home/sid/.m2/repository/it/uniroma3/mat/extendedset/1.3.4/extendedset-1.3.4.jar:/home/sid/.m2/repository/com/ning/compress-lzf/0.8.4/compress-lzf-0.8.4.jar:/home/sid/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/home/sid/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/sid/.m2/repository/com/ibm/icu/icu4j/4.8.1/icu4j-4.8.1.jar:/home/sid/.m2/repository/org/mozilla/rhino/1.7R4/rhino-1.7R4.jar:/home/sid/.m2/repository/com/davekoelle/alphanum/1.0.3/alphanum-1.0.3.jar:/home/sid/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/sid/.m2/repository/org/glassfish/javax.el/3.0.0/javax.el-3.0.0.jar:/home/sid/.m2/repository/com/amazonaws/aws-java-sdk/1.6.0.1/aws-java-sdk-1.6.0.1.jar:/home/sid/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/sid/.m2/repository/org/apache/httpcomponents/httpclient/4.2/httpclient-4.2.jar:/home/sid/.m2/repository/org/apache/httpcomponents/httpcore/4.2/httpcore-4.2.jar:/home/sid/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.2.3/jackson-jaxrs-json-provider-2.2.3.jar:/home/sid/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.2.3/jackson-jaxrs-base-2.2.3.jar:/home/sid/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.2.3/jackson-module-jaxb-annotations-2.2.3.jar:/home/sid/.m2/repository/com/sun/jersey/jersey-server/1.17.1/jersey-server-1.17.1.jar:/home/sid/.m2/repository/asm/asm/3.1/asm-3.1.jar:/home/sid/.m2/repository/com/sun/jersey/jersey-core/1.17.1/jersey-core-1.17.1.jar:/home/sid/.m2/repository/com/google/inject/extensions/guice-servlet/4.0-beta/guice-servlet-4.0-beta.jar:/home/sid/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.17.1/jersey-guice-1.17.1.jar:/home/sid/.m2/repository/com/sun/jersey/jersey-servlet/1.17.1/jersey-servlet-1.17.1.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-server/9.2.2.v20140723/jetty-server-9.2.2.v20140723.jar:/home/sid/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-http/9.2.2.v20140723/jetty-http-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-util/9.2.2.v20140723/jetty-util-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-io/9.2.2.v20140723/jetty-io-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-proxy/9.2.2.v20140723/jetty-proxy-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-client/9.2.2.v20140723/jetty-client-9.2.2.v20140723.jar:/home/sid/.m2/repository/io/tesla/aether/tesla-aether/0.0.5/tesla-aether-0.0.5.jar:/home/sid/.m2/repository/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.jar:/home/sid/.m2/repository/org/eclipse/aether/aether-spi/0.9.0.M2/aether-spi-0.9.0.M2.jar:/home/sid/.m2/repository/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar:/home/sid/.m2/repository/org/eclipse/aether/aether-impl/0.9.0.M2/aether-impl-0.9.0.M2.jar:/home/sid/.m2/repository/org/eclipse/aether/aether-connector-file/0.9.0.M2/aether-connector-file-0.9.0.M2.jar:/home/sid/.m2/repository/io/tesla/aether/aether-connector-okhttp/0.0.9/aether-connector-okhttp-0.0.9.jar:/home/sid/.m2/repository/com/squareup/okhttp/okhttp/1.0.2/okhttp-1.0.2.jar:/home/sid/.m2/repository/org/apache/maven/wagon/wagon-provider-api/2.4/wagon-provider-api-2.4.jar:/home/sid/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.jar:/home/sid/.m2/repository/org/apache/maven/maven-aether-provider/3.1.1/maven-aether-provider-3.1.1.jar:/home/sid/.m2/repository/org/apache/maven/maven-model/3.1.1/maven-model-3.1.1.jar:/home/sid/.m2/repository/org/apache/maven/maven-model-builder/3.1.1/maven-model-builder-3.1.1.jar:/home/sid/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.19/plexus-interpolation-1.19.jar:/home/sid/.m2/repository/org/apache/maven/maven-repository-metadata/3.1.1/maven-repository-metadata-3.1.1.jar:/home/sid/.m2/repository/org/apache/maven/maven-settings-builder/3.1.1/maven-settings-builder-3.1.1.jar:/home/sid/.m2/repository/org/apache/maven/maven-settings/3.1.1/maven-settings-3.1.1.jar:/home/sid/.m2/repository/org/antlr/antlr4-runtime/4.0/antlr4-runtime-4.0.jar:/home/sid/.m2/repository/org/abego/treelayout/org.abego.treelayout.core/1.0.1/org.abego.treelayout.core-1.0.1.jar:/home/sid/.m2/repository/com/google/code/simple-spring-memcached/spymemcached/2.8.4/spymemcached-2.8.4.jar:/home/sid/.m2/repository/net/jpountz/lz4/lz4/1.1.2/lz4-1.1.2.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-servlet/9.2.2.v20140723/jetty-servlet-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-security/9.2.2.v20140723/jetty-security-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-servlets/9.2.2.v20140723/jetty-servlets-9.2.2.v20140723.jar:/home/sid/.m2/repository/org/eclipse/jetty/jetty-continuation/9.2.2.v20140723/jetty-continuation-9.2.2.v20140723.jar:/home/sid/.m2/repository/com/ircclouds/irc/irc-api/1.0-0011/irc-api-1.0-0011.jar:/home/sid/.m2/repository/com/maxmind/geoip2/geoip2/0.4.0/geoip2-0.4.0.jar:/home/sid/.m2/repository/com/maxmind/maxminddb/maxminddb/0.2.0/maxminddb-0.2.0.jar:/home/sid/.m2/repository/com/google/http-client/google-http-client/1.15.0-rc/google-http-client-1.15.0-rc.jar:/home/sid/.m2/repository/xpp3/xpp3/1.1.4c/xpp3-1.1.4c.jar:/home/sid/.m2/repository/com/google/http-client/google-http-client-jackson2/1.15.0-rc/google-http-client-jackson2-1.15.0-rc.jar:/home/sid/.m2/repository/io/druid/druid-indexing-service/0.6.146/druid-indexing-service-0.6.146.jar:/home/sid/.m2/repository/io/druid/druid-indexing-hadoop/0.6.146/druid-indexing-hadoop-0.6.146.jar:/home/sid/.m2/repository/net/java/dev/jets3t/jets3t/0.9.1/jets3t-0.9.1.jar:/home/sid/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/home/sid/.m2/repository/mx4j/mx4j/3.0.2/mx4j-3.0.2.jar:/home/sid/.m2/repository/javax/mail/mail/1.4.7/mail-1.4.7.jar:/home/sid/.m2/repository/org/bouncycastle/bcprov-jdk15/1.46/bcprov-jdk15-1.46.jar:/home/sid/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/1.0/java-xmlbuilder-1.0.jar:/home/sid/.m2/repository/net/iharder/base64/2.3.8/base64-2.3.8.jar:/home/sid/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/home/sid/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/home/sid/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/home/sid/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/home/sid/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/opt/intellij-idea-ce/lib/idea_rt.jar
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/opt/intellij-idea-ce/bin::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.5.0-54-generic
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=sid
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/sid
2014-12-19 17:24:24,237 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/sid/GitProjects/Test
2014-12-19 17:24:24,239 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@45f676cb
2014-12-19 17:24:24,273 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-19 17:24:24,302 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-12-19 17:24:24,347 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14a611bbe670020, negotiated timeout = 40000
2014-12-19 17:24:24,363 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2014-12-19 17:24:24,363 WARN [ConnectionStateManager-0] org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
2014-12-19 17:24:25,351 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.0.1.Final
2014-12-19 17:24:25,986 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], localRepository='/home/sid/.m2/repository', remoteRepositories=[http://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]
2014-12-19 17:24:26,717 INFO [main] com.metamx.emitter.core.LoggingEmitter - Start: started [true]
2014-12-19 17:24:27,072 INFO [main] com.twitter.finagle - Finagle version 6.18.0 (rev=a12a6ab5ce5d213c3753ad5884daa712df1b973b) built at 20140625-103953
2014-12-19 17:24:27,257 INFO [main] com.metamx.common.scala.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set()
2014-12-19 17:24:27,294 WARN [main] finagle - Name resolution is pending
2014-12-19 17:24:27,366 INFO [main] com.metamx.tranquility.finagle.FinagleRegistry - Created client for service: druid:overlord
value:[{campaignid=1, date=2014-10-01, region=us, impressions=100, clicks=10}, {campaignid=1, date=2014-10-02, region=us, impressions=100, clicks=9}, {campaignid=2, date=2014-10-03, region=uk, impressions=110, clicks=1}, {campaignid=3, date=2014-10-02, region=ca, impressions=100, clicks=20}]
2014-12-19 17:24:27,681 INFO [ClusteredBeam-ZkFuturePool-357dee11-1c89-40c1-a8f0-d7a344dd1a2f] com.metamx.tranquility.beam.ClusteredBeam - Creating new beams for identifier[druid:overlord/tranq_test] timestamp[2014-10-01T00:00:00.000+05:30] (target = 1, actual = 0)
2014-12-19 17:24:27,930 INFO [ClusteredBeam-ZkFuturePool-357dee11-1c89-40c1-a8f0-d7a344dd1a2f] com.metamx.common.scala.control$ - Creating druid indexing task with id: index_realtime_tranq_test_2014-10-01T00:00:00.000+05:30_0_0_kbcoceaa (service = druid:overlord)
2014-12-19 17:25:58,021 ERROR [ClusteredBeam-ZkFuturePool-357dee11-1c89-40c1-a8f0-d7a344dd1a2f] com.metamx.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/tranq_test
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:overlord while waiting for a response for the request, including retries (if applicable)
    at com.twitter.finagle.NoStacktrace(Unknown Source)
2014-12-19 17:25:58,069 WARN [ClusteredBeam-ZkFuturePool-357dee11-1c89-40c1-a8f0-d7a344dd1a2f] com.metamx.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create beams: druid:overlord/tranq_test
{ }
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/tranq_test] timestamp[2014-10-01T00:00:00.000+05:30]
    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:242)
    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:239)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
    at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:802)
    at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:801)
    at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:93)
    at com.twitter.util.Promise$Transformer.k(Promise.scala:93)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:102)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:84)
    at com.twitter.util.Promise$$anon$2.run(Promise.scala:324)
    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:184)
    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:155)
    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:210)
    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:84)
    at com.twitter.util.Promise.runq(Promise.scala:310)
    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:605)
    at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:111)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:overlord while waiting for a response for the request, including retries (if applicable)
    at com.twitter.finagle.NoStacktrace(Unknown Source)
2014-12-19 17:25:58,100 INFO [ClusteredBeam-ZkFuturePool-357dee11-1c89-40c1-a8f0-d7a344dd1a2f] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2014-12-19T17:25:58.081+05:30","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create beams: druid:overlord/tranq_test","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/tranq_test] timestamp[2014-10-01T00:00:00.000+05:30]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/tranq_test] timestamp[2014-10-01T00:00:00.000+05:30]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:242)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:239)\n\tat scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:802)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:801)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:93)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:93)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:102)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:84)\n\tat com.twitter.util.Promise$$anon$2.run(Promise.scala:324)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:184)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:155)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:210)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:84)\n\tat com.twitter.util.Promise.runq(Promise.scala:310)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:605)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:111)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:overlord while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n"}}]
Message:Failed to save new beam for identifier[druid:overlord/tranq_test] timestamp[2014-10-01T00:00:00.000+05:30]

Regards
Siddhartha

[Gian Merlino]

Do you have druid.indexer.task.chathandler.type=announce set in your middle manager properties? If not, please try setting that; it's necessary for tranquility to be able to find tasks once they are started.

If you do have that property set, and things are still not working, do you mind sharing the logs from your task as well as the logs from tranquility?

...

[siddhartha dutta]

Hi Gian,

I was trying to ingest data without running any middle manager.
I had just the ingestion service on.
A clarification : If I am running a local cluster of druid do I need to run a middle manager instance?

Created a runtime property for Middle Manager with below details as given in Documentation with the following command:
command: java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:config/middleManager io.druid.cli.Main server middleManager
Configuration:
druid.host=localhost
druid.port=8091
druid.service=middleManager
druid.zk.service.host=localhost
druid.indexer.task.chathandler.type=announce

druid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid
druid.db.connector.user=reap
druid.db.connector.password=reap
druid.selectors.indexing.serviceName=overlord

druid.indexer.runner.startPort=8092
druid.indexer.fork.property.druid.computation.buffer.size=268435456

It is giving an error:
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) druid.port - must be greater than or equal to 0
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:112)
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:112)
  while locating com.google.common.base.Supplier<io.druid.server.DruidNode> annotated with @io.druid.guice.annotations.Self()

Giam Is there any document where there is some sample code and step by step integration of Tranquility & Druid.?

[Fangjin Yang]

Hi Siddhartha, see inline.

On Monday, December 22, 2014 3:25:53 AM UTC-8, siddhartha dutta wrote:

Hi Gian,

I was trying to ingest data without running any middle manager.
I had just the ingestion service on.
A clarification : If I am running a local cluster of druid do I need to run a middle manager instance?

No. You can run the overlord in local mode and you can use it to do low scale, simple POCs without needing middle managers. Distributing out the indexing service is recommended if you care about availability, scalability and flexibility.

Created a runtime property for Middle Manager with below details as given in Documentation with the following command:
command: java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:config/middleManager io.druid.cli.Main server middleManager
Configuration:
druid.host=localhost
druid.port=8091
druid.service=middleManager
druid.zk.service.host=localhost
druid.indexer.task.chathandler.type=announce
druid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid
druid.db.connector.user=reap
druid.db.connector.password=reap
druid.selectors.indexing.serviceName=overlord
druid.indexer.runner.startPort=8092
druid.indexer.fork.property.druid.computation.buffer.size=268435456

It is giving an error:
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) druid.port - must be greater than or equal to 0
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:112)
  at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:112)
  while locating com.google.common.base.Supplier<io.druid.server.DruidNode> annotated with @io.druid.guice.annotations.Self()

Under config/middleManager, is there a runtime.properties file with the properties you listed above? I believe that is what this error is complaining about.

[Gian Merlino]

If you're running a single-machine setup, then you don't need middleManagers, but then you do need "druid.indexer.task.chathandler.type=announce" set on your overlord. In 0.6, that property needs to be set on whichever thing is actually doing the spawning of peons. (Overlord if you're running in local mode, Middle Manager if you're running in distributed mode.)

There isn't currently a tutorial geared towards setting up Tranquility and Druid. We hope to have one written for 0.7 when 0.7 is released, although for 0.6, the best documentation currently available is the combination of the indexing-service docs on druid.io and the tranquility readme on github. There's some sample code in the tranquility readme. I'm also happy to help you here- if this most recent advice isn't enough to get things going, please feel free to share your properties files for the various druid services, and your tranquility code, so we can take a look at what might be going wrong.

[siddhartha dutta]

Hi Fangjin,

There is a runtime.properties file under config/middleManager.
The exception is 'druid.port - must be greater than or equal to 0', but runtime.properties has a configuration druid.port=8091.
Why is Middle Manager throwing error?

Regards
Siddhartha Dutta

[siddhartha dutta]

Hi Gian/Fangjin,

As of now I am just doing small POC's in local mode. Had added the property "druid.indexer.task.chathandler.type=announce" in overlord still data is not getting into Druid through Tranquility.

Sharing in my code, all the configuration files and output that I am getting on my console for Tranquility code.
Thanks for all the help & please let me know if you need any other information.

Regards
Siddhartha Dutta

[Gian Merlino]

It looks like your tranquility is not pointing at the same service key that the overlord is announcing. Can you try changing this:

    final static String indexService = "druid:overlord";

To this:

final static String indexService = "overlord";

[siddhartha dutta]

Hi Gian,

Had tried it once before without any success. Tried it again still same result.

Regards
Siddhartha Dutta

[Gian Merlino]

How far does tranquility get when you make that change? It should at least get a little farther than it did.

[siddhartha dutta]

Hi Gian,

There was no change in the output but if I changed the following in the code
.tuning(ClusteredBeamTuning.create(Granularity.HOUR, new Period("PT1M"), new Period("P80D"), 1, 1))
to
.tuning(ClusteredBeamTuning.create(Granularity.HOUR, new Period("PT1M"), new Period("P3M"), 1, 1))

getting the following Exception:
2014-12-23 20:50:57,009 WARN [Hashed wheel timer #1] com.metamx.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: overlord/tranq_test
{
  "eventCount" : 2,
  "timestamp" : "2014-10-02T00:00:00.000+05:30",
  "beams" : "HashPartitionBeam(DruidBeam(timestamp = 2014-10-02T00:00:00.000+05:30, partition = 0, tasks = [index_realtime_tranq_test_2014-10-02T00:00:00.000+05:30_0_0_miljknff/tranq_test-00-0000-0000]))"
}
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)
    at com.twitter.finagle.NoStacktrace(Unknown Source)
2014-12-23 20:50:57,081 INFO [Hashed wheel timer #1] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2014-12-23T20:50:57.035+05:30","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: overlord/tranq_test","data":{"exceptionType":"com.twitter.finagle.GlobalRequestTimeoutException","exceptionStackTrace":"com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2014-10-02T00:00:00.000+05:30","beams":"HashPartitionBeam(DruidBeam(timestamp = 2014-10-02T00:00:00.000+05:30, partition = 0, tasks = [index_realtime_tranq_test_2014-10-02T00:00:00.000+05:30_0_0_miljknff/tranq_test-00-0000-0000]))","eventCount":2,"exceptionMessage":"exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)"}}]
2014-12-23 20:50:57,775 WARN [Hashed wheel timer #1] com.metamx.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: overlord/tranq_test
{
  "eventCount" : 1,
  "timestamp" : "2014-10-03T00:00:00.000+05:30",
  "beams" : "HashPartitionBeam(DruidBeam(timestamp = 2014-10-03T00:00:00.000+05:30, partition = 0, tasks = [index_realtime_tranq_test_2014-10-03T00:00:00.000+05:30_0_0_kgckiemn/tranq_test-00-0000-0000]))"
}
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)
    at com.twitter.finagle.NoStacktrace(Unknown Source)
2014-12-23 20:50:57,776 INFO [Hashed wheel timer #1] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2014-12-23T20:50:57.776+05:30","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: overlord/tranq_test","data":{"exceptionType":"com.twitter.finagle.GlobalRequestTimeoutException","exceptionStackTrace":"com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2014-10-03T00:00:00.000+05:30","beams":"HashPartitionBeam(DruidBeam(timestamp = 2014-10-03T00:00:00.000+05:30, partition = 0, tasks = [index_realtime_tranq_test_2014-10-03T00:00:00.000+05:30_0_0_kgckiemn/tranq_test-00-0000-0000]))","eventCount":1,"exceptionMessage":"exceeded 1.minutes+30.seconds to druid:firehose:tranq_test-00-0000-0000 while waiting for a response for the request, including retries (if applicable)"}}]
numSent:0, numSentFuture:Promise@976294176(state=Done(Return(0)))
2014-12-23 20:50:57,782 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down
2014-12-23 20:50:57,783 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x14a713472870034 closed
End of Tranquility

Regards
Siddhartha Dutta

[Gian Merlino]

Ah okay. That makes sense- if you're using a fixed dataset for testing, you'll have to keep increasing the windowPeriod as time marches on.

This exception probably indicates that tranquility successfully spawned a task, but now can't communicate with it. This usually means some sort of service discovery configuration issue. Do you see the task "index_realtime_tranq_test_2014-10-02T00:00:00.000+05:30_0_0_miljknff" in your overlord console? If you do, can you follow the "log (all)" link and share the logs? Particularly interesting will be the log lines related to the task announcing itself in service discovery, and what service discovery path it thinks it's using.

Also, what version of druid and what version of tranquility are you using?

[siddhartha dutta]

Hi Gian,

I am getting the following Exception in logs:
java.lang.UnsupportedOperationException: Cannot convert to Duration as this period contains months and months vary in length
    at org.joda.time.Period.checkYearsAndMonths(Period.java:1514)
    at org.joda.time.Period.toStandardDuration(Period.java:1496)
    at io.druid.segment.realtime.plumber.RealtimePlumber.startPersistThread(RealtimePlumber.java:570)
    at io.druid.segment.realtime.plumber.RealtimePlumber.startJob(RealtimePlumber.java:152)
    at io.druid.indexing.common.task.RealtimeIndexTask.run(RealtimeIndexTask.java:324)
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:218)
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:197)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Regards
Siddhartha Dutta

[Gian Merlino]

Ah ok. Can you please try using a windowPeriod of P90D (or P100D or something higher if needed) instead of P3M? It looks like druid prefers its windowPeriods to be based on things with standard durations (weeks, days, hours, minutes, seconds).

[siddhartha dutta]

Hi Gian,

Did the change but it is still showing the same exception.
If it is ok can you please run my sample code with configurations.

Regards
Siddhartha Dutta

[siddhartha dutta]

Hi Gian,

The code finally ran.

Key takeaways from this:
1) The key of DATE field in any data should not be named date. For example in the code json was this 'String json = "[{ \"campaignid\":\"1\", \"date\":\"2014-10-01\", \"region\":\"us\", \"impressions\":100, \"clicks\":10 }, { \"campaignid\":\"1\", \"date\":\"2014-10-02\", \"region\":\"us\", \"impressions\":100, \"clicks\":9 }, { \"campaignid\":\"2\", \"date\":\"2014-10-03\", \"region\":\"uk\", \"impressions\":110, \"clicks\":1 }, { \"campaignid\":\"3\", \"date\":\"2014-10-02\", \"region\":\"ca\", \"impressions\":100, \"clicks\":20 }] ";'. Changed date to timestamp

2) The data should be inside the windowPeriod signified by the line  "tuning(ClusteredBeamTuning.

create(Granularity.DAY, new Period("PT0.5S"), new Period("P1D"), 1, 1))". It should be sufficiently large to handle all the data.

3) Added the following to the code
timestampSpec(new TimestampSpec("timestamp", "auto")

Regards
Siddhartha Dutta

[Gian Merlino]

Great. Thanks for reporting back!

Fwiw, it should be okay to call your timestamp field "date" if you want, so long as you supply the builder with .timestampSpec(new TimestampSpec("date", "auto")) and make sure that your Timestamper knows how to extract the timestamp from where it lives.