Error when sending data to Druid through Tranquility

[Pascal]

Hey !

I am new to Druid and still in the process of discovering the technologies...
So far, I have completed the Getting Started section, and in particular the loading stream data section. This works so far !

Now I would like to use Tranquility so that I can also stream data from my own application. I have been trying to use the sample code and reusing the same server configuration json file as in the getting started example.



server.json

 {
  "dataSources" : {
    "pageviews" : {
      "spec" : {
        "dataSchema" : {
          "dataSource" : "pageviews",
          "parser" : {
            "type" : "string",
            "parseSpec" : {
              "timestampSpec" : {
                "column" : "time",
                "format" : "auto"
              },
              "dimensionsSpec" : {
                "dimensions" : ["url", "user"]
              },
              "format" : "json"
            }
          },
          "granularitySpec" : {
            "type" : "uniform",
            "segmentGranularity" : "hour",
            "queryGranularity" : "none"
          },
          "metricsSpec" : [
            {"name": "views", "type": "count"},
            {"name": "latencyMs", "type": "doubleSum", "fieldName": "latencyMs"}
          ]
        },
        "ioConfig" : {
          "type" : "realtime"
        },
        "tuningConfig" : {
          "type" : "realtime",
          "maxRowsInMemory" : "100000",
          "intermediatePersistPeriod" : "PT10M",
          "windowPeriod" : "PT10M"
        }
      },
      "properties" : {
        "task.partitions" : "1",
        "task.replicants" : "1"
      }
    }
  },
  "properties" : {
    "zookeeper.connect" : "localhost",
    "druid.discovery.curator.path" : "/druid/discovery",
    "druid.selectors.indexing.serviceName" : "druid/overlord",
    "http.port" : "8200",
    "http.threads" : "8"
  }
}



The test code:

public class Main {
    private static final Logger log = new Logger(Main.class);

    public static void main(String[] args) {

        // Read config from "server.json" on the classpath.
        final InputStream configStream = Main.class.getClassLoader().getResourceAsStream("server.json");
        final TranquilityConfig<PropertiesBasedConfig> config = TranquilityConfig.read(configStream);
        final DataSourceConfig<PropertiesBasedConfig> pageViewConf = config.getDataSource("pageviews");
        final Tranquilizer<Map<String, Object>> sender = DruidBeams.fromConfig(pageViewConf)
                .buildTranquilizer(pageViewConf.tranquilizerBuilder());

        sender.start();

        try {
            // Send 10000 objects
            for (int i = 0; i < 10000; i++) {
                // Build a sample event to send; make sure we use a current date
                final Map<String, Object> obj = ImmutableMap.<String, Object>of(
                        "time", new DateTime().toString(),
                        "url", "/foo/bar",
                        "user", "alice",
                        "latencyMs", i
                );

                // Asynchronously send event to Druid:
                sender.send(obj).addEventListener(
                        new FutureEventListener<BoxedUnit>() {

                            public void onSuccess(BoxedUnit value) {
                                log.info("Sent message: %s", obj);
                            }


                            public void onFailure(Throwable e) {
                                if (e instanceof MessageDroppedException) {
                                    log.warn(e, "Dropped message: %s", obj);
                                } else {
                                    log.error(e, "Failed to send message: %s", obj);
                                }
                            }
                        }
                );
            }
        } finally {
            sender.flush();
            sender.stop();
        }
    }
}


I have started all 5 nodes (historical, broker, coordinator, overlord and middleManager) on local (I am still doing everything local so far) and when starting the code, I get the following error:

Exception in thread "main" java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)
...

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234)

...
Caused by: java.lang.ClassNotFoundException: com.metamx.common.parsers.JSONPathParser
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)



From the error I suspect it is because the compiler does not find com.metamx.common.parsers.JSONPathParser? If this is the case, how can I include this in my project?

Thank for your help !

Pascal

[Pascal]

Okay,

After manually adding the missing packages to my classpath as well as multiple other ones, it worked :).
I think it would be nice to update the documentation about that or to provide a starting project with the dependencies resolved !

Thanks for your work anyway !

[Fangjin Yang]

Hi Pascal, great to see you've resolved your issue. Tranquility docs are located at https://github.com/druid-io/tranquility/tree/master/docs. It would be great if you could help contribute an update around your workarounds.

[Pascal]

Hi Fangjin !

Thanks for your reply. Yes, I read the whole docs about Tranquility and did not find the solution to my problem, unfortunately... As I said, I solved it by importing all the missing dependencies, manually. As a starting point, I took the Java Example code and imported the Core API. However, it appears that this is not enough to run the example code because I was still missing JSONPathParser, which can be found here. I made a jar out of it and included it on my classpath (did not find it on maven...). In my specific case, I also needed jsonpath from jayway, which can be found here. Adding these dependencies was enough for me! :)

Anyway, thanks for your work, keep up !

[Rasmus Schults]

Hey,

I am having similar issue with Tranquility core API. Issue started for me since tranquility 0.8.0. Adding JSONPathParser.java to your source and including json-path artifact it started working again. Current workaround seems hackish and I am not sure how good idea it would be to update docs to use that.

[Gian Merlino]

Hey Rasmus,

This should be getting pulled in automatically as a dependency, especially if you are using Maven (or similar) to incorporate tranquility-core into your project. Is that not happening?

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/abc17fcc-3637-45aa-b7be-fe70a1824736%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rasmus Schults]

Hey Gian,

Yes, it looks like it is not imported correctly.

Here is the pom.xml i'm using:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mycompany.app</groupId>

  <artifactId>my-app</artifactId>

  <packaging>jar</packaging>

  <version>1.0-SNAPSHOT</version>

  <name>my-app</name>

  <url>http://maven.apache.org</url>

  <dependencies>

    <dependency>

      <groupId>junit</groupId>

      <artifactId>junit</artifactId>

      <version>3.8.1</version>

      <scope>test</scope>

    </dependency>

    <dependency>

      <groupId>io.druid</groupId>

      <artifactId>tranquility-core_2.11</artifactId>

      <version>0.8.2</version>

    </dependency>

  </dependencies>

  <properties>

    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

    <maven.compiler.source>1.8</maven.compiler.source>

    <maven.compiler.target>1.8</maven.compiler.target>

  </properties>

  <build>

    <plugins>

      <plugin>

        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-dependency-plugin</artifactId>

        <version>2.10</version>

        <executions>

          <execution>

            <id>copy-dependencies</id>

            <phase>package</phase>

            <goals>

              <goal>copy-dependencies</goal>

            </goals>

            <configuration>

              <outputDirectory>${project.build.directory}/lib</outputDirectory>

            </configuration>

          </execution>

        </executions>

      </plugin>

    </plugins>

  </build>

</project>

And here is some test code:

package com.mycompany.app;

import java.io.InputStream;

import java.util.Map;

import com.metamx.tranquility.config.DataSourceConfig;

import com.metamx.tranquility.config.PropertiesBasedConfig;

import com.metamx.tranquility.config.TranquilityConfig;

import com.metamx.tranquility.druid.DruidBeams;

import com.metamx.tranquility.tranquilizer.Tranquilizer;

public class App  {

    public static void main( String[] args ) throws InterruptedException {

        final InputStream configStream = App.class.getClassLoader().getResourceAsStream("indexing.yml");

        final TranquilityConfig<PropertiesBasedConfig> druidConfig = TranquilityConfig.read(configStream);

        String dataSource = druidConfig.globalConfig().properties().getProperty("datasource");

        final DataSourceConfig<PropertiesBasedConfig> dataSourceConfig = druidConfig.getDataSource(dataSource);

        Tranquilizer<Map<String, Object>> service = DruidBeams.fromConfig(dataSourceConfig).buildTranquilizer(

            dataSourceConfig.tranquilizerBuilder()

        );

        service.start();

    }

}

Running this will crash with output:

Exception in thread "main" java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)

at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:101)

at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:301)

at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)

at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:123)

at com.metamx.tranquility.druid.DruidBeams.fromConfig(DruidBeams.scala)

at com.mycompany.app.App.main(App.java:32)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405)

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234)

at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167)

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398)

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064)

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264)

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156)

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126)

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113)

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84)

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132)

at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41)

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769)

... 7 more

Caused by: java.lang.ClassNotFoundException: com.metamx.common.parsers.JSONPathParser

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at io.druid.data.input.impl.JSONParseSpec.makeParser(JSONParseSpec.java:72)

at io.druid.data.input.impl.StringInputRowParser.<init>(StringInputRowParser.java:56)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125)

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230)

... 18 more

[邸星星]

<!-- https://mvnrepository.com/artifact/com.metamx/java-util -->
<dependency>
    <groupId>com.metamx</groupId>
    <artifactId>java-util</artifactId>
    <version>0.27.9</version>
</dependency>
<!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.9.4</version>
</dependency>

在 2016年5月18日星期三 UTC+8下午4:53:27，Pascal写道：