Druid 0.11.0 release

[Gian Merlino]

We're excited to announce our latest release, Druid 0.11.0!

Druid 0.11.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds two major security features, TLS support and extension points for authentication and authorization.

Major new features include:

- TLS (a.k.a. SSL) support

- Extension points for authentication and authorization

- Double columns support

- cachingCost Balancer Strategy

- jq expression support in JSON parser

- Redis cache extension

- GroupBy performance improvements

- Various improvements to Druid SQL

You can download the release here: http://druid.io/downloads.html

The full release notes are here: https://github.com/druid-io/druid/releases/druid-0.11.0

Documentation for this release is at: http://druid.io/docs/0.11.0/

Thanks to everyone who contributed!

Gian

[zhou...@gmail.com]

Hi,

When i using Druid 0.11.0 with kafka-emitter extension to emit Druid metrics, i found kafka-emitter does not work normally . I suspect this is a bug .

[root@gpsegment1 druid-0.11.0]# ls extensions/kafka-emitter/
kafka-clients-0.10.2.0.jar  kafka-emitter-0.11.0.jar  lz4-1.3.0.jar  slf4j-api-1.6.4.jar  snappy-java-1.1.2.6.jar

[root@gpsegment1 druid-0.11.0]# grep "kafka-emitter" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties 
druid.extensions.loadList=["druid-hdfs-storage", "kafka-emitter", "druid-caffeine-cache", "druid-kafka-indexing-service", "mysql-metadata-storage", "druid-datasketches", "druid-histogram"]

[root@gpsegment1 druid-0.11.0]# grep "emitter.kafka" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties 
druid.emitter=kafka
druid.emitter.logging.logLevel=info
druid.emitter.kafka.bootstrap.servers=192.168.1.115:9092
druid.emitter.kafka.metric.topic=druid-metrics-monitor
druid.emitter.kafka.alert.topic=druid-metrics-alert
druid.emitter.kafka.producer.config={"max.block.ms":10000}

But error occurs when i startup services . My kafka version is 0.10.2 .

2017-12-27T15:55:43,216 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@20a7953c]
2017-12-27T15:55:43,337 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
2017-12-27T15:55:43,340 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2017-12-27T15:55:43,416 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d]
2017-12-27T15:55:43,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.security.Escalator] from props[druid.escalator.] as [io.druid.server.security.NoopEscalator@78422efb]
2017-12-27T15:55:43,480 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain='null', authorizers='null'}]
Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:

1. Problem parsing object at prefix[druid.emitter.kafka]: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null.
   at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
   at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
   while locating com.google.common.base.Supplier<io.druid.emitter.kafka.KafkaEmitterConfig>
   at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:134) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
   while locating io.druid.emitter.kafka.KafkaEmitterConfig
   for the 1st parameter of io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56)
   at io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
   while locating com.metamx.emitter.core.Emitter annotated with @com.google.inject.name.Named(value=kafka)
   at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:118)
   at io.druid.server.emitter.EmitterModule.configure(EmitterModule.java:78) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.emitter.EmitterModule)

1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:401)
at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:62)
at io.druid.cli.ServerRunnable.run(ServerRunnable.java:49)
at io.druid.cli.Main.main(Main.java:108)
Caused by: java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)
at io.druid.guice.JsonConfigurator.configurate(JsonConfigurator.java:103)
at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:200)
at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:80)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at io.druid.guice.SupplierProvider.get(SupplierProvider.java:52)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402)
at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at io.druid.server.emitter.EmitterModule$EmitterProvider.findEmitter(EmitterModule.java:142)
at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:120)
at io.druid.server.emitter.EmitterModule$EmitterProvider$$FastClassByGuice$$622b285e.invoke()
at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:54)
at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:89)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93)
at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80)
at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:223)
at com.google.inject.internal.Initializer.injectAll(Initializer.java:132)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:174)
... 8 more

Then i had to rollback to Druid 0.10.1, it works very well . 

Is this a bug of Druid 0.11.0 ?

Best wishes !

在 2017年12月5日星期二 UTC+8下午3:25:20，Gian Merlino写道：

[Alberto González Mesas]

Hi there!!

I am upgrading to kafka 2.11_1.0.0 (lastest) and Druid 0.11.0. In the older version, I was using tranquility but now I want to use kafka-indexing-service. I have problems when Druid trigger the handoff:

    "java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.FileSystem.rename(Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/Options$Rename;)V from class org.apache.hadoop.fs.HadoopFsWrapper"

If I install druid_0.10.0 with druid-hdfs-storage extensión (jars from 0.9.0 version) it runs fine but don't with own version jars (0.10.0).

Looking for into google, I found related links:

   https://github.com/druid-io/druid/pull/4363
   https://github.com/druid-io/druid/pull/4348
   https://github.com/druid-io/druid/pull/3650
   https://github.com/druid-io/druid/pull/3787

I can see that the bug isn't fixed yet, is true?

Thanks!

[Gian Merlino]

Hi Zhou,

Sorry, but yes this is a bug. It will be fixed in Druid 0.12.0 by this patch: http://github.com/druid-io/druid/pull/5201

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/5170d79e-39f3-4fdc-af6a-9a9662248fa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gian Merlino]

Hi Alberto,

Could you try double checking the hadoop jars you have on your classpath?

I think this should work fine out of the box, so I suspect you have some "extra" jars somewhere.

Gian

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/5aa478e8-2efe-424e-a418-0388ec6617ad%40googlegroups.com.

[Himanshu Gupta]

https://github.com/druid-io/druid/pull/3787#issuecomment-356443955

[Alberto González Mesas]

Gian, You were right, the problem was the loaded wrong libs.

My old systemd file:

    [Service]
    Type=simple

    User=druid
    Group=druid
    Environment='CLASSPATH=/opt/druid/lib/*:/opt/druid/conf/_common:/opt/druid/conf/_common/hadoop_conf:/opt/druid/conf/overlord:/opt/druid/hadoop-dependencies/hadoop-client/2.3.0/*'
    Environment='JMX=-Dcom.sun.management.jmxremote.port=3003 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false'


I fixed the hadoop-client path to 2.7.3

Thanks!!

Alberto

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

[zhou...@gmail.com]

Hi Gian,

I got it, Thank you very much !

Best wishes !

在 2018年1月9日星期二 UTC+8下午1:15:01，Gian Merlino写道：

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.