How to config log in CDAP so that user applications can have other log appenders?

chuncheng

unread,

Sep 29, 2016, 5:43:51 AM9/29/16

to CDAP User

Hi,

Beside the default the log appender provided by CDAP which writes logs to log files, we also need to send the logs to central log servers via kafka.

For example, below is the appender we want to add.

<level>${logging.console.level:-INFO}</level>

</filter>

<prefix>MyPrefix</prefix>

</layout>

</encoder>

<topic>event.aggregation.default</topic>

<alertTopic>event.aggregation.priority</alertTopic>

<bootstrapServer>kafka-bold-qed.int.thomsonreuters.com:9092</bootstrapServer>

<syncSend>false</syncSend>

<headerNames>timestamp,eventSourceUUID,eventType,eventSeverity</headerNames>

</appender>

And I think adding this configuration to logback-container.xml is not the best way, because we only want to send logs of our application to the central log servers, not including CDAP internal services.

Shankar Selvam

unread,

Sep 29, 2016, 9:19:45 PM9/29/16

to cdap...@googlegroups.com

Hi chuncheng,

you can create a logger for your application classes and attach the appender ECPKafkaLogbackAppender for them.

Example, If say all your application classes are in package "com.example"

you can add this to logback-container.xml

<appender-ref ref="ECPKafkaLogbackAppender"/>

</logger>

This will add the logs from "com.example" classes to ECPKafkaLogbackAppender along with sending it to CDAP's RollingFileAppender, but the system services logs aren't sent to ECPKafkaLogbackAppender as we have configured only the logger for "com.example" to use ECPKafkaLogbackAppender.

Reference http://logback.qos.ch/manual/configuration.html

Thanks

Shankar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/51271374-828f-46d3-80ec-2de8bbfff013%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

chuncheng

unread,

Sep 30, 2016, 4:55:05 AM9/30/16

to CDAP User

Hi Shankar,

Thank you for the reply. I still have one question, where can I put the configuration of appender "ECPKafkaLogbackAppender"? also in logback-container.xml?

I mean, do I need to put below lines in logback-container.xml?

<level>${logging.console.level:-INFO}</level>

</filter>

<prefix>MyPrefix</prefix>

</layout>

</encoder>

<topic>event.aggregation.default</topic>

<alertTopic>event.aggregation.priority</alertTopic>

<bootstrapServer>kafka-bold-qed.int.thomsonreuters.com:9092</bootstrapServer>

<syncSend>false</syncSend>

<headerNames>timestamp,eventSourceUUID,eventType,eventSeverity</headerNames>

</appender>

Also, when I want to have a try on the standalone CDAP, I see below errors in the log:

2016-09-30 16:27:41,935 - ERROR [ServiceHttpServer-executor-2:c.c.c.d.a.KafkaAuditPublisher@74] - Got exception publishing audit message AuditMessage{version=1, time=1475224061933, entityId=dataset:default.whom, user='', type=ACCESS, payload=AccessPayload{accessType=READ, accessor=program_run:default.LogHelloWorld.service.LogGreeting.87b32215-86e7-11e6-83a3-0000001a904f} AuditPayload{}}. Exception:

java.util.concurrent.ExecutionException: java.lang.IllegalStateException: No kafka producer available.

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]

at co.cask.cdap.data2.audit.KafkaAuditPublisher.publish(KafkaAuditPublisher.java:72) ~[co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.audit.AuditPublishers.publishAccess(AuditPublishers.java:72) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework.publishAudit(LineageWriterDatasetFramework.java:211) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework.writeLineage(LineageWriterDatasetFramework.java:186) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework$BasicDatasetAccessRecorder.recordLineage(LineageWriterDatasetFramework.java:263) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.dataset2.DefaultDatasetRuntimeContext.recordAccess(DefaultDatasetRuntimeContext.java:178) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.dataset2.DefaultDatasetRuntimeContext.onMethodEntry(DefaultDatasetRuntimeContext.java:166) [co.cask.cdap.cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.api.dataset.lib.KeyValueTable.read(KeyValueTable.java) [na:na]

at co.cask.cdap.examples.helloworld.LogHelloWorld$LogGreetingHandler.greet(LogHelloWorld.java:187) [unpacked/:na]

at co.cask.cdap.examples.helloworld.LogHelloWorld$LogGreetingHandlerb532e958911516c047bef3bd21bcdf4e.greet(Unknown Source) [na:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]

at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:80) [co.cask.http.netty-http-0.15.0.jar:na]

at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:38) [co.cask.http.netty-http-0.15.0.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]

Caused by: java.lang.IllegalStateException: No kafka producer available.

If the kafka producer can't work, will it affect my testing?

My OS is windows7, CDAP version 3.5.1, Node.js 4.4.0, Maven 3.3.9

Thanks,

Chuncheng

在 2016年9月30日星期五 UTC+8上午9:19:45，shankar写道：

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

Shankar Selvam

unread,

Oct 3, 2016, 8:48:53 PM10/3/16

to cdap...@googlegroups.com

Hi ChunCheng,

Replies in line.

where can I put the configuration of appender "ECPKafkaLogbackAppender"? also in logback-container.xml?

Yes. it has to be in logback-conatiner.xml.

Also, when I want to have a try on the standalone CDAP, I see below errors in the log:

For standalone you would use the logback.xml in standalone module,
https://github.com/caskdata/cdap/blob/develop/cdap-standalone/src/main/resources/logback.xml

2016-09-30 16:27:41,935 - ERROR [ServiceHttpServer-executor-2:c.c.c.d.a.KafkaAuditPublisher@74] - Got exception publishing audit message AuditMessage{version=1, time=1475224061933, entityId=dataset:default.whom, user='', type=ACCESS, payload=AccessPayload{accessType=READ, accessor=program_run:default.LogHelloWorld.service.LogGreeting.87b32215-86e7-11e6-83a3-0000001a904f} AuditPayload{}}. Exception:

java.util.concurrent.ExecutionException: java.lang.IllegalStateException: No kafka producer available.

If the kafka producer can't work, will it affect my testing?

Standalone uses kafka for audit logging and seems like the kafka server isn't running causing audit log message failures. are you running the CDAP SDK ?

About your question will it affect your testing, it depends on how the com.thomsonreuters.enterpriselogging.appender.logback.EnterpriseKafkaLogbackAppender is implemented.

Thanks

Shankar

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/b4ff0dac-c9a6-4f21-b2f2-fab0a92cd86c%40googlegroups.com.

chuncheng

unread,

Oct 11, 2016, 8:57:47 AM10/11/16

to CDAP User

Hi Shankar,

If the configuration needs to be put in logback-container.xml, the class "com.thomsonreuters.enterpriselogging.appender.logback.EnterpriseKafkaLogbackAppender" need to be visible to CDAP system class loader so that the configuration can work.

We need a way that we can upload custom jars. I can see a open request https://issues.cask.co/browse/CDAP-6183 for this issue.

Is there a workaround?

Thanks,

Chuncheng

Shankar Selvam

unread,

Oct 11, 2016, 5:07:06 PM10/11/16

to cdap...@googlegroups.com

Hi Chuncheng,

One workaround for this issue is to make the custom jar available across the cluster and include the path to that jar in "yarn.application.classpath" in yarn-site.xml.

If the jar is a fat jar and it includes other dependencies, you might want to make sure this jar doesn't cause conflicts with libraries used by CDAP, Example : CDAP depends on Kafka-0.8.2.2 version, you would want to make sure your jar doesn't pull different version of kafka, etc.

Try it out and Let us know if you face any issues.

Thanks

Shankar

--

You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/32ccb7bd-1fc3-466b-9db7-f024e96c8429%40googlegroups.com.

Rohit Sinha

unread,

Oct 17, 2016, 4:19:57 PM10/17/16

to CDAP User

Hello Chuncheng,

I wanted to follow up with you to see if you got a chance to try the workaround mentioned by Shankar earlier.

Please let us know if you need any other help in this regard.

Thanks,

Rohit

On Tuesday, October 11, 2016 at 2:07:06 PM UTC-7, shankar wrote:

Hi Chuncheng,

One workaround for this issue is to make the custom jar available across the cluster and include the path to that jar in "yarn.application.classpath" in yarn-site.xml.

If the jar is a fat jar and it includes other dependencies, you might want to make sure this jar doesn't cause conflicts with libraries used by CDAP, Example : CDAP depends on Kafka-0.8.2.2 version, you would want to make sure your jar doesn't pull different version of kafka, etc.

Try it out and Let us know if you face any issues.

Thanks
Shankar

On Tue, Oct 11, 2016 at 5:57 AM, chuncheng <web.fee...@gmail.com> wrote:

Hi Shankar,

If the configuration needs to be put in logback-container.xml, the class "com.thomsonreuters.enterpriselogging.appender.logback.EnterpriseKafkaLogbackAppender" need to be visible to CDAP system class loader so that the configuration can work.
We need a way that we can upload custom jars. I can see a open request https://issues.cask.co/browse/CDAP-6183 for this issue.

Is there a workaround?

Thanks,
Chuncheng

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

Reply all

Reply to author

Forward