Liar : td-agent or log4j.xml ? (Critical)

[sona sun]

Hi

  How come same logs processed in two ways? The logs which have which are saved as json in a file are stopped by td-agent saying pattern (which is json) not match

 This is my log4j.xml configuration for logs which saves (see below first Green section) the logs to /opt/tpa/logs/SPSServer.log and also, forwards them to fluentd (see below second Green section) at 8514 port.

    <!-- ================================= -->

    <!-- Preserve messages in a local file -->

    <!-- ================================= -->

    <appender name="SPSServerFileAppender"

              class="com.alcatel.tpapps.common.server.logging.log4jappenders.RollingZipFileAppender">

        <errorHandler class="org.apache.log4j.helpers.OnlyOnceErrorHandler"/>

        <param name="File" value="/opt/tpa/logs/SPSServer.log"/>

        <param name="Folding" value="true"/>

        <param name="Debug" value="true"/>

        <param name="MaxFileSize" value="30MB"/>

        <param name="CompressLogs" value="true"/>

        <param name="MaxBackupIndex" value="15"/>

        <param name="MaxBackupDir" value="-1"/>

        <!-- retain backup folders for 3 months (90 days)

             (units of the following parameter are ms)-->

        <param name="MaxBackupTime" value="7776000000"/>

        <param name="SavePreviousLogs" value="true"/>

        <param name="MaxArchiveFileNumber" value="500"/>

        <param name="MaxArchiveTime" value="-1"/>

        <param name="BackupDirTag" value="SPSServer"/>

        <param name="ConsoleLog" value="server_console.log"/>

        <layout class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedLoggingFormat">

        </layout>

        <filter class="com.alcatel.tpapps.common.utils.AddMDCFilter"/>

    </appender>

      <!--Use to forward logs to Fluentd-->

     <appender name="SocketAppender" class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedFormatSocketAppender">

     <param name="ReconnectionDelay" value="10000"/>

     <param name="Application" value="SPS"/>

     <param name="Port" value="8514"/>

     <param name="RemoteHost" value="localhost"/>

    </appender>

So logs in SPSServer.log are:

{"host":"sps-me1-diameterapp-0","level":"WARN","log":{"classname":"com.nokia.as.mux.reactor.Agent:76","message":"ReactorProvider Mux Agent started!","stacktrace":"","threadname":"Main-reactor"},"process":"init","service":"","time":"2021-01-08T01:21:33.470Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

{"host":"sps-me1-comsvc-0","level":"INFO","log":{"classname":"com.nokia.csf.cdr.common.util.Env:17","message":"Env is not initialized.","stacktrace":"","threadname":"FelixDispatchQueue"},"process":"isDockerEnv","service":"","time":"2021-01-08T01:19:54.815Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

{"host":"sps-me1-oame-0","level":"INFO","log":{"classname":"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245","message":"HB-sps-me1-R8292-sps-me2-O> Unable to contact service for 'sps-me2'.","stacktrace":"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]","threadname":"Heartbeat-4"},"process":"getMemberStatusesInternal","service":"","time":"2021-01-08T09:42:24.081Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

Now, my td-agent configuration to handle those logs at 8514 is

<worker 0>

  <source>

    @type tcp

    port 8514

    bind 0.0.0.0

    format /(^(?<header>[^\{]+)?(?<message>\{.+type.+\})$)|(^(?<log>[^\{].+))/

    tag system

  </source>

</worker>

<match system syslog.**>

  @type rewrite_tag_filter

  <rule>

    key message

    pattern .+

    tag nokia.logging.json

  </rule>

  <rule>

    key log

    pattern .+

    tag nokia.logging.clear

  </rule>

</match>

<filter nokia.logging.json>

  @type parser

  key_name message

  <parse>

    @type json

    time_key time

    time_type string

    keep_time_key true

  </parse>

</filter>

But td-agent gives following errors when parsing the same logs which are saved in SPSServer.log by log4j.xml (the same above logs.)

2021-01-08 06:51:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-diameterapp-0\",\"level\":\"WARN\",\"log\":{\"classname\":\"com.nokia.as.mux.reactor.Agent:76\",\"message\":\"ReactorProvider Mux Agent started!\",\"stacktrace\":\"\",\"threadname\":\"Main-reactor\"},\"process\":\"init\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:21:33.470Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 06:51:33.471639265 +0530 record={"header"=>"yt\x01", "message"=>"{{\"host\":\"sps-me1-diameterapp-0\",\"level\":\"WARN\",\"log\":{\"classname\":\"com.nokia.as.mux.reactor.Agent:76\",\"message\":\"ReactorProvider Mux Agent started!\",\"stacktrace\":\"\",\"threadname\":\"Main-reactor\"},\"process\":\"init\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:21:33.470Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

2021-01-08 09:41:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-comsvc-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.csf.cdr.common.util.Env:17\",\"message\":\"Env is not initialized.\",\"stacktrace\":\"\",\"threadname\":\"FelixDispatchQueue\"},\"process\":\"isDockerEnv\",\"service\":\"SPS\",\"time\":\"2021-01-08T04:11:33.780Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 09:41:33.781026491 +0530 record={"header"=>"yt\x01", "message"=>"{{\"host\":\"sps-me1-comsvc-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.csf.cdr.common.util.Env:17\",\"message\":\"Env is not initialized.\",\"stacktrace\":\"\",\"threadname\":\"FelixDispatchQueue\"},\"process\":\"isDockerEnv\",\"service\":\"SPS\",\"time\":\"2021-01-08T04:11:33.780Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

2021-01-08 07:08:44 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-oame-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245\",\"message\":\"HB-sps-me1-R30-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"stacktrace\":\"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]\",\"threadname\":\"Heartbeat-1\"},\"process\":\"getMemberStatusesInternal\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:38:44.064Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 07:08:44.067375936 +0530 record={"header"=>"yt\x11", "message"=>"{{\"host\":\"sps-me1-oame-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245\",\"message\":\"HB-sps-me1-R30-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"stacktrace\":\"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]\",\"threadname\":\"Heartbeat-1\"},\"process\":\"getMemberStatusesInternal\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:38:44.064Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

2021-01-08 07:09:44 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-oame-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245\",\"message\":\"HB-sps-me1-R60-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"stacktrace\":\"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]\",\"threadname\":\"Heartbeat-3\"},\"process\":\"getMemberStatusesInternal\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:39:44.083Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 07:09:44.091157349 +0530 record={"header"=>"yt\x11", "message"=>"{{\"host\":\"sps-me1-oame-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245\",\"message\":\"HB-sps-me1-R60-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"stacktrace\":\"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208), at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130), at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124), at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121), at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035), at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004), at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862), at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448), at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93), at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77), at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147), at com.sun.proxy.$Proxy267.getHeartBeatStatus(Unknown Source), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160), at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194), at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), at java.lang.Thread.run(Thread.java:748), Caused by: java.net.ConnectException: Connection refused (Connection refused), at java.net.PlainSocketImpl.socketConnect(Native Method), at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350), at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206), at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188), at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392), at java.net.Socket.connect(Socket.java:607), at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660), at sun.net.NetworkClient.doConnect(NetworkClient.java:175), at sun.net.www.http.HttpClient.openServer(HttpClient.java:463), at sun.net.www.http.HttpClient.openServer(HttpClient.java:558), at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264), at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191), at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205), at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056), at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340), at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315), at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264), at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104), ... 23 more]\",\"threadname\":\"Heartbeat-3\"},\"process\":\"getMemberStatusesInternal\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:39:44.083Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

[Mr. Fiber]

From td-agent log, td-agent seems to receive broken log.

log shows your data is '{{...}"}'

This seems broken json. How to generate this data?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/650fc30a-35cc-4559-bb9a-0b765e11aacan%40googlegroups.com.