Protocol message end-group tag did not match expected tag

[Tahir Rauf]

I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf file which dictates to forward all messages matching particular pattern to my Riemann server

$template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,%msg:::json%}" 
:syslogtag,isequal,"tahir:" @127.0.0.1:15555;ls_json

Then I send a message using logger utility as following

logger "image [7a3cbb43:42:9]<340:01:52:53.032833>[50]: imghandler: Invoked for image" --tag="tahir"

I am expecting that Riemann will receive and print this message. However, I got following error on Riemann side





WARN [2015-12-06 04:18:54,453] defaultEventExecutorGroup-2-1 - riemann.transport.udp - UDP handler caught
io.netty.handler.codec.DecoderException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at riemann.transport.proxy$io.netty.handler.codec.MessageToMessageDecoder$ff19274a.channelRead(Unknown Source)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
        at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
        at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:139)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:168)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:174)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
        at io.netty.handler.codec.protobuf.ProtobufDecoder.decode(ProtobufDecoder.java:119)
        at io.netty.handler.codec.protobuf.ProtobufDecoder.decode(ProtobufDecoder.java:63)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
        ... 15 more

Can anyone tell me what I am doing wrong ???

[Aphyr]

On 12/06/2015 12:16 PM, Tahir Rauf wrote:
>
> I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf file
> which dictates to forward all messages matching particular pattern to my Riemann
> server

> |WARN [2015-12-0604:18:54,453]defaultEventExecutorGroup-2-1-riemann.transport.udp
> -UDP handler caught

> Can anyone tell me what I am doing wrong ???

UDP messages can be arbitrarily truncated. Why not use TCP?

--Kyle

[Tahir Rauf]

Thank you very much for your prompt response.

I agree that UDP message can be truncated arbitrarily and I will try TCP.

However, It seems that there is something else going on here as I am receiving these warnings for EVERY event. There is a low probability that EVERY message gets truncated. Is there any bug or am I doing something obvious wrong?

Thanks again for your very prompt response.

--
You received this message because you are subscribed to a topic in the Google Groups "Riemann Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/riemann-users/_JpAXpHU8Pk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to riemann-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tahir Rauf

--

(408) 797-5986

[Aphyr]

Dunno! Your encoder could be broken, but 9/10 times this is the network truncating packets.

You received this message because you are subscribed to the Google Groups "Riemann Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to riemann-user...@googlegroups.com.

[Marc Fournier]

Excerpts from Tahir Rauf's message of 2015-12-06 12:16:03 -0800:

>
> I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf
> file which dictates to forward all messages matching particular pattern to
> my Riemann server
>
> $template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,%msg:::json%}" :syslogtag,isequal,"tahir:" @127.0.0.1:15555;ls_json

With this you are sending JSON directly to Riemann's socket, aren't you ?

Riemann doesn't decode JSON-encoded events, only its own protobuf-based
protocol. And rsyslog doesn't have such an encoder.

You'll have to use something else than rsyslog, either as a replacement
or as a proxy. syslog-ng or logstash both can decode json sent over a
tcp socket or /dev/log and emit native riemann protobuf events. The
former having a much more performant encoder, which would make a
difference if your log rate is high.


HTH,
Marc

[Tahir Rauf]

Hi Marc,

Thanks alot. That precisely describes the problem.

Regards