Custom logging with lua-resty-logger-socket

[Kamil Gorlo]

Hi,

I am trying to setup custom logging with https://github.com/cloudflare/lua-resty-logger-socket altogether with https://github.com/detailyang/lua-resty-rfc5424.

Everything works, but what I see in rsyslog (in UDP mode) is that lines are mixed up. 

Here is one of lines in rsyslog output file:

>>

Jun  4 11:27:17 localhost nginx[47871]: Hello World!#012<135>Jun 04 11:27:19 localhost nginx[47871]: Hello World!

<<

What I understand is that when lua-resty-logger-socket buffers multiple log lines it simply adds them to buffer and then sends whole buffer as one message to syslog. Syslog treats it as one line (it replaces new line with #012 and facility_severity with <135>).

What can be done to prevent this?

Cheers,

Kamil

[Kamil Gorlo]

Answering my own question:

https://tools.ietf.org/html/rfc5426#section-3.1

3.1.  One Message Per Datagram

   Each syslog UDP datagram MUST contain only one syslog message, which
   MAY be complete or truncated.  The message MUST be formatted and
   truncated according to RFC 5424 [2].  Additional data MUST NOT be
   present in the datagram payload.

So, change to TCP is needed (then it works). TCP is also probably better in terms of big log messages (for sure for > 1KB).

[Bingwu Yang]

Hello.

What's your server rsyslog version ? my rsyslog version is
rsyslog-7.4.7-16.el7.x86_64 on Centos7. lua-resty-rfc5424 explicitly
append '\n" to allow rsyslog split the multi message in udp packet.

> --
> You received this message because you are subscribed to the Google Groups
> "openresty-en" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to openresty-en...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Kamil Gorlo]

I have exact the same version of rsyslog.

I know that lua-resty-rfc5424 explicitly adds '\n'. In fact you can see it in my example (#012). But as stated in RFC, in UDP mode it looks that rsyslog expects only one message per datagram.

When I change mode to TCP then everything works (also thanks to this '\n').

You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/byM_p2mjIwE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.

[Bingwu Yang]

Well. RFC-5426 said Each syslog UDP datagram MUST contain only one
syslog message. However, in my production rsyslog allow UDP datagram
with multiple messags.

Or you can set flush_limit to 0, force lua-resty-logger send message
every time :)

[Kamil Gorlo]

So, how it works in YOUR production? :)

[Bingwu Yang]

Well. I check the production again. And make a mistake, the
lua-resty-logger-socket's type is default tcp rather than udp. I'm so
sorry for misleading you.
My production config is as the following:

1. TCP with appropriate flush_limit which depends your env
2. UDP with flush_limit = 0, But you should notice the UDP datagram
max length is 2 ^ 16

Almost is TCP because people requires do not lost message event if
is in LAN. Wish it can help you :)

[Kamil Gorlo]

Ok, makes sense now, thanks!