logstash-logback-encoder and 8 KB limit in UDP syslog

[Gaël Marziou]

Hi all,

It seems UDP or syslog has a limit of 8 KB, when app logs an exception stack trace and the json encoded log statement is larger than 8 KB, logstsh is confused and truncate the statement resulting in invalid JSON fragment.

logstash-logback-encoder has few options to limit stack trace depth:

https://github.com/logstash/logstash-logback-encoder#customizing-stack-traces

I think it's something we should consider.

Cheers,

Gael

[Gaël Marziou]

In fact, in ch.qos.logback.core.net.SyslogAppenderBase#setMaxMessageSize javadoc says:

/**
 * Maximum size for the syslog message (in characters); messages
 * longer than this are truncated. The default value is 65400 (which
 * is near the maximum for syslog-over-UDP). Note that the value is
 * characters; the number of bytes may vary if non-ASCII characters
 * are present.
 */

Maybe the 8 KB limit we experienced is due to app running on Windows, so this setting seems to be important enough to be added to config.

As we define the appenders in LoggingConfiguration, I suppose that the configuration in logback-spring.xml cannot be used at same time, so it means we might add other properties under jhipster.logging.logstash in application.yml.

Or maybe the appenders could be defined in logback-spring.xml and then added in LoggingConfiguration based on value of jhipster.logging.logstash.enabled.

Cheers,

Gael

[Pierre BESSON]

Hi Gael,

Thanks for giving feedback on logstash-encoder usage. You are correct in saying that what is set up in LoggingConfiguration cannot be customized in logback-spring.xml. That's why I felt a little bad when I hardcoded it. However, I didn't find an alternative way to enrich the logs with the additional information needed for microservices like appname, host, port. Also I wanted it to be easy to toggle on and off from the config server.

Also in a microservice context. It might be better for all services to report their log consistently, so bypassing the .XML kind of make sense to me. If you find custom configuration setups that seems important don't hesitate to add custom properties. Especially if it's to correct a windows problem, I see no problem with that.

Regards,
Pierre Besson

--
You received this message because you are subscribed to the Google Groups "JHipster dev team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jhipster-dev...@googlegroups.com.
To post to this group, send email to jhipst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/2d5b1b60-3a43-4cfd-9002-fa4f7b67a272%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gaël Marziou]

Hi Pierre,

There may be some easy way to do some day because currently spring cloud config server can serve other files than just app config files.

http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html#_serving_plain_text

combined with an enhancement to spring-boot that may happen someday:

https://github.com/spring-projects/spring-boot/issues/4061

Anyway, next week I'll try to find out what are the logstash-encoder settings that would be worth adding to jhipster config.

Cheers,

Gael

[Gaël Marziou]

I just found that the 8 KB limit comes from logstash itself.

https://github.com/elastic/logstash/issues/1505

So it will happen on all platforms, not only on Windows, but it can be worked around by changing logstash settings or by changing the logback settings on apps.

[Julien Dubois]

Gaël, could you open a issue for that? I guess that could affect quite a lot of people.

8K is still quite big, so maybe we should just drop some parts of the stacktrace, when it's too big? I mean, a 8Kb stacktrace isn't normal, anyway.

--
You received this message because you are subscribed to the Google Groups "JHipster dev team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jhipster-dev...@googlegroups.com.
To post to this group, send email to jhipst...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/f186ffcf-48b0-4d31-9ba7-f0ac58903fe1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Julien Dubois

Twitter: @juliendubois
Phone: +33 (0)6 25 02 34 18

[Gaël Marziou]

Here it is

https://github.com/jhipster/generator-jhipster/issues/3364

I won't have time to work on it until next week.

Gael