logstash-encoder support

[Oleg Sigida]

Hi All,

We are developing some set of applications and use micro-services as architecture approach. 

So to keep logs in one place we are using logstash and kibana, which are quite popular this days.

Current implementation of Dropwizard does not allow us to send logs directly to logstash. One has to use some middle layer or to develop some extra code.

However there is a solution dropwizard-logstash-encoder (https://github.com/Wikia/dropwizard-logstash-encoder) which allows to send logs directly to logstash and configure it via native dropwizard way.

Unfortunately, the implementation has some issues (f.ex. not compatible with Java 7).

From my point of view, dropwizard-logstash-encoder should be a part of dropwizard as a independent module or maybe event built in as default option.

Could you please share your ideas on the topic? Does it worse to be ported to Dropwizard code base?

thanks a lot,

Oleg

[Matt Hurne]

Here at CommerceHub we have a couple of internal libraries
(logstash-support-logback and logstash-support-dropwizard) that we
have been quite happy with so far. Open-sourcing them has been on our
"someday" list for a while. They are designed to send logging events
to Redis. Then we have a logstash process grab the events from Redis
and put them in ElasticSearch. A logback Layout is provided which
produces a JSON representation of the logging events.
logstash-support-logback includes the Layout and Redis Appender;
logstash-support-dropwizard adds classes that can be used to easily
configure the Layout and Appender in a Dropwizard application. We use
these classes with both Java 7 and Java 8. If there's a lot of
interest perhaps I can prioritize getting this out under
https://github.com/commercehub-oss .

Matt Hurne

> --
> You received this message because you are subscribed to the Google Groups
> "dropwizard-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dropwizard-de...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ted M. Young]

+1 from me, would like to see these open-sourced.

;ted

[Oleg Sigida]

Ted, 

it is already open sourced https://github.com/Wikia/dropwizard-logstash-encoder 

has couple of bugs, but if you are on Java8 should works fine 

I was able to make it running on Java 7 

However the question was is it a good idea to port it into DW codebase?

I can submit a solution within 2 days if there will be a place anywhere.

thanks a lot,

Oleg

пятница, 27 февраля 2015 г., 1:45:36 UTC+1 пользователь Ted M. Young написал:

[Ted M. Young]

Hi Oleg,

Thanks for the pointer. However, I was actually replying to Matt about open-sourcing the CommerceHub libraries as well.

As for your question, I'm not in a good position to say, though my preference is generally to keep such libraries in separate modules that can be used as needed.

;ted

[Matt Hurne]

The documentation definitely needs work (in fact, there's really only
documentation for logstash-support-dropwizard at the moment), but here
it is:

https://github.com/commercehub-oss/logstash-support
https://github.com/commercehub-oss/logstash-support/tree/master/logstash-support-dropwizard

Let me know if you have any questions, concerns, trouble, etc!

Matt Hurne

[Carlo Barbara]

Thanks for open sourcing and sharing.

If you want people to find it easier, and you plan on supporting it, then transferring it to the DW organization is a possibility. There are few already there:

https://github.com/dropwizard

> email to dropwizard-dev+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "dropwizard-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-dev+unsubscribe@googlegroups.com.

[Smiti Sharma]

Hey Oleg,

How can I use https://github.com/Wikia/dropwizard-logstash-encoder  to send logs to logstash. After adding dependencies I am getting exception while running Dropwizard application as :

Exception in thread "main" java.lang.NoClassDefFoundError: io/dropwizard/logging/BootstrapLogging

...

Caused by: java.lang.ClassNotFoundException: io.dropwizard.logging.BootstrapLogging

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

Thanks

Smiti

[Oleg Sigida]

Dear Smiti,

I guess you have missed some dependencies. 

btw, I'm not a maintainer of dropwizard-logstash-encoder...

Best regards,

Oleg

понедельник, 17 октября 2016 г., 6:22:04 UTC+2 пользователь Smiti написал:

[Elad Wertzberger]

We had the same issue and we solve it in the Application by the following code:

@Override
public void run(C config, Environment environment) throws Exception {
    // TODO We need to align to the dropwizard standard
    // TODO "The intention of Dropwizard as a framework is to provide a single point of configuration (the configuration file) for all parts involved"
    // TODO This means that we need to use the dropwizard yml options and if we need to use a custom Logback appender, we will have to add it programatically.
    // TODO Have a look at io.dropwizard.logging.ConsoleAppenderFactory for an example
    String logBackConfigPath = "";
    try {
        //tell dropwizard to use Logback
        LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
        JoranConfigurator configurator = new JoranConfigurator();
        configurator.setContext(loggerContext);
        // Call context.reset() to clear any previous configuration, e.g. default
        // configuration. For multi-step configuration, omit calling context.reset().
        loggerContext.reset();
        logBackConfigPath = System.getProperty("logback.configurationFile");
        configurator.doConfigure(logBackConfigPath);
        LOGGER.info("Logback path is: {}", logBackConfigPath);
    } catch (Exception e) {
      ....