Upcoming Fluent Bit v0.13 and Kubernetes

[Eduardo Silva]

Hello Everyone, 

first of all I hope you have a successful 2018. 

Fluent Bit v0.13 is just around the corner, despite is still under development, there are a few new features that are coming that are waiting for your test and feedback:

1. Native Kafka Output
2. HTTP/REST Interface / Info and Metrics
3. Improved Kubernetes Filter

1. Native Kafka Output 

I'd say this is one of the primary requirements on 2017 by the community, hopefully the integration is done and we are in the test phase. The new plugin uses the librdkafka C library which is already bundled in Fluent Bit sources. 

A common configuration to test the kafka output is:

[OUTPUT]

    Name    kafka

    Match    *

    Brokers  127.0.0.1,192.168.1.5

              Format   json

              Topics    fluent-bit

You can set multiple brokers separated by commas, same applies for Topics. If you want to set multiple topics and let Fluent Bit route messages based on log content, specify the option Topic_Key, so if a record have a key that matches Topic_Key, it will take the value of Topic_Key and use the pre-defined Topic in the configuration.

2. HTTP/REST interface / Info and  Metrics

The new version also comes with a built-in HTTP server that provides information about the running service. The available end-points are:

- / : basic info about the service like version, built flags, edition, etc

- /api/v1/metrics: internal metrics in JSON format

- /api/v1/metrics/prometheus/: internal metrics in Prometheus format

- /api/v1/plugins: list of available plugins

When enabled, the HTTP server listens on TCP port 2020, you can enable and configure it in the SERVICE section as follows:

[SERVICE]

    HTTP_Server  On

    HTTP_Listen   0.0.0.0

    HTTP_Port      2020

3. Improved Kubernetes Filter

The big improvement here is the ability to let Pod (applications) 'suggest' a parser through annotations. The workflow is as follows:

- Fluent Bit runs with the new k8s-logging.parser feature enabled in the filter.

- Application deployed (not annotated later) have an annotation which suggest a known/pre-defined parser by Fluent Bit, e.g: logging.parser: apache

- Filter Kubernetes in Fluent Bit, process the log and look for metadata, if it finds a logging.parser reference and validate that the option is enabled (previous step), it will transform the log to structured mode using the parser specified.

As an example, run the following Pod/application that only writes random Apache logs:

apiVersion: v1

kind: Pod

metadata:

  name: apache-logs

  namespace: default

  annotations:

    logging.parser: apache

spec:

  containers:

  - name: apache-logs

    image: edsiper/apache_logs

    imagePullPolicy: Always

  restartPolicy: Always

How to test ?

In order to do testing easy, I've published a special branch in our fluent-bit-kubernetes-logging repository with further recipes and instructions, the Docker image pointed in the recipes is updated frequently:

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev

Please test and send us your feedback through our channels: Fluentd-Slack or Github repository

best, 

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Eduardo Silva]

hi Marc, 

thanks for the feedback, I did some tests but I could not reproduce the problem, looking at the recent code changes I've found a wrong conditional that may be related to the behavior you just mentioned.

I've just released the new version of the image with the latest changes and improvements:

   https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev

please retry using fluent/fluent-bit-0.13-dev:0.5 and let me know how that goes, 

regards, 

On Tue, Jan 23, 2018 at 9:38 AM, Marc Kamerbeek <marc.ka...@gmail.com> wrote:

Hi Eduardo,

I've tested your solution with the annotation driven parser. It seems to be working great! But I noticed the following:

I'm tailing some files with *_<NAMESPACE>*log. A log file from a pod with annotations is parsed correctly. So far so good. But a file from a pod without that annotation is not send to the output at all. 

Is the default behaviour of a pod without annotation that its not parsed to the output at all? I would expect that its sent to the output without touching any parser. Or am I missing something?

Regards,

Marc

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/cb86659b-7c0c-4be6-bd83-4060a356395b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eduardo Silva]

FYI: 

0.13-dev:0.7 has been released:

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev

general improvements on memory handling, out_kafka and filter_kubernetes.

[Eduardo Silva]

FYI:

0.13-dev:0.8 has been released:
 
https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev

this image comes with a critical fix on in_tail that affects memory usage when rotation happens at high frequency within others.

[Eduardo Silva]

FYI:

0.13-dev:0.9 has been released:

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev