regard: syslog output plugin

963 views
Skip to first unread message

Krishna vinayagam

unread,
Apr 26, 2021, 5:17:10 AM4/26/21
to Fluent Bit
Hi Team,

Can you please help me to find the solution for my requirement?

Currently, I am trying to configure fluent-bit in 2 different Kubernetes clusters. (ex: A & B)
Fluent-bit version: 1.7

A cluster:

fluent-bit pod: tail(input) plugin to syslog (output) plugin setup

Note: tail log should be transferred to B cluster fluent-bit Syslog server by using a public load balancer.  

Configurations:
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Flush 1
        Daemon Off
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        #Parser docker
        #Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name syslog
        Mode udp
        Match *
        Port 5140
        syslog_format rfc5424
        syslog_maxsize 2048
        syslog_severity_key severity
        syslog_facility_key facility
        syslog_hostname_key hostname

B cluster:

fluent-bit pod: syslog(input)plugin to es (output) plugin setup

I am not getting any log from A cluster logs to B cluster by using Syslog plugin. I am not sure, do we need to set up a Syslog server also with fluent bit pod?

configurations:
[INPUT]
        Name syslog
        Mode udp
        Parser syslog-rfc5424
        Port 5140
        Listen 0.0.0.0
        Tag kube.*


    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name es
        Match kube.*
        Host logs.xxxxx.com
        Port 443
        tls  On
        HTTP_User zzzz
        HTTP_Passwd zzzz
        Logstash_Format On
        Retry_Limit False

A cluster fluent-bit pod logs:
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

[2021/04/26 08:14:04] [ info] [engine] started (pid=1)
[2021/04/26 08:14:04] [ info] [storage] version=1.1.1, initializing...
[2021/04/26 08:14:04] [ info] [storage] in-memory
[2021/04/26 08:14:04] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/04/26 08:14:04] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2021/04/26 08:14:04] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2021/04/26 08:14:04] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2021/04/26 08:14:04] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2021/04/26 08:14:04] [ info] [output:syslog:syslog.0] setup done for a174692d65fa34528a03dc16bb1b9866-5a744667f9f453d4.elb.us-west-2.amazonaws.com:5140
[2021/04/26 08:14:04] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/04/26 08:14:04] [ info] [sp] stream processor started
[2021/04/26 08:14:04] [ info] [input:tail:tail.0] inotify_fs_add(): inode=193150445 watch_fd=1 name=/var/log/containers/grafana-f57995bdb-fn4tm_petuumos_download-dashboards-95ab3014db45cff02f4c2cfb5c9ef2bb3707be543aa89c85c6976e863a87ec6a.log
[2021/04/26 08:14:04] [ info] [input:tail:tail.0] inotify_fs_add(): inode=33628476 watch_fd=2 name=/var/log/containers/grafana-f57995bdb-fn4tm_petuumos_grafana-7e5ad8929032f7d17bf55480a700bfddd593beda2d2141c5f43789c81e08df2d.log
[2021/04/26 08:14:04] [ info] [input:tail:tail.0] inotify_fs_add(): inode=415832 watch_fd=3 name=/var/log/containers/grafana-f57995bdb-fn4tm_petuumos_grant-permission-c18e0bd30acff4e167b802b343935e8e8d149d9929e94ff0ae1f9d8786d38fba.log
[2021/04/26 08:14:04] [ info] [input:tail:tail.0] inotify_fs_add(): inode=143180331 watch_fd=

B Cluster fluent-bit pod:

Fluent Bit v1.7.3
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

[2021/04/26 08:54:37] [ info] [engine] started (pid=1)
[2021/04/26 08:54:37] [ info] [storage] version=1.1.1, initializing...
[2021/04/26 08:54:37] [ info] [storage] in-memory
[2021/04/26 08:54:37] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/04/26 08:54:37] [ info] [in_syslog] UDP buffer size set to 32768 bytes
[2021/04/26 08:54:37] [ info] [in_syslog] UDP server binding 0.0.0.0:5140
[2021/04/26 08:54:37] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2021/04/26 08:54:37] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2021/04/26 08:54:37] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2021/04/26 08:54:37] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2021/04/26 08:54:37] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/04/26 08:54:37] [ info] [sp] stream processor started

Thanks,
Krishna kumar.V



Eduardo Silva

unread,
May 20, 2021, 9:32:48 AM5/20/21
to Krishna vinayagam, Fluent Bit
do you have any metrics information from Fluent BIt ?

--
You received this message because you are subscribed to the Google Groups "Fluent Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/369642c5-4a68-4534-a405-63bc765c48fcn%40googlegroups.com.


--
Reply all
Reply to author
Forward
0 new messages