crashloopbackoff issue with Fluent bit on a 3 node k8's cluster with exit code 139 Need Help
1,855 views
Skip to first unread message
Venkateswarareddy Nakka
Apr 5, 2018, 11:14:15 AM4/5/18
to Fluent-Bit
Hi I am using latest fluent bit version and I deployed to kubernetes cluster UBUNTU on the three nodes on of the node is going to crashloopbackoff . Please help me on this
Eduardo Silva
Apr 5, 2018, 11:21:33 AM4/5/18
to Venkateswarareddy Nakka, Fluent-Bit
a Kubernetes error "crashloopbackoff" can be originated by different things.
Whats the output of 'kubectl describe pod YOUR_FLUENTBIT_PODNAME' ?
On Thu, Apr 5, 2018 at 9:14 AM, Venkateswarareddy Nakka <venn...@gmail.com> wrote:
Hi I am using latest fluent bit version and I deployed to kubernetes cluster UBUNTU on the three nodes on of the node is going to crashloopbackoff . Please help me on this
--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/63bd784d-c232-4012-9900-09add19767df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Message has been deleted
Venkateswarareddy Nakka
Apr 5, 2018, 12:15:52 PM4/5/18
to Fluent-Bit
Venkateswarareddy Nakka
Apr 5, 2018, 12:23:03 PM4/5/18
to Fluent-Bit
Please find the below Logs
===========================
kubectl describe pod fluent-bit-lzw6z -n logging
Name: fluent-bit-lzw6z
Namespace: logging
Node: host
Start Time: Thu, 05 Apr 2018 10:47:50 -0400
Labels: controller-revision-hash=1188023158
k8s-app=fluent-bit-logging
pod-template-generation=1
version=v1
Annotations: <none>
Status: Running
IP: 000000000000
Controlled By: DaemonSet/fluent-bit
Containers:
fluent-bit:
Container ID: docker://abc8237e7c09bd40e70e3599e4e802e60b6dac68c25a8b54f991492e6cf5b806
Image: fluent/fluent-bit-0.13-dev:0.14
Image ID: docker-pullable://fluent/fluent-bit-0.13-dev@sha256:662dfce052022637d7086e3e1e00ddd929b35d9db31dab0a19b0a7cf47ff4356
Port: <none>
State: Running
Started: Thu, 05 Apr 2018 11:21:39 -0400
Last State: Terminated
Reason: Error
Exit Code: 139
Started: Thu, 05 Apr 2018 11:21:21 -0400
Finished: Thu, 05 Apr 2018 11:21:23 -0400
Ready: True
Restart Count: 5
Limits:
memory: 200Mi
Requests:
cpu: 100m
memory: 200Mi
Environment:
FLUENT_COLLECTIONSERVER_HOST: host
FLUENT_COLLECTIONSERVER_PORT: 24224
Mounts:
/fluent-bit/etc/ from fluent-bit-config (rw)
/var/lib/docker/containers from varlibdockercontainers (ro)
/var/log from varlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from fluent-bit-token-mhpkr (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
varlibdockercontainers:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
fluent-bit-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: fluent-bit-config
Optional: false
fluent-bit-token-mhpkr:
Type: Secret (a volume populated by a Secret)
SecretName: fluent-bit-token-mhpkr
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 57m (x4 over 1h) kubelet, host Back-off restarting failed container
Normal Pulled 57m (x6 over 1h) kubelet, host Container image "fluent/fluent-bit-0.13-dev:0.14" already present on machine
Normal Created 57m (x6 over 1h) kubelet, host Created container
Normal Started 57m (x6 over 1h) kubelet, host Started container
==============================================
On Thursday, April 5, 2018 at 11:14:15 AM UTC-4, Venkateswarareddy Nakka wrote:
Eduardo Silva
Apr 5, 2018, 12:34:03 PM4/5/18
to Venkateswarareddy Nakka, Fluent-Bit
did you follow the Yaml recipes from https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev ?
--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/41a13e9e-0404-42e9-8a1d-21e4dff2c58f%40googlegroups.com.
Venkateswarareddy Nakka
Apr 5, 2018, 1:34:36 PM4/5/18
to Fluent-Bit
Yes We followed the Same .
Thanks
On Thursday, April 5, 2018 at 11:14:15 AM UTC-4, Venkateswarareddy Nakka wrote:
Venkateswarareddy Nakka
Apr 6, 2018, 9:29:35 AM4/6/18
to Fluent-Bit
I have tried the same on the 6 node cluster with the same yaml recipes . I am getting same error on one of the node . Can anyone Please look into this issue .
On Thursday, April 5, 2018 at 11:14:15 AM UTC-4, Venkateswarareddy Nakka wrote:
Eduardo Silva
Apr 10, 2018, 9:17:53 AM4/10/18
to Venkateswarareddy Nakka, Fluent-Bit
are you running some architecture different than x86_64 or your cluster don't have access to docker hub ?, I cannot find the root cause of the problem, I did a local tests and recipes works without problems.
--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/2b54f283-2238-4da9-9dad-1269cb064824%40googlegroups.com.
Venkateswarareddy Nakka
Apr 10, 2018, 10:01:11 AM4/10/18
to Fluent-Bit
- We are Using Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-87-generic x86_64)
- Our cluster has access DockerHub .
- When we click on the fluentbit PODs On Kubernetes dashboard , Then one of the POD is going to Crashloopbackoff .
On Thursday, April 5, 2018 at 11:14:15 AM UTC-4, Venkateswarareddy Nakka wrote:
Eduardo Silva
Apr 13, 2018, 6:37:16 PM4/13/18
to Venkateswarareddy Nakka, Fluent-Bit
Would you please check if you face the same issue with docker image fluent/fluent-bit:0.12.17 ?
--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/19e45328-f703-432f-95d4-c609de829e8c%40googlegroups.com.
Venkateswarareddy Nakka
Apr 17, 2018, 10:07:58 AM4/17/18
to Fluent-Bit
Hi Eduardo ,
We trying to forward JSON format message to fluentd , When we add following keys to the config map one of the POD is going to Crashloopbackoff . Please take a look into this and help me on this Keys
==========================
Merge_JSON_Log On
Merge_JSON_Key log
==========================
Below is the config map we are using
=============================
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_JSON_Log On
Merge_JSON_Key log
K8S-Logging.Parser On
output-collection-server.conf: |
[OUTPUT]
Name forward
Match *
Host ${FLUENT_COLLECTIONSERVER_HOST}
Port ${FLUENT_COLLECTIONSERVER_PORT}
Logstash_Format On
Time_as_Integer On
Retry_Limit False
=================================================================================================================
On Thursday, April 5, 2018 at 11:14:15 AM UTC-4, Venkateswarareddy Nakka wrote:
Eduardo Silva
Apr 24, 2018, 12:22:17 PM4/24/18
to Venkateswarareddy Nakka, Fluent-Bit
would you please retry with the new 0.13-dev image ?:
--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/6926b7d2-eeed-4e2d-99fe-cd75d4e96a3e%40googlegroups.com.
Priya Menon
Oct 16, 2019, 3:56:28 AM10/16/19
to Fluent-Bit
I see the same problem. I just moved my cluster from 3 node to 5 nodes. since then i see the error, restarting failed container and the pods are in CrashLoopBackOff
Reply all
Reply to author
Forward
Message has been deleted
0 new messages