fluentd /var/log/containers differnt symlinks

CD Truong

unread,

Sep 20, 2017, 2:15:01 PM9/20/17

to Fluentd Google Group

Hi,

I'm getting a behavior where the /var/log/containers symbolic link is not /var/lib/docker/container in docker 1.12.6 .

Originally, running docker 1.12.5 the /var/log/containers symlink is /var/lib/docker/container.....

Since docker 1.12.5 is no longer available in the docker repo, I have to use 1.12.6

Now that docker is running 1.12.6 the /var/log/containers symlinks is /var/log/pods/uid_folder/log_file.log

I'm running fluentd v0.12.31 + alpine 3.4

Thanks,

DT

Eduardo Silva

unread,

Sep 20, 2017, 3:05:35 PM9/20/17

to flu...@googlegroups.com

hi,

Is this related to a Kubernetes environment ?

the target file path of the link should not affect the behavior on in_tail and kubernetes metadata filter. Some questions:

1. which Kubernetes distro are you using ? (tectonic, openshift,..)

2. what's the main problem you are facing from a log collection perspective ?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

CD Truong

unread,

Sep 20, 2017, 3:44:52 PM9/20/17

to Fluentd Google Group

On Wednesday, September 20, 2017 at 2:05:35 PM UTC-5, Eduardo Silva wrote:

hi,

Is this related to a Kubernetes environment ?

Yes.

the target file path of the link should not affect the behavior on in_tail and kubernetes metadata filter. Some questions:

1. which Kubernetes distro are you using ? (tectonic, openshift,..)

Is neither openshift or tectonic. I believe is directly from kubernetes

2. what's the main problem you are facing from a log collection perspective ?

Normally /var/lib/docker/containers symlink is /var/log/containers. Now /var/log/container is the symlink for /var/log/pods and fluentd can't find the logs. What's causing the change in the path?

2017-09-19 23:33:28 +0000 [warn]: /var/log/containers/zookeeper-0_default_zookeeper-5ee49d0f703da37947e4f95434e7ec3106c2e79a80a0b21b3a83c21720189f3b.log unreadable. It is excluded and would be examined next time.

On Wed, Sep 20, 2017 at 12:15 PM, CD Truong <dcvt...@gmail.com> wrote:

Hi,

I'm getting a behavior where the /var/log/containers symbolic link is not /var/lib/docker/container in docker 1.12.6 .

Originally, running docker 1.12.5 the /var/log/containers symlink is /var/lib/docker/container.....

Since docker 1.12.5 is no longer available in the docker repo, I have to use 1.12.6

Now that docker is running 1.12.6 the /var/log/containers symlinks is /var/log/pods/uid_folder/log_file.log

I'm running fluentd v0.12.31 + alpine 3.4

Thanks,
DT

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Eduardo Silva

unread,

Sep 20, 2017, 3:48:19 PM9/20/17

to flu...@googlegroups.com

please share your daemonset yaml file (need to check the volumes and mount paths)

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

CD Truong

unread,

Sep 20, 2017, 3:56:56 PM9/20/17

to Fluentd Google Group

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

namespace: {{ KUBE_COMMON_NS }}

spec:

template:

metadata:

labels:

app: fluentd

namespace: {{ KUBE_COMMON_NS }}

spec:

containers:

- name: fluentd

image: Components.fluentd.image

command: [ "fluentd", "-c", "/etc/fluent/fluent.conf" ]

env:

- name: Container

value: fluentd

imagePullPolicy: Always

securityContext:

privileged: true

resources:

requests:

cpu: "1"

memory: 256Mi

volumeMounts:

# Mount log directory of the host readonly to access all log files.

- name: config-volume

mountPath: /etc/fluent

- name: varlogcontainers

mountPath: /var/log/containers

readOnly: true

- name: fluentdposfiles

mountPath: /var/lib/fluentd

- name: varlibdockercontainers

mountPath: /var/lib/docker/containers

readOnly: true

- name: varlog

mountPath: /var/log

readOnly: true

restartPolicy: Always

imagePullSecrets:

- name: tricregkey

terminationGracePeriodSeconds: 30

serviceAccountName: fluentd

volumes:

- name: config-volume

configMap:

- name: varlogcontainers

hostPath:

path: /var/log/containers

- name: fluentdposfiles

hostPath:

path: /tmp/fluentd

- name: varlibdockercontainers

hostPath:

path: /var/lib/docker/containers

- name: varlog

hostPath:

path: /var/log

Eduardo Silva

unread,

Sep 20, 2017, 5:31:09 PM9/20/17

to flu...@googlegroups.com

We need to know the Kubernetes distro you are using, e.g:

1. tectonic

2. openshift

3. minikube

4. etc

I ask this because at least on Minikube there are some mount path changes required, which could lead to broken links

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

CD Truong

unread,

Sep 20, 2017, 6:07:33 PM9/20/17

to Fluentd Google Group

it would be kubeadm

Eduardo Silva

unread,

Sep 20, 2017, 6:25:58 PM9/20/17

to flu...@googlegroups.com

please SSH into one of your worker nodes and do:

ls -l /var/log/containers.log

then provide the output

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

CD Truong

unread,

Sep 20, 2017, 6:35:32 PM9/20/17

to Fluentd Google Group

Are you referring to the Jenkin slave?

CD Truong

unread,

Sep 20, 2017, 6:45:58 PM9/20/17

to Fluentd Google Group

ls -l /var/log/containers.log

This is the fluentd container:

zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log

zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log

At the directory /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55

zookeeper_1.log -> /var/lib/docker/containers/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b-json.log

zookeeper_2.log -> /var/lib/docker/containers/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b-json.log

The path /var/lib/docker/containers is empty

This is on the host path node:

zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log

zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log

CD Truong

unread,

Sep 20, 2017, 6:54:34 PM9/20/17

to Fluentd Google Group

Additional information on the host path.

ls -l /var/log/containers.log

This is the fluentd container:
zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log
zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log

At the directory /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55
zookeeper_1.log -> /var/lib/docker/containers/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b-json.log
zookeeper_2.log -> /var/lib/docker/containers/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b-json.log

The path /var/lib/docker/containers is empty

This is on the host path node:
zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log
zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log -> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log

At the path when I list the file /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/

zookeeper_1.log -> /var/lib/docker/containers/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b-json.log

At the path /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/ I was able to view the logs with,

# more zookeeper_1.log

I can see the logs.

Message has been deleted

CD Truong

unread,

Sep 25, 2017, 12:13:15 PM9/25/17

to Fluentd Google Group

Hi Eduardo,

Is there any update on the /var/lib/docker/containers symlinks /var/log/pods/....?

Thanks.

Eduardo Silva

unread,

Sep 25, 2017, 12:51:14 PM9/25/17

to flu...@googlegroups.com

note: at this level the problem is when configuring Fluentd Pod, related to Kubernetes not Fluentd it self.

Please describe one of your FLuentd pods, e.g:

$ kubectl describe pod fluentd-...

the goal of that command is to verify that all volumes were mounted properly. Note that "/var/lib/docker/containers" must not be empty.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

CD Truong

unread,

Sep 25, 2017, 1:31:51 PM9/25/17

to Fluentd Google Group

Name: fluentd-6qp5p

Namespace: default

Node: node_hostname/172.22.16.61

Start Time: Fri, 22 Sep 2017 18:19:20 +0000

Labels: app=fluentd

pod-template-generation=1

Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"default","name":"fluentd","uid":"8d7743d9-9fc2-11e7-a086-00505691ce55","apiVersion...

Status: Running

IP: 10.x.0.x

Controllers: DaemonSet/fluentd

Containers:

fluentd:

Container ID: docker://8a4665a6c0d60522d4fcc1eb51326a15b0a63abe7e7dfd57e1081980b1265ce8

Image: devhub-docker.cisco.com/vnf-docker-dev/fluentd:0.12.31.345.95edf97a

Image ID: docker-pullable://devhub-docker.cisco.com/vnf-docker-dev/fluentd@sha256:bb390bd767533b386b9290680143dce161aa96de38feb433abc3c7bef443a7c4

Port:

Command:

fluentd

-c

/etc/fluent/fluent.conf

State: Running

Started: Fri, 22 Sep 2017 18:19:24 +0000

Ready: True

Restart Count: 0

Requests:

cpu: 1

memory: 256Mi

Environment:

Container: fluentd

Mounts:

/etc/fluent from config-volume (rw)

/var/lib/docker/containers from varlibdockercontainers (ro)

/var/lib/fluentd from fluentdposfiles (rw)

/var/log from varlog (ro)

/var/log/containers from varlogcontainers (ro)

/var/run/secrets/kubernetes.io/serviceaccount from fluentd-token-jn5bl (ro)

Conditions:

Type Status

Initialized True

Ready True

PodScheduled True

Volumes:

config-volume:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: fluentd-config

Optional: false

varlogcontainers:

Type: HostPath (bare host directory volume)

Path: /var/log/containers

fluentdposfiles:

Type: HostPath (bare host directory volume)

Path: /tmp/fluentd

varlibdockercontainers:

Type: HostPath (bare host directory volume)

Path: /var/lib/docker/containers

varlog:

Type: HostPath (bare host directory volume)

Path: /var/log

fluentd-token-jn5bl:

Type: Secret (a volume populated by a Secret)

SecretName: fluentd-token-jn5bl

Optional: false

QoS Class: Burstable

Node-Selectors: <none>

Tolerations: node.alpha.kubernetes.io/notReady=:Exists:NoExecute

node.alpha.kubernetes.io/unreachable=:Exists:NoExecute

Events: <none>

Rich Megginson

unread,

Sep 25, 2017, 1:35:44 PM9/25/17

to flu...@googlegroups.com, CD Truong

Remove /var/log/containers - you already get that with /var/log

> /var/run/secrets/kubernetes.io/serviceaccount from
> fluentd-token-jn5bl (ro)
> Conditions:
> Type Status
> Initialized True
> Ready True
> PodScheduled True
> Volumes:
> config-volume:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: fluentd-config
> Optional: false
> varlogcontainers:
> Type: HostPath (bare host directory volume)
> Path: /var/log/containers

remove /var/log/containers - you already get that with /var/log

> *zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log
> ->
> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log*
> * zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log
> ->
> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log
> *
>
> At the
> directory* /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55*

> zookeeper_1.log ->
> /var/lib/docker/containers/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b-json.log
> zookeeper_2.log ->
> /var/lib/docker/containers/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b/94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b-json.log
>

> The path*/var/lib/docker/containers* is empty
> *
> *

> This is on the host path node:

> * zookeeper-0_default_zookeeper-0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b.log
> ->
> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_1.log*
> * zookeeper-0_default_zookeeper-94839840173e0c2a7d9170668cddf36f2ba551cf85eecc5cd813cbb8cfa80d4b.log
> ->
> /var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/zookeeper_2.log*

>
>
> At the path when I list the file

> */var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/ *
> zookeeper_1.log ->
> /var/lib/docker/containers/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b/0fb430a5b497f94353899c1d4b6f28435a9d9d5df1abb12423bb4e80b571843b-json.log
> *
> *
> * At the path
> **/var/log/pods/21634f7b-9990-11e7-bafe-00505691ce55/ **I
> was able to view the logs with,*
> * # more **zookeeper_1.log*
> *
> *
> * I can see the logs.*

> *2017-09-19 23:33:28 +0000

> [warn]:
> /var/log/containers/zookeeper-0_default_zookeeper-5ee49d0f703da37947e4f95434e7ec3106c2e79a80a0b21b3a83c21720189f3b.log
> unreadable. It is excluded and

> would be examined next time. *

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Eduardo Silva
> Open Source, Treasure Data
> http://www.treasuredata.com/opensource

> <http://www.treasuredata.com/opensource>
> http://twitter.com/edsiper
> <http://twitter.com/edsiper>
> http://www.linkedin.com/in/edsiper

> <http://www.linkedin.com/in/edsiper>
>
>
> --
> You received this message
> because you are subscribed to
> the Google Groups "Fluentd
> Google Group" group.
> To unsubscribe from this group
> and stop receiving emails from
> it, send an email to
> fluentd+u...@googlegroups.com.
> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Eduardo Silva
> Open Source, Treasure Data
> http://www.treasuredata.com/opensource

> <http://www.treasuredata.com/opensource>
> http://twitter.com/edsiper
> <http://twitter.com/edsiper>
> http://www.linkedin.com/in/edsiper

> <http://www.linkedin.com/in/edsiper>
>
> --
> You received this message because you
> are subscribed to the Google Groups
> "Fluentd Google Group" group.
> To unsubscribe from this group and
> stop receiving emails from it, send an
> email to fluentd+u...@googlegroups.com.
> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Eduardo Silva
> Open Source, Treasure Data
> http://www.treasuredata.com/opensource

> <http://www.treasuredata.com/opensource>
> http://twitter.com/edsiper
> <http://twitter.com/edsiper>
> http://www.linkedin.com/in/edsiper

> <http://www.linkedin.com/in/edsiper>
>
> --
> You received this message because you are
> subscribed to the Google Groups "Fluentd
> Google Group" group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to
> fluentd+u...@googlegroups.com.
> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Eduardo Silva
> Open Source, Treasure Data
> http://www.treasuredata.com/opensource

> <http://www.treasuredata.com/opensource>
> http://twitter.com/edsiper
> <http://twitter.com/edsiper>
> http://www.linkedin.com/in/edsiper

> <http://www.linkedin.com/in/edsiper>
>
> --
> You received this message because you are subscribed to the
> Google Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to fluentd+u...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Eduardo Silva
> Open Source, Treasure Data
> http://www.treasuredata.com/opensource

> <http://www.treasuredata.com/opensource>
> http://twitter.com/edsiper <http://twitter.com/edsiper>
> http://www.linkedin.com/in/edsiper

> <http://www.linkedin.com/in/edsiper>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fluentd+u...@googlegroups.com

> <mailto:fluentd+u...@googlegroups.com>.

Andrew Emmons

unread,

Oct 25, 2017, 8:50:10 AM10/25/17

to Fluentd Google Group

What configuration do you use for minikube? I have mounted /var/lib/docker/containers in my fluentd daemonset but still get an "unreadable" warning in fluentd logs for all currently running containers.

The /var/lib/docker directory is owned by root and has the following permissions: drwx--x--x 10 root root 4096 Oct 24 09:14 docker

In my Fluentd Dockerfile I specify `User root`.

What do I need to do to allow the fluentd process read access to these files?

My Setup:

Macosx 10.12.6

vm-driver: virtualbox

$ docker --version

Docker version 1.12.6, build 78d1802

minikube version: v0.22.3

kubectl:

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-10-05T00:40:04Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-10-06T20:53:14Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Reply all

Reply to author

Forward