Re: [kubernetes/kubernetes] StatefulSet pod stuck in ContainerCreating because (#54263)

[krmayankk]

@kubernetes/sig-storage-bugs

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[krmayankk]

@rootfs

[Jing Xu]

@krmayankk do you still have the master controller log so that we can take a look at it?

[krmayankk]

@jingxu97 it is easy to repro for me. I dont see anything interesting at log level 2. Are you iinterested in pv controller logs ? Do you have a log level in mind ?

[Huamin Chen]

/assign

[Huamin Chen]

@krmayankk it looks to me your kubelet is running with cAdvisor. Can you disable it? i.e. use --cadvisor-port=0 kubelet option

[Ian Chakeres]

Here is a related to @rootfs' hypothesis that cadvisor is involved - google/cadvisor#771

[krmayankk]

@rootfs our finding so far is

• we can confirm that the TearDown was not only failing for RBD volumes but also for secrets and empty dirs
• we had some daemonsets mounting hostPath / in their pods
• this resulted in all mount points for pods on a given host, to show up inside the daemonset pod which was mounting the /. This includes the rbd volumes, secrets, and emptydirs
• we have so far fixed the daemonset pod, and after that are unable to repro the issue of TearDown failing
• We can also confirm that our daemon sets pods were only mounting the root but not writing anything to it but still the TearDown was failing because of device or resource busy
• I would like to keep this issue open for a while to see if we find anything else. I will update the bug.

Thanks for helping out with all the pointers and issues.

[krmayankk]

More pointers from @rootfs placing here for everyone to look at correlate moby/moby#20560 (comment)
openshift/origin#14428 (comment)
#45935
#53373
#51835

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[k8s-ci-robot]

Closed #54263.

[krmayankk]

this is safe to close now
/close

[FANLONGFANLONG]

i run into same issue, may i know what solution is to solve the issue?

[FANLONGFANLONG]

@krmayankk