FailedPostStartHook

Derek Mahar

unread,

Sep 14, 2016, 2:03:00 PM9/14/16

to Kubernetes user discussion and Q&A

Why does the following container fail due to FailedPostStartHook?

# Image javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT

FROM busybox

# Add BRVM configuration files.
ADD brvm/* /config/

# Add script to copy configuration files into shared volume.
ADD copy.sh /

VOLUME /etc/opes/platinum/fix/brvm

# Run script.
CMD ["tail" "-f" "/dev/null"]

---

# Pod defintion

apiVersion: v1
kind: Pod
metadata:
name: platinum-fix
spec:
containers:

- name: fix-brvm
    image: javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT
    imagePullPolicy: Always
    volumeMounts:
    - name: fix-brvm-config
      mountPath: /etc/opes/platinum/fix/brvm
    lifecycle:
      postStart:
        exec:
          command: [ "/bin/sh", "copy.sh" ]

volumes:
- name: fix-brvm-config
    emptyDir: {}

---

derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/platinum/platinum-fix-group/platinum-fix/src/main/kubernetes$ kubectl create -f platinum-fix-pod-2.yaml
pod "platinum-fix" created
derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/platinum/platinum-fix-group/platinum-fix/src/main/kubernetes$ kubectl describe pod platinum-fix
Name:        platinum-fix
Namespace:    default
Node:        172.17.4.201/172.17.4.201
Start Time:    Wed, 14 Sep 2016 13:56:14 -0400
Labels:        <none>
Status:        Running
IP:        10.2.89.15
Controllers:    <none>
Containers:
fix-brvm:
    Container ID:        docker://01d0159aff38756663044720afacd85cba4386d24ce0776c52f8cfa871fa125e
    Image:            javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT
    Image ID:            docker://sha256:f54145b319cfc7bb18be0c355126caede6217c262f2d4f84f2d69fbaaa74cc61
    Port:
    State:            Waiting
      Reason:            CrashLoopBackOff
    Last State:            Terminated
      Reason:            Error
      Exit Code:        127
      Started:            Wed, 14 Sep 2016 13:56:16 -0400
      Finished:            Wed, 14 Sep 2016 13:56:16 -0400
    Ready:            False
    Restart Count:        1
    Environment Variables:    <none>
Conditions:
Type        Status
Initialized     True
Ready     False
PodScheduled     True
Volumes:
fix-brvm-config:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
default-token-1lggo:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-1lggo
QoS Tier:    BestEffort
Events:
FirstSeen    LastSeen    Count    From            SubobjectPath            Type        Reason            Message
---------    --------    -----    ----            -------------            --------    ------            -------
9s        9s        1    {default-scheduler }                    Normal        Scheduled        Successfully assigned platinum-fix to 172.17.4.201
8s        8s        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Started            Started container with docker id c6fbd5b818a3
8s        8s        2    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Pulling            pulling image "javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT"
8s        8s        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Created            Created container with docker id c6fbd5b818a3
8s        8s        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Killing            Killing container with docker id c6fbd5b818a3: PostStart handler: Unrecognized input header
8s        7s        2    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Warning        FailedPostStartHook
8s        7s        2    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Pulled            Successfully pulled image "javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT"
8s        7s        2    {kubelet 172.17.4.201}                    Warning        FailedSync        Error syncing pod, skipping: failed to "StartContainer" for "fix-brvm" with RunContainerError: "PostStart handler: Unrecognized input header"

7s    7s    1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal    Created        Created container with docker id 01d0159aff38
7s    7s    1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal    Started        Started container with docker id 01d0159aff38
7s    7s    1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal    Killing        Killing container with docker id 01d0159aff38: PostStart handler: Unrecognized input header
6s    5s    2    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Warning    BackOff        Back-off restarting failed docker container
6s    5s    2    {kubelet 172.17.4.201}                    Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "fix-brvm" with CrashLoopBackOff: "Back-off 10s restarting failed container=fix-brvm pod=platinum-fix_default(876c61bc-7aa4-11e6-a592-0800278ab84d)"

Rodrigo Campos

unread,

Sep 14, 2016, 5:54:03 PM9/14/16

to kubernet...@googlegroups.com

Probably because there is no shebang on the script? And does it have exec permission? The error said it didn't recognize it..

Also, you don't specify the location of the script to run.

You use command: [ "/bin/sh", "copy.sh" ]

But probably copy.sh is not in the PATH, so you need to use "/copy.sh".

Does that fix it?

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Derek Mahar

unread,

Sep 14, 2016, 6:56:20 PM9/14/16

to Kubernetes user discussion and Q&A

Script copy.sh shouldn't need shebang (#!) or exec permission because the command invokes the script through the given shell. Shouldn't need to specify the root directory, either, because according to the documentation for ExecAction, the command working directory is root:

"Command is the command line to execute inside the container, the working directory for the command is root (/) in the container’s filesystem. The command is simply exec’d, it is not run inside a shell, so traditional shell instructions ('

', etc) won’t work. To use a shell, you need to explicitly call out to that shell. Exit status of 0 is treated as live/healthy and non-zero is unhealthy."

I'll add the root path, just in case, though.

Note that this command works when used inside a Docker image.

Derek

Derek Mahar

unread,

Sep 14, 2016, 6:58:56 PM9/14/16

to Kubernetes user discussion and Q&A

Still fails even as [ "/bin/sh", "/copy.sh" ].

Derek Mahar

unread,

Sep 14, 2016, 7:00:41 PM9/14/16

to Kubernetes user discussion and Q&A

Even [ "ls" ] fails.

Derek Mahar

unread,

Sep 14, 2016, 7:02:03 PM9/14/16

to Kubernetes user discussion and Q&A

...and [ "/bin/ls" ].

Cole Mickens

unread,

Sep 14, 2016, 7:06:00 PM9/14/16

to kubernet...@googlegroups.com

There are a number of confusing, and ongoing issues in the Docker repo that feature that error message: ""PostStart handler: Unrecognized input header"". I couldn't really make heads or tails of it from the quick look I had last night.

--

Rodrigo Campos

unread,

Sep 14, 2016, 7:13:53 PM9/14/16

to kubernet...@googlegroups.com

And do they work if you run the container locally? Does busybox install sh, ls, etc?

On Wednesday, September 14, 2016, Derek Mahar <derek...@gmail.com> wrote:

--

Derek Mahar

unread,

Sep 14, 2016, 7:43:57 PM9/14/16

to Kubernetes user discussion and Q&A

I apologize. I forgot to show you copy.sh! Had I shown it to you, I might have caught my error. Turns out that I had left "tail -f /dev/null" at the end of the script, blocking the post-start hook. Moving the tail to the container command, the post-start hook script correctly copies the files, exits, and the endless tail in the container blocks the termination of the pod, which is what I want:

# copy.sh
cp /config/* /etc/opes/platinum/fix/brvm/
ls /etc/opes/platinum/fix/brvm/
#tail -f /dev/null

---

FROM busybox

# Add BRVM configuration files.
ADD brvm/* /config/

# Add script to copy configuration files into shared volume.
ADD copy.sh /

VOLUME /etc/opes/platinum/fix/brvm

# Run script.

CMD ["/bin/tail", "-f", "/dev/null"]

---

apiVersion: v1
kind: Pod
metadata:
name: platinum-fix
spec:
containers:

- name: fix-brvm
    image: javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT
    imagePullPolicy: Always
    volumeMounts:
    - name: fix-brvm-config
      mountPath: /etc/opes/platinum/fix/brvm
    lifecycle:
      postStart:
        exec:

command: [ "/bin/sh", "/copy.sh" ]

volumes:
- name: fix-brvm-config
emptyDir: {}

---

derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/platinum/platinum-fix-group/platinum-fix/src/main/kubernetes$ kubectl describe pod platinum-fix
Name:        platinum-fix
Namespace:    default
Node:        172.17.4.201/172.17.4.201

Start Time:    Wed, 14 Sep 2016 19:30:49 -0400
Labels:        <none>
Status:        Running
IP:        10.2.89.22
Controllers:    <none>
Containers:
fix-brvm:
    Container ID:        docker://61df8621e2afbf324155ba46cdd1380ede7095f34be202401beb87bcb1f12a03
    Image:            javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT
    Image ID:            docker://sha256:1816c46ecd422a7b7be62b1519e28bec963a9b93b71194fcf8d5b39d29a655b9
    Port:
    State:            Running
      Started:            Wed, 14 Sep 2016 19:30:49 -0400
    Ready:            True
    Restart Count:        0

    Environment Variables:    <none>
Conditions:
Type        Status
Initialized     True

Ready True

PodScheduled     True
Volumes:
fix-brvm-config:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
default-token-1lggo:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-1lggo
QoS Tier:    BestEffort
Events:
FirstSeen    LastSeen    Count    From            SubobjectPath            Type        Reason        Message
---------    --------    -----    ----            -------------            --------    ------        -------

7m        7m        1    {default-scheduler }                    Normal        Scheduled    Successfully assigned platinum-fix to 172.17.4.201
7m        7m        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Pulling        pulling image "javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT"
7m        7m        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Pulled        Successfully pulled image "javaqa:5000/opessoftware/platinum-fix-brvm:1.1.0-SNAPSHOT"
7m        7m        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Created        Created container with docker id 61df8621e2af
7m        7m        1    {kubelet 172.17.4.201}    spec.containers{fix-brvm}    Normal        Started        Started container with docker id 61df8621e2af

---

derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/platinum/platinum-fix-group/platinum-fix/src/main/kubernetes$ kubectl exec platinum-fix -c fix-brvm -- ls /etc/opes/platinum/fix/brvm/
brvm-fix44.xml
keystore.jks
settings.txt

Thank you for helping me find the bug!

Reply all

Reply to author

Forward