Error syncing pod, skipping: failed to "StartContainer" for "exposecontroller"

[kurr...@gmail.com]

Hello again

For some reason the exposecontroller doesnt start. Does anyone know how to solve this issue?

$ oc get pods

NAME                              READY     STATUS    RESTARTS   AGE

configmapcontroller-1-397qh       1/1       Running   3          5d

docker-registry-1-hlxdm           1/1       Running   3          8d

exposecontroller-2-9hbc9          0/1       Error     0          27s

fabric8-2-tbldf                   1/1       Running   3          5d

fabric8-docker-registry-1-an81b   1/1       Running   3          5d

fabric8-forge-1-ivmfl             1/1       Running   3          5d

gogs-2-fs5f3                      1/1       Running   3          5d

jenkins-1-1d0ts                   1/1       Running   4          5d

nexus-1-8s9r2                     1/1       Running   3          5d

router-1-t86jr                    1/1       Running   3          8d

staffservice-1-deploy             0/1       Error     0          20h

anton@anton-K93SV:~/git/hughestech/shiftwork/staffservice$ oc describe pod exposecontroller-2-9hbc9

Name: exposecontroller-2-9hbc9

Namespace: default

Security Policy: restricted

Node: 176.9.36.15/176.9.36.15

Start Time: Mon, 23 Jan 2017 19:46:06 +0100

Labels: deployment=exposecontroller-2

deploymentconfig=exposecontroller

group=io.fabric8.devops.apps

project=exposecontroller

provider=fabric8

version=2.2.309

Status: Running

IP: 172.17.0.10

Controllers: ReplicationController/exposecontroller-2

Containers:

  exposecontroller:

    Container ID: docker://1270223139f7c42b27a7972dcd445b472f56d1fe3343e86e4ce85029c34fa130

    Image: fabric8/exposecontroller:2.3.2

    Image ID: docker://sha256:f2ce398a7fd5338176e0921172ccdb485060f7e6078c7a372304606e5204328f

    Port:

    State: Waiting

      Reason: CrashLoopBackOff

    Last State: Terminated

      Reason: Error

      Exit Code: 255

      Started: Mon, 23 Jan 2017 19:46:35 +0100

      Finished: Mon, 23 Jan 2017 19:46:35 +0100

    Ready: False

    Restart Count: 1

    Liveness: exec [cat /tmp/restarteveryday] delay=86164s timeout=1s period=10s #success=1 #failure=3

    Volume Mounts:

      /etc/exposecontroller from config-volume (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from exposecontroller-token-2qytg (ro)

    Environment Variables:

      KUBERNETES_NAMESPACE: default (v1:metadata.namespace)

      FABRIC8_EXPOSECONTROLLER_CONFIGMAP: config.yml=domain: 

exposer: Route

apiserver: openshift.medomain.co:8443

Conditions:

  Type Status

  Initialized True 

  Ready False 

  PodScheduled True 

Volumes:

  config-volume:

    Type: ConfigMap (a volume populated by a ConfigMap)

    Name: exposecontroller

  exposecontroller-token-2qytg:

    Type: Secret (a volume populated by a Secret)

    SecretName: exposecontroller-token-2qytg

QoS Tier: BestEffort

Events:

  FirstSeen LastSeen Count From SubobjectPath Type Reason Message

  --------- -------- ----- ---- ------------- -------- ------ -------

  46s 46s 1 {default-scheduler } Normal Scheduled Successfully assigned exposecontroller-2-9hbc9 to 176.9.36.15

  40s 40s 1 {kubelet 176.9.36.15} spec.containers{exposecontroller} Normal Created Created container with docker id 292eaa4a649f

  40s 40s 1 {kubelet 176.9.36.15} spec.containers{exposecontroller} Normal Started Started container with docker id 292eaa4a649f

  43s 20s 2 {kubelet 176.9.36.15} spec.containers{exposecontroller} Normal Pulled Container image "fabric8/exposecontroller:2.3.2" already present on machine

  17s 17s 1 {kubelet 176.9.36.15} spec.containers{exposecontroller} Normal Created Created container with docker id 1270223139f7

  17s 17s 1 {kubelet 176.9.36.15} spec.containers{exposecontroller} Normal Started Started container with docker id 1270223139f7

  16s 11s 3 {kubelet 176.9.36.15} spec.containers{exposecontroller} Warning BackOff Back-off restarting failed docker container

  16s 11s 3 {kubelet 176.9.36.15} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "exposecontroller" with CrashLoopBackOff: "Back-off 10s restarting failed container=exposecontroller pod=exposecontroller-2-9hbc9_default(32c9adf6-e19c-11e6-91a8-406186becd9d)"

[kurr...@gmail.com]

Futhermore

$ kubectl logs exposecontroller-2-9hbc9
I0123 18:57:38.717065       1 exposecontroller.go:47] Using build: '2.3.2'
F0123 18:57:38.784987       1 exposecontroller.go:66] failed to create new strategy: failed to create ingress expose strategy: failed to get a domain: no known automatic ways to get an external ip to use with nip.  Please configure exposecontroller configmap manually see https://github.com/fabric8io/exposecontroller#configuration

On https://github.com/fabric8io/exposecontroller#configuration it says I am to:

cat <<EOF | kubectl create -f -
apiVersion: "v1"
data:
  config.yml: |-
    exposer: "Ingress"
    domain: "replace.me.io"
kind: "ConfigMap"
metadata:
  name: "exposecontroller"
EOF

$ cat <<EOF | kubectl create -f -
> apiVersion: "v1"
> data:
>   config.yml: |-
>     exposer: "Ingress"
>     domain: "mydomain.co"
> kind: "ConfigMap"
> metadata:
>   name: "exposecontroller"
> EOF
Error from server (AlreadyExists): error when creating "STDIN": configmaps "exposecontroller" already exists

Unfortuately google gives 0 results for this error.

This is a worry. 

[kurr...@gmail.com]

I found I can edit this file, but it currently looks like:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  config.yml: "domain: \nexposer: Route\napiserver: mydomain.co:8443\n"
kind: ConfigMap
metadata:
  creationTimestamp: 2017-01-17T22:01:39Z
  labels:
    provider: fabric8
  name: exposecontroller
  namespace: default
  resourceVersion: "78314"
  selfLink: /api/v1/namespaces/default/configmaps/exposecontroller
  uid: 85da7b77-dd00-11e6-9da0-406186becd9d

I dont see what is wrong with it.

How does one debug this?

Can any readers of this forum see anything wrong with the file?

[James Rawlings]

I can try and have a guess at what the problem is but we could probably do with a bit more information.  Having said that I've added a few things to try...

Can you tell us a bit more about your setup please?  

Are you running on a custom kubernetes cluster or using a public cloud maybe?  

Do you have a node in your cluster that is externally accessible via a public ip?  So that the exposecontroller can create the ingress rules it needs to know where the nginx ingress controller is, my guess is it's not running.

Were there any errors reported when you ran `gofabric8 deploy`?

gofabric8 deploy should have run an nginx ingress controller in the fabric8-system namespace, this is what's used to automatically generate the exposecontroller domain config.  Do you see the nginx pod running?

kubectl get pods -n fabric8-system

if not are there any events (replace nginx-ingress-xxxxx with the name of the pod in your cluster)

kubectl describe pod nginx-ingress-xxxxx  -n fabric8-system

It may be that gofabric8 deploy had an issue finding a scheduleble node to label with and if there's no nodes with that label the the nginx pod cant find a node to run on, therefore preventing the expos controller domain config from being added.

are there any results if you run...

kubectl get nodes -l fabric8.io/externalIP=true

if not can you try and label a node of your choice that has an external / public ip and label it using..

kubectl get nodes

kubectl label node [chose a node name from above] fabric8.io/externalIP=true` 

And see if the nginx pod runs now, if it does bounce the exposecontroller pod so that it can figure out the default config.

James.

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Rawlings]

I've just been looking at some other correspondence from you and I'm now thinking you're using fabric8 on openshift, is that right?

If so we currently don't have a way to automatically work out the node that the openshift router is on, to then figure out the domain for exposecontroller.  So when deploying on openshift you need to supply the domain you have for accessing applications in your cluster.

For example:

gofabric8 deploy -d example.fabric8.io

http://fabric8.io/guide/getStarted/openshift.html#run-the-gofabric8-installer

[kurr...@gmail.com]

Hi again James - you were extremely helpful a few weeks back on the Fabric8 irc chat. 

 

I've just been looking at some other correspondence from you and I'm now thinking you're using fabric8 on openshift, is that right?

Yes, correct. 

I think I did give that information during setup.

And when I look at the config.yml for the exposecontroller it appears correct. Although I do not know what I am looking for. But it does have the correct domain name.

[kurr...@gmail.com]

I have installed fabric8 in the default project.

$ kubectl get pods -n default

NAME                              READY     STATUS             RESTARTS   AGE
configmapcontroller-1-397qh       1/1       Running            3          5d
docker-registry-1-hlxdm           1/1       Running            3          8d

exposecontroller-2-9hbc9          0/1       CrashLoopBackOff   25         1h

[kurr...@gmail.com]

 Were there any errors reported when you ran `gofabric8 deploy`?

I dont think so.  

gofabric8 deploy should have run an nginx ingress controller in the fabric8-system namespace, this is what's used to automatically generate the exposecontroller domain config.  Do you see the nginx pod running?

kubectl get pods -n fabric8-system

if not are there any events (replace nginx-ingress-xxxxx with the name of the pod in your cluster)

kubectl describe pod nginx-ingress-xxxxx  -n fabric8-system

I dont appear to have a fabric8-system project/namespace.

And I dont think there is an nginx pod running.  

[James Rawlings]

Ok great, so if the exposecontroller config map has a value for the `domain:` attribute then the pod shouldn't have any problems starting. Can you try and manually delete the pod so that it restarts with the correct config?

kubectl delete pod exposecontroller-2-9hbc9 --now

If it still fails to start then check the exposecontroller pod logs for errors again.

Fingers crossed!

BTW lets forget about the other email about the fabric8-system, you wont have that currently when deploying on openshift.

[James Rawlings]

I've got to call it a night but if you're still having an error hopefully we can catch up on the fabric8 IRC channel tomorrow.

[Anton]

Thanks James!

The exposecontroller pod is now running.

Thank you very much for your help.

[James Rawlings]

That's great new :)