How to get router deployed correctly?

rishi.in...@gmail.com

unread,

Oct 31, 2016, 9:21:34 AM10/31/16

to Deis Users List

I am still having difficulties getting router component deployed in Kubernetes. It always CrashLoops with following error:

/======================================/

# kubectl logs --namespace=deis deis-router-1833288991-7vet3

2016/10/31 13:06:04 INFO: Starting nginx...

2016/10/31 13:06:04 INFO: nginx started.

2016/10/31 13:06:04 Failed to create config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

# kubectl describe pod --namespace=deis deis-router-1833288991-7vet3

Name: deis-router-1833288991-7vet3

Namespace: deis

Node: 127.0.0.1/127.0.0.1

Start Time: Mon, 31 Oct 2016 09:04:39 -0400

Labels: app=deis-router

pod-template-hash=1833288991

Status: Running

IP: 172.17.0.18

Controllers: ReplicaSet/deis-router-1833288991

Containers:

deis-router:

Container ID: docker://15f852985a21567de254205c28f17e669e0a7b636d23df67e1f83f12b007767b

Image: quay.io/deis/router:v2.6.3

Image ID: docker://sha256:39dab29fa36470fcbb85914bf143eff1a6a2635cef2cd78223abeb3e4b95c07c

Ports: 8080/TCP, 6443/TCP, 2222/TCP, 9090/TCP

State: Terminated

Reason: Error

Exit Code: 1

Started: Mon, 31 Oct 2016 09:04:59 -0400

Finished: Mon, 31 Oct 2016 09:05:00 -0400

Last State: Terminated

Reason: Error

Exit Code: 1

Started: Mon, 31 Oct 2016 09:04:41 -0400

Finished: Mon, 31 Oct 2016 09:04:41 -0400

Ready: False

Restart Count: 2

Liveness: http-get http://:9090/healthz delay=10s timeout=1s period=10s #success=1 #failure=3

Readiness: http-get http://:9090/healthz delay=1s timeout=1s period=10s #success=1 #failure=3

Environment Variables:

POD_NAMESPACE: deis (v1:metadata.namespace)

Conditions:

Type Status

Initialized True

Ready False

PodScheduled True

No volumes.

QoS Tier: BestEffort

Events:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

--------- -------- ----- ---- ------------- -------- ------ -------

30s 30s 1 {default-scheduler } Normal Scheduled Successfully assigned deis-router-1833288991-7vet3 to 127.0.0.1

29s 29s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Created Created container with docker id 10a7850903a6

29s 29s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Started Started container with docker id 10a7850903a6

28s 28s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Created Created container with docker id 4de48b9570ce

28s 28s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Started Started container with docker id 4de48b9570ce

27s 23s 3 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "deis-router" with CrashLoopBackOff: "Back-off 10s restarting failed container=deis-router pod=deis-router-1833288991-7vet3_deis(952b0e9e-9f6a-11e6-ba4c-0200000000d9)"

29s 10s 3 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Pulled Container image "quay.io/deis/router:v2.6.3" already present on machine

10s 10s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Created Created container with docker id 15f852985a21

10s 10s 1 {kubelet 127.0.0.1} spec.containers{deis-router} Normal Started Started container with docker id 15f852985a21

27s 9s 4 {kubelet 127.0.0.1} spec.containers{deis-router} Warning BackOff Back-off restarting failed docker container

9s 9s 1 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "deis-router" with CrashLoopBackOff: "Back-off 20s restarting failed container=deis-router pod=deis-router-1833288991-7vet3_deis(952b0e9e-9f6a-11e6-ba4c-0200000000d9)"

/======================================/

Router Manifest is as follows:

/================================/

kind: Deployment

metadata:

name: deis-router

namespace: deis

labels:

heritage: deis

annotations:

helm-keep: "true"

component.deis.io/version: v2.6.3

spec:

replicas: 1

strategy:

rollingUpdate:

maxSurge: 1

maxUnavailable: 0

type: RollingUpdate

selector:

matchLabels:

app: deis-router

template:

metadata:

labels:

app: deis-router

spec:

serviceAccount: deis-router

containers:

- name: deis-router

image: quay.io/deis/router:v2.6.3

imagePullPolicy: IfNotPresent

env:

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

ports:

- containerPort: 8080

hostPort: 80

- containerPort: 6443

hostPort: 443

- containerPort: 2222

hostPort: 2222

- containerPort: 9090

hostPort: 9090

livenessProbe:

httpGet:

path: /healthz

port: 9090

initialDelaySeconds: 10

timeoutSeconds: 1

readinessProbe:

httpGet:

path: /healthz

port: 9090

initialDelaySeconds: 1

timeoutSeconds: 1

/================================/

Wonder if deployment manifest is missing something. Please let me know how do I get around debugging it. From logs it seems nginx has started correctly.

Thanks.

Matthew Fisher

unread,

Oct 31, 2016, 12:03:00 PM10/31/16

to Deis user list

Once again forgot to hit reply-all. CC'ing this back to the group.

On Mon, Oct 31, 2016 at 9:02 AM, Matthew Fisher <mfi...@deis.com> wrote:

The missing file comes from Kubernetes. The router is missing its service account token, which is requested in the router manifest and provided by Kubernetes. It allows the router to be able to be authenticated with the Kubernetes API.

Some relevant links:

https://github.com/deis/charts/blob/master/workflow-v2.8.0/manifests/deis-router-service-account.yaml
https://github.com/deis/charts/blob/0f19b593e2309c42d4ba98fdb1d94e338fd16d4f/workflow-v2.8.0/tpl/deis-router-deployment.yaml#L30
http://kubernetes.io/docs/user-guide/service-accounts/

Matt

--
You received this message because you are subscribed to the Google Groups "Deis Users List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deis-users+unsubscribe@googlegroups.com.
To post to this group, send email to deis-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/deis-users/a59b0208-9741-42c4-8985-085015a82194%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

rishi.in...@gmail.com

unread,

Oct 31, 2016, 2:55:07 PM10/31/16

to Deis Users List

Thanks Matt.

Service account already existed - I even created (and imported) a secret - "router-secret.yaml" file with following content hoping it will works but that did not help either.

apiVersion: v1

kind: Secret

metadata:

name: deis-router-secret

namespace: deis

labels:

heritage: deis

annotations:

kubernetes.io/service-account.name: deis-router

type: kubernetes.io/service-account-token

root@lozlnxk:/localbox/rishi/deis# kubectl get secrets --namespace=deis NAME TYPE DATA AGE

builder-key-auth Opaque 1 6d

builder-ssh-private-keys Opaque 3 6d

database-creds Opaque 2 6d

deis-router-dhparam Opaque 1 3d

deis-router-secret kubernetes.io/service-account-token 0 10m

django-secret-key Opaque 1 6d

logger-redis-creds Opaque 1 11d

objectstorage-keyfile Opaque 2 6d

Should deis-router-dhparam be linked to deis-router service account?

# kubectl get serviceaccounts --namespace=deis NAME SECRETS AGE

default 0 11d

deis-builder 0 6d

deis-controller 0 6d

deis-database 0 6d

deis-logger 0 6d

deis-logger-fluentd 0 6d

deis-minio 0 6d

deis-monitor-telegraf 0 6d

deis-nsqd 0 6d

deis-registry 0 6d

deis-router 0 3d

deis-workflow-manager 0 6d

To unsubscribe from this group and stop receiving emails from it, send an email to deis-users+...@googlegroups.com.

Matthew Fisher

unread,

Oct 31, 2016, 4:39:43 PM10/31/16

to rishi.in...@gmail.com, Deis Users List

So to confirm, `kubectl --namespace=deis get serviceaccounts` shows a deis-router service account? If so then I'd start looking into why the service account token isn't being mounted into the container. That is the responsibility of Kubernetes so it's not a Deis issue per se.

To unsubscribe from this group and stop receiving emails from it, send an email to deis-users+unsubscribe@googlegroups.com.

To post to this group, send email to deis-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/deis-users/559951ab-75f3-43b9-a2c7-1135f31beb52%40googlegroups.com.

Reply all

Reply to author

Forward