apiserver shuts itself down without apparent reason
2,099 views
Skip to first unread message
Dylan Humphreys
Mar 8, 2018, 11:15:54 AM3/8/18
to Kubernetes developer/contributor discussion
Hi Everyone, hoping someone can help me. Ive been throwing myself at this problem for a couple of days:
I'm following this guide here:
This is my manifest:
Does anyone have any suggestions of what might be causing it, or how I can find out what IS causing it?
I'm following this guide here:
setting it up on our private cloud.
When I run the apiserver, it runs for a while, but then seems to shut itself down.
This appears in the logs:
I0308 16:09:04.822858 1 round_trippers.go:417] curl -k -v -XGET -H "Authorization: Bearer ea6aa36d-d1d6-40ce-b59e-369949e04add" -H "User-Agent: hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2" -H "Accept: application/vnd.kuberne
tes.protobuf, */*" https://127.0.0.1:6443/api/v1/namespaces/default/endpoints/kubernetes
I0308 16:09:04.823186 1 handler.go:160] kube-aggregator: GET "/api/v1/namespaces/default/endpoints/kubernetes" satisfied by nonGoRestful
I0308 16:09:04.823318 1 pathrecorder.go:247] kube-aggregator: "/api/v1/namespaces/default/endpoints/kubernetes" satisfied by prefix /api/
I0308 16:09:04.823434 1 handler.go:150] kube-apiserver: GET "/api/v1/namespaces/default/endpoints/kubernetes" satisfied by gorestful with webservice /api/v1
I0308 16:09:04.825764 1 wrap.go:42] GET /api/v1/namespaces/default/endpoints/kubernetes: (2.612162ms) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:04.826203 1 round_trippers.go:436] GET https://127.0.0.1:6443/api/v1/namespaces/default/endpoints/kubernetes 200 OK in 3 milliseconds
I0308 16:09:04.826342 1 round_trippers.go:442] Response Headers:
I0308 16:09:04.826474 1 round_trippers.go:445] Date: Thu, 08 Mar 2018 16:09:04 GMT
I0308 16:09:04.826604 1 round_trippers.go:445] Content-Type: application/vnd.kubernetes.protobuf
I0308 16:09:04.826729 1 round_trippers.go:445] Content-Length: 199
I0308 16:09:04.826891 1 request.go:871] Response Body:
00000000 6b 38 73 00 0a 0f 0a 02 76 31 12 09 45 6e 64 70 |k8s.....v1..Endp|
00000010 6f 69 6e 74 73 12 ab 01 0a 84 01 0a 0a 6b 75 62 |oints........kub|
00000020 65 72 6e 65 74 65 73 12 00 1a 07 64 65 66 61 75 |ernetes....defau|
00000030 6c 74 22 2f 2f 61 70 69 2f 76 31 2f 6e 61 6d 65 |lt"//api/v1/name|
00000040 73 70 61 63 65 73 2f 64 65 66 61 75 6c 74 2f 65 |spaces/default/e|
00000050 6e 64 70 6f 69 6e 74 73 2f 6b 75 62 65 72 6e 65 |ndpoints/kuberne|
00000060 74 65 73 2a 24 33 36 63 30 37 38 64 65 2d 32 32 |tes*$36c078de-22|
00000070 30 66 2d 31 31 65 38 2d 61 33 39 35 2d 63 61 65 |0f-11e8-a395-cae|
00000080 66 37 33 66 33 62 30 30 33 32 06 32 33 37 30 35 |f73f3b0032.23705|
00000090 32 38 00 42 08 08 d9 de ff d4 05 10 00 7a 00 12 |28.B.........z..|
000000a0 22 0a 0f 0a 0b 31 30 2e 31 36 2e 32 33 2e 33 31 |"....10.16.23.31|
000000b0 1a 00 1a 0f 0a 05 68 74 74 70 73 10 ab 32 1a 03 |......https..2..|
000000c0 54 43 50 1a 00 22 00 |TCP..".|
I0308 16:09:12.886690 1 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2" -H "Authorization: Bearer ea6aa36d-d1d6-40ce-
b59e-369949e04add" https://127.0.0.1:6443/api/v1/namespaces/default/endpoints/kubernetes
I0308 16:09:12.886921 1 autoregister_controller.go:160] Shutting down autoregister controller
I0308 16:09:12.886969 1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0308 16:09:12.886988 1 naming_controller.go:285] Shutting down NamingConditionController
I0308 16:09:12.887002 1 customresource_discovery_controller.go:163] Shutting down DiscoveryController
I0308 16:09:12.887016 1 crdregistration_controller.go:139] Shutting down crd-autoregister controller
I0308 16:09:12.887031 1 apiservice_controller.go:124] Shutting down APIServiceRegistrationController
I0308 16:09:12.887047 1 available_controller.go:274] Shutting down AvailableConditionController
I0308 16:09:12.887331 1 controller.go:90] Shutting down OpenAPI AggregationController
I0308 16:09:12.887740 1 wrap.go:42] GET /api/v1/namespaces/default/endpoints/kubernetes: (149.597µs) 500
...
logging error output: "Apisever is shutting down.\n"
[[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.887845 1 wrap.go:42] GET /api/v1/resourcequotas?resourceVersion=275790&timeoutSeconds=492&watch=true: (28.204535352s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.887958 1 wrap.go:42] GET /apis/apiregistration.k8s.io/v1beta1/apiservices?resourceVersion=275790&timeoutSeconds=553&watch=true: (28.219690667s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866
]
I0308 16:09:12.888008 1 wrap.go:42] GET /api/v1/serviceaccounts?resourceVersion=275790&timeoutSeconds=374&watch=true: (28.213622251s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.888064 1 wrap.go:42] GET /api/v1/services?resourceVersion=275790&timeoutSeconds=477&watch=true: (28.202558628s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.888171 1 wrap.go:42] GET /api/v1/endpoints?resourceVersion=275790&timeoutSeconds=309&watch=true: (28.18743168s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.888237 1 wrap.go:42] GET /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?resourceVersion=275790&timeoutSeconds=374&watch=true: (28.204758934s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127
.0.0.1:39866]
I0308 16:09:12.888303 1 serve.go:129] Stopped listening on [::]:6443
I0308 16:09:12.888374 1 serve.go:129] Stopped listening on 10.16.23.31:80
I0308 16:09:12.888401 1 round_trippers.go:436] GET https://127.0.0.1:6443/api/v1/namespaces/default/endpoints/kubernetes 500 Internal Server Error in 1 milliseconds
I0308 16:09:12.888415 1 round_trippers.go:442] Response Headers:
I0308 16:09:12.888424 1 round_trippers.go:445] Date: Thu, 08 Mar 2018 16:09:12 GMT
I0308 16:09:12.888432 1 round_trippers.go:445] Content-Type: text/plain; charset=utf-8
I0308 16:09:12.888468 1 round_trippers.go:445] X-Content-Type-Options: nosniff
I0308 16:09:12.888479 1 round_trippers.go:445] Content-Length: 27
I0308 16:09:12.888661 1 wrap.go:42] GET /api/v1/namespaces?resourceVersion=275790&timeoutSeconds=544&watch=true: (28.201684777s) 200 [[hyperkube/v1.9.2 (linux/amd64) kubernetes/5fa2db2] 127.0.0.1:39866]
I0308 16:09:12.888674 1 request.go:873] Response Body: Apisever is shutting down.
This is my manifest:
---
kind: Pod
apiVersion: v1
metadata:
name: kube-apiserver
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: docker.comodoca.net/k8s/hyperkube:v1.9.2
command: ["/bin/sh"]
args: ["-c", "/hyperkube apiserver --insecure-bind-address=10.16.23.31 --insecure-port 80 --etcd-servers=http://bddcbpkbn1:4001,http://bddcbpkbn2:4001,http://bddcbpkbn3:4001,http://bddcbpkbn4:4001 --admission-control=NamespaceLifecyc
le,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --service-cluster-ip-range=172.16.0.0/12 --token-auth-file=/var/lib/k8s/known_tokens.csv --v=9 --allow-privileged=False" ]
ports:
- name: "http"
hostPort: 80
containerPort: 80
volumeMounts:
- name: tokenfiles
mountPath: /var/lib/k8s/
readOnly: false
- name: k8sca
mountPath: /opt
readOnly: true
livenessProbe:
httpGet:
scheme: HTTP
host: 127.0.0.1
port: 2379
path: ""
initialDelaySeconds: 15
timeoutSeconds: 15
volumes:
- name: tokenfiles
hostPath:
path: /var/lib/kubernetes/kube-apiserver/
- name: k8sca
hostPath:
path: /opt
Does anyone have any suggestions of what might be causing it, or how I can find out what IS causing it?
Thanks in advance
Daniel Smith
Mar 8, 2018, 11:22:59 AM3/8/18
to Dylan Humphreys, Kubernetes developer/contributor discussion
Since you are running it via kubelet, I would look in the kubelet logs as a first step. System OOM or health check fail is my guess.
Actually now that I mention that, I note your livenessProbe clause seems to be copy-pasta from an etcd manifest and is checking an etcd port instead of an apiserver port. That's my guess.
--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/df31e65d-e817-4216-81e8-5bd717ac6ae1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Dylan Humphreys
Mar 8, 2018, 4:12:08 PM3/8/18
to Kubernetes developer/contributor discussion
#facepalm. Yep, spot on. Thank you Daniel
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages