Knative operator pod failing

[Carlos Meza]

Hello all ! 

I am looking to test Knative operator on k8s.

Operator webhook is running nicely but Knative operator pod is on a loop restarting .

Logs:

2022/05/13 16:57:18 Registering 3 clients
2022/05/13 16:57:18 Registering 3 informer factories
2022/05/13 16:57:18 Registering 3 informers
2022/05/13 16:57:18 Registering 2 controllers
{"severity":"INFO","timestamp":"2022-05-13T16:57:18.145132924Z","caller":"logging/config.go:117","message":"Successfully created the logger."}
{"severity":"INFO","timestamp":"2022-05-13T16:57:18.145186541Z","caller":"logging/config.go:118","message":"Logging level set to: info"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:18.145226611Z","caller":"logging/config.go:80","message":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:18.145260251Z","logger":"knative-operator","caller":"profiling/server.go:64","message":"Profiling enabled: false","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:18.150476248Z","logger":"knative-operator","caller":"leaderelection/context.go:47","message":"Running with Standard leader election","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
I0513 16:57:19.251198       1 request.go:665] Waited for 1.096897829s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
{"severity":"INFO","timestamp":"2022-05-13T16:57:20.903449381Z","logger":"knative-operator.manifestival","caller":"manife...@v0.7.1/manifestival.go:71","message":"Parsing manifest","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:20.903634567Z","logger":"knative-operator","caller":"knativeserving/controller.go:67","message":"Setting up event handlers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.648897455Z","logger":"knative-operator.manifestival","caller":"manife...@v0.7.1/manifestival.go:71","message":"Parsing manifest","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.649084674Z","logger":"knative-operator","caller":"knativeeventing/controller.go:67","message":"Setting up event handlers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.654367757Z","logger":"knative-operator","caller":"sharedmain/main.go:253","message":"Starting configuration manager...","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.656519116Z","logger":"knative-operator","caller":"metrics/metrics_worker.go:76","message":"Flushing the existing exporter before setting up the new exporter.","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.656646349Z","logger":"knative-operator","caller":"metrics/prometheus_exporter.go:51","message":"Created Prometheus exporter with config: &{knative.dev/operator knative_operator prometheus 5000000000 <nil>  false 9090 0.0.0.0}. Start the server for Prometheus exporter.","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.65666866Z","logger":"knative-operator","caller":"metrics/metrics_worker.go:91","message":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/operator knative_operator prometheus 5000000000 <nil>  false 9090 0.0.0.0}","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"level":"info","ts":1652461043.7554286,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856352484Z","logger":"knative-operator","caller":"sharedmain/main.go:281","message":"Starting controllers...","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856469427Z","logger":"knative-operator","caller":"leaderelection/context.go:149","message":"knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01 will run in leader-elected mode with id \"knative-operator-6c7f69f87c-z49pg_7e7161dc-8050-4952-8001-9dfad9809521\"","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856515982Z","logger":"knative-operator","caller":"controller/controller.go:486","message":"Starting controller and workers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeeventing.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeEventing"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856525299Z","logger":"knative-operator","caller":"controller/controller.go:496","message":"Started workers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeeventing.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeEventing"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856560107Z","logger":"knative-operator","caller":"leaderelection/context.go:149","message":"knative-operator.knative.dev.operator.pkg.reconciler.knativeserving.reconciler.00-of-01 will run in leader-elected mode with id \"knative-operator-6c7f69f87c-z49pg_9654d848-30d6-406b-9afa-7216174d3c90\"","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856574286Z","logger":"knative-operator","caller":"controller/controller.go:486","message":"Starting controller and workers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:23.856579373Z","logger":"knative-operator","caller":"controller/controller.go:496","message":"Started workers","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing"}
I0513 16:57:23.856678       1 leaderelection.go:248] attempting to acquire leader lease default/knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01...
I0513 16:57:23.856916       1 leaderelection.go:248] attempting to acquire leader lease default/knative-operator.knative.dev.operator.pkg.reconciler.knativeserving.reconciler.00-of-01...
I0513 16:57:14.683047 1 leaderelection.go:258] successfully acquired lease default/knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01
{"severity":"INFO","timestamp":"2022-05-13T16:57:14.683252145Z","logger":"knative-operator","caller":"leaderelection/context.go:158","message":"\"knative-operator-6c7f69f87c-z49pg_2e9eef9a-65b6-4293-9814-38b791ecfac4\" has started leading \"knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01\"","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
I0513 16:57:16.997100 1 leaderelection.go:258] successfully acquired lease default/knative-operator.knative.dev.operator.pkg.reconciler.knativeserving.reconciler.00-of-01
{"severity":"INFO","timestamp":"2022-05-13T16:57:16.997247503Z","logger":"knative-operator","caller":"leaderelection/context.go:158","message":"\"knative-operator-6c7f69f87c-z49pg_c1ed7dcc-ad28-4bad-a8f1-2f8ca89f5140\" has started leading \"knative-operator.knative.dev.operator.pkg.reconciler.knativeserving.reconciler.00-of-01\"","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg"}
{"severity":"INFO","timestamp":"2022-05-13T16:57:17.000710914Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:81","message":"Deleting cluster-scoped resources","knative.dev/pod":"knative-operator-6c7f69f87c-z49pg","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"4f4cbaab-749f-4338-9c3f-bccd56edbd73","knative.dev/key":"monitor/knative-serving"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11dfcd8]
goroutine 431 [running]:
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetKind(...)
k8s.io/apimac...@v0.23.5/pkg/apis/meta/v1/unstructured/unstructured.go:226
github.com/manifestival/manifestival.okToDelete(0x0)
github.com/manifestival/manife...@v0.7.1/manifestival.go:234 +0x38
github.com/manifestival/manifestival.Manifest.delete({{0xc000fd4000, 0x34, 0x40}, {0x1dad7d0, 0xc00004c1b0}, {{0x1dcfd60, 0xc0004756e0}, 0x0}}, 0x1dad7d0, {0x0, ...})
github.com/manifestival/manife...@v0.7.1/manifestival.go:179 +0xae
github.com/manifestival/manifestival.Manifest.Delete({{0xc000fd4000, 0x34, 0x40}, {0x1dad7d0, 0xc00004c1b0}, {{0x1dcfd60, 0xc0004756e0}, 0x0}}, {0x0, 0x0, ...})
github.com/manifestival/manife...@v0.7.1/manifestival.go:116 +0x1e5
knative.dev/operator/pkg/reconciler/common.Uninstall(0xc00093fbc0)
knative.dev/operator/pkg/reconciler/common/install.go:77 +0x278
knative.dev/operator/pkg/reconciler/knativeserving.(*Reconciler).FinalizeKind(0xc00045e000, {0x1dad6b8, 0xc0009565a0}, 0xc000375260)
knative.dev/operator/pkg/reconciler/knativeserving/knativeserving.go:87 +0x365
knative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving.(*reconcilerImpl).Reconcile(0xc0000bc0a0, {0x1dad6b8, 0xc000956570}, {0xc000054ff0, 0x17})
knative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving/reconciler.go:240 +0x418
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc0003c29c0)
knative.dev/p...@v0.0.0-20220505013700-a8b7d99374a3/controller/controller.go:542 +0x5a5
knative.dev/pkg/controller.(*Impl).RunContext.func3()
knative.dev/p...@v0.0.0-20220505013700-a8b7d99374a3/controller/controller.go:491 +0x68
created by knative.dev/pkg/controller.(*Impl).RunContext
knative.dev/p...@v0.0.0-20220505013700-a8b7d99374a3/controller/controller.go:489 +0x31c 

Any idea ?

Thank you all !

Omar

[Reto Lehmann]

Hi Omar,

Could you please provide your `KnativeServing` yaml? From the stacktrace (k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetKind(...)) it seems like the yaml could be malformed.

Best regards

Reto