IngressNotConfigured: Operation cannot be fulfilled on ingresses.networking.internal.knative.dev

1,838 views
Skip to first unread message

Assaf Sauer

unread,
Sep 6, 2019, 9:24:31 AM9/6/19
to Knative Users
Hello!

i upgraded my Knative to the latest version and since then im getting route-controller operation error when trying to create new service.
Did anyone experienced similar issue? 

Best...

root@ubuntu:/home/viewadmin/knative# kubectl get route
NAME            URL                                        READY     REASON
helloworld-go   http://helloworld-go.default.example.com   Unknown   IngressNotConfigured
web-php-1       http://web-php-1.default.example.com       Unknown   IngressNotConfigured


root@ubuntu:/home/viewadmin/knative# kubectl describe route web-php-1
Name:         web-php-1
Namespace:    default
Kind:         Route
Metadata:
  Creation Timestamp:  2019-09-06T13:11:51Z
  Finalizers:
  Generation:  1
  Owner References:
    API Version:           serving.knative.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Service
    Name:                  web-php-1
    UID:                   e3f36568-d0a7-11e9-97dc-0050568a6fd7
  Resource Version:        185533
  UID:                     e3fc193b-d0a7-11e9-97dc-0050568a6fd7
Spec:
  Traffic:
    Configuration Name:  web-php-1
    Latest Revision:     true
    Percent:             100
Status:
  Address:
  Conditions:
    Last Transition Time:  2019-09-06T13:12:06Z
    Status:                True
    Type:                  AllTrafficAssigned
    Last Transition Time:  2019-09-06T13:12:06Z
    Message:               Ingress has not yet been reconciled.
    Reason:                IngressNotConfigured
    Status:                Unknown
    Type:                  IngressReady
    Last Transition Time:  2019-09-06T13:12:06Z
    Message:               Ingress has not yet been reconciled.
    Reason:                IngressNotConfigured
    Status:                Unknown
    Type:                  Ready
  Observed Generation:     1
  Traffic:
    Latest Revision:  true
    Percent:          100
    Revision Name:    web-php-1-8f8wt
  URL:                http://web-php-1.default.example.com
Events:
  Type     Reason         Age   From              Message
  ----     ------         ----  ----              -------
  Normal   Created        4m    route-controller  Created placeholder service "web-php-1"
  Normal   Created        4m    route-controller  Created Ingress "web-php-1"
  Warning  InternalError  4m    route-controller  Operation cannot be fulfilled on ingresses.networking.internal.knative.dev "web-php-1": the object has been modified; please apply your changes to the latest version and try again
root@ubuntu:/home/viewadmin/knative# 




root@ubuntu:/home/viewadmin/knative# kubectl get pods --namespace istio-system
NAME                                       READY     STATUS      RESTARTS   AGE
custom-ingressgateway-6787df7888-rdnf5     1/1       Running     0          158m
istio-citadel-7d64db8bcf-ng8qf             1/1       Running     0          8h
istio-cleanup-secrets-5g2fm                0/1       Completed   0          8h
istio-egressgateway-6ddf4c8bd6-8jvzn       1/1       Running     0          8h
istio-egressgateway-6ddf4c8bd6-dwgtt       1/1       Running     0          8h
istio-egressgateway-6ddf4c8bd6-ldvmr       1/1       Running     0          8h
istio-egressgateway-6ddf4c8bd6-prbcl       1/1       Running     0          8h
istio-egressgateway-6ddf4c8bd6-tx4pp       1/1       Running     0          8h
istio-galley-7dd996474-dpnt8               1/1       Running     0          8h
istio-ingressgateway-64b798d69d-8c2vn      1/1       Running     0          70m
istio-pilot-6b596fb9c5-5q6kw               1/1       Running     0          70m
istio-policy-5c4d9ff96b-28hrq              2/2       Running     0          8h
istio-policy-5c4d9ff96b-frxl9              2/2       Running     0          8h
istio-policy-5c4d9ff96b-vxcs2              2/2       Running     0          8h
istio-policy-5c4d9ff96b-zkwl9              2/2       Running     0          7h27m
istio-policy-5c4d9ff96b-zpfnh              2/2       Running     0          8h
istio-sidecar-injector-6977b5cf5b-z7k28    1/1       Running     0          8h
istio-statsd-prom-bridge-b44b96d7b-bzv5z   1/1       Running     0          8h
istio-telemetry-7676df547f-5qwmq           2/2       Running     0          8h
istio-telemetry-7676df547f-92vwp           2/2       Running     0          8h
istio-telemetry-7676df547f-pbt84           2/2       Running     0          8h
istio-telemetry-7676df547f-pcg5s           2/2       Running     0          8h
istio-telemetry-7676df547f-v7q6l           2/2       Running     0          8h
zipkin-6878fd494-tj4zk                     1/1       Running     0          8h
root@ubuntu:/home/viewadmin/knative# 

William Tam

unread,
Sep 6, 2019, 11:46:31 AM9/6/19
to Assaf Sauer, Knative Users
Hi Assaf,

Do you see any errors in controller and networking-istio pods?

Thanks.
William


--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/1c6c60a5-095d-44b0-9c43-dda3b8d710b1%40googlegroups.com.

Assaf Sauer

unread,
Sep 6, 2019, 12:31:23 PM9/6/19
to William Tam, Knative Users

Hi William,

all the pods are up and running , do u want me to grep for errors log for specific pod ? 

root@ubuntu:/home/viewadmin# kubectl get pods --namespace istio-system

NAME                                       READY     STATUS      RESTARTS   AGE
custom-ingressgateway-6787df7888-rdnf5     1/1       Running     0          5h36m
istio-citadel-7d64db8bcf-ng8qf             1/1       Running     0          11h
istio-cleanup-secrets-5g2fm                0/1       Completed   0          11h
istio-egressgateway-6ddf4c8bd6-8jvzn       1/1       Running     0          11h
istio-egressgateway-6ddf4c8bd6-dwgtt       1/1       Running     0          11h
istio-egressgateway-6ddf4c8bd6-ldvmr       1/1       Running     0          11h
istio-egressgateway-6ddf4c8bd6-prbcl       1/1       Running     0          11h
istio-egressgateway-6ddf4c8bd6-tx4pp       1/1       Running     0          11h
istio-galley-7dd996474-dpnt8               1/1       Running     0          11h
istio-ingressgateway-64b798d69d-8c2vn      1/1       Running     0          4h8m
istio-pilot-6b596fb9c5-5q6kw               1/1       Running     0          4h8m
istio-policy-5c4d9ff96b-28hrq              2/2       Running     0          11h
istio-policy-5c4d9ff96b-frxl9              2/2       Running     0          11h
istio-policy-5c4d9ff96b-vxcs2              2/2       Running     0          11h
istio-policy-5c4d9ff96b-zkwl9              2/2       Running     0          10h
istio-policy-5c4d9ff96b-zpfnh              2/2       Running     0          11h
istio-sidecar-injector-6977b5cf5b-z7k28    1/1       Running     0          11h
istio-statsd-prom-bridge-b44b96d7b-bzv5z   1/1       Running     0          11h
istio-telemetry-7676df547f-5qwmq           2/2       Running     0          11h
istio-telemetry-7676df547f-92vwp           2/2       Running     0          11h
istio-telemetry-7676df547f-pbt84           2/2       Running     0          11h
istio-telemetry-7676df547f-pcg5s           2/2       Running     0          11h
istio-telemetry-7676df547f-v7q6l           2/2       Running     0          11h
zipkin-6878fd494-tj4zk                     1/1       Running     0          11h

William Tam

unread,
Sep 6, 2019, 1:22:05 PM9/6/19
to Assaf Sauer, Knative Users
kubectl logs networking-xxxx -n knative-serving
kubectl logs controller-xxxxx -n knative-serving.

Thanks,
William

Assaf Sauer

unread,
Sep 6, 2019, 2:23:35 PM9/6/19
to William Tam, Knative Users
Hi William,

Ohh ok.  thanks!

well. pls find errors below:

root@ubuntu:/home/viewadmin# kubectl log networking-istio-5dcb6b97c4-pkmjt -n knative-serving | grep -i error
ler.go:282"}
{"level":"error","ts":"2019-09-06T18:20:24.443Z","logger":"istiocontroller","caller":"controller/controller.go:345","msg":"Reconcile error","error":"admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"info","ts":"2019-09-06T18:20:24.446Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"03cbdfd2-d0ad-11e9-97dc-0050568a6fd7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"189209\", FieldPath:\"\"}): type: 'Warning' reason: 'CreationFailed' Failed to create VirtualService default/helloworld-go-mesh: admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2019-09-06T18:20:24.446Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"03cbdfd2-d0ad-11e9-97dc-0050568a6fd7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"189209\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","knative.dev/controller":"ingress-controller"}




kubectl log controller-6d9c97d6b9-2hvnw -n knative-serving | grep -i error
{"level":"error","ts":"2019-09-06T13:48:31.977Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:345","msg":"Reconcile error","knative.dev/controller":"serverlessservice-controller","error":"Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"helloworld-go-v46j2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"error","ts":"2019-09-06T13:48:32.880Z","logger":"controller.route-controller","caller":"route/reconcile_resources.go:99","msg":"Failed to update Ingress","knative.dev/controller":"route-controller","knative.dev/traceid":"75053a45-2652-41aa-8641-802bb99f262d","knative.dev/key":"default/helloworld-go","error":"Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcileIngress\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/reconcile_resources.go:99\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcileIngressResources\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:297\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcile\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:259\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).Reconcile\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:113\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:330\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"error","ts":"2019-09-06T13:48:32.993Z","logger":"controller.route-controller","caller":"controller/controller.go:345","msg":"Reconcile error","knative.dev/controller":"route-controller","error":"Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"info","ts":"2019-09-06T13:48:32.994Z","logger":"controller.route-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"fd3f00e1-d0ac-11e9-97dc-0050568a6fd7\", APIVersion:\"serving.knative.dev/v1alpha1\", ResourceVersion:\"189198\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","knative.dev/controller":"route-controller"}

Nghia Tran

unread,
Sep 6, 2019, 2:29:27 PM9/6/19
to Knative Users
Do you happen to use Istio 1.0?  Since Istio 1.0 is end-of-life, we no longer support it and switching to use Istio 1.1 features.



On Friday, September 6, 2019 at 11:23:35 AM UTC-7, Assaf Sauer wrote:
Hi William,

Ohh ok.  thanks!

well. pls find errors below:

root@ubuntu:/home/viewadmin# kubectl log networking-istio-5dcb6b97c4-pkmjt -n knative-serving | grep -i error
ler.go:282"}
{"level":"error","ts":"2019-09-06T18:20:24.443Z","logger":"istiocontroller","caller":"controller/controller.go:345","msg":"Reconcile error","error":"admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"info","ts":"2019-09-06T18:20:24.446Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"03cbdfd2-d0ad-11e9-97dc-0050568a6fd7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"189209\", FieldPath:\"\"}): type: 'Warning' reason: 'CreationFailed' Failed to create VirtualService default/helloworld-go-mesh: admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2019-09-06T18:20:24.446Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"03cbdfd2-d0ad-11e9-97dc-0050568a6fd7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"189209\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' admission webhook \"pilot.validation.istio.io\" denied the request: error decoding configuration: YAML decoding error: gateways:\n- mesh\nhosts:\n- helloworld-go.default.svc.cluster.local\n- 234424b5e02e80948fab86db87d788ba.probe.invalid\nhttp:\n- match:\n  - authority:\n      regex: ^helloworld-go\\.default(\\.svc(\\.cluster\\.local)?)?(?::\\d{1,5})?$\n    gateways:\n    - mesh\n  retries:\n    attempts: 3\n    perTryTimeout: 10m0s\n  route:\n  - destination:\n      host: helloworld-go-v46j2.default.svc.cluster.local\n      port:\n        number: 80\n    headers:\n      request:\n        add:\n          Knative-Serving-Namespace: default\n          Knative-Serving-Revision: helloworld-go-v46j2\n    weight: 100\n  timeout: 10m0s\n  websocketUpgrade: true\n- fault:\n    abort:\n      httpStatus: 200\n      percent: 100\n  match:\n  - authority:\n      exact: 234424b5e02e80948fab86db87d788ba.probe.invalid\n  route:\n  - destination:\n      host: null.invalid\n      port:\n        number: 80\n    weight: 0\n unknown field \"headers\" in v1alpha3.DestinationWeight","knative.dev/controller":"ingress-controller"}




kubectl log controller-6d9c97d6b9-2hvnw -n knative-serving | grep -i error
{"level":"error","ts":"2019-09-06T13:48:31.977Z","logger":"controller.serverlessservice-controller","caller":"controller/controller.go:345","msg":"Reconcile error","knative.dev/controller":"serverlessservice-controller","error":"Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"helloworld-go-v46j2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"error","ts":"2019-09-06T13:48:32.880Z","logger":"controller.route-controller","caller":"route/reconcile_resources.go:99","msg":"Failed to update Ingress","knative.dev/controller":"route-controller","knative.dev/traceid":"75053a45-2652-41aa-8641-802bb99f262d","knative.dev/key":"default/helloworld-go","error":"Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcileIngress\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/reconcile_resources.go:99\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcileIngressResources\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:297\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).reconcile\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:259\nknative.dev/serving/pkg/reconciler/route.(*Reconciler).Reconcile\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/route/route.go:113\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:330\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"error","ts":"2019-09-06T13:48:32.993Z","logger":"controller.route-controller","caller":"controller/controller.go:345","msg":"Reconcile error","knative.dev/controller":"route-controller","error":"Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:345\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:331\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func1\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:282"}
{"level":"info","ts":"2019-09-06T13:48:32.994Z","logger":"controller.route-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"fd3f00e1-d0ac-11e9-97dc-0050568a6fd7\", APIVersion:\"serving.knative.dev/v1alpha1\", ResourceVersion:\"189198\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' Operation cannot be fulfilled on ingresses.networking.internal.knative.dev \"helloworld-go\": the object has been modified; please apply your changes to the latest version and try again","knative.dev/controller":"route-controller"}

To unsubscribe from this group and stop receiving emails from it, send an email to knativ...@googlegroups.com.

Assaf Sauer

unread,
Sep 6, 2019, 3:10:19 PM9/6/19
to Nghia Tran, Knative Users
Hi Nghia,

i followed the instruction , Istio version is 1.7

root@ubuntu:/home/viewadmin/knative# istioctl version
version.BuildInfo{Version:"1.1.7", GitRevision:"eec7a74473deee98cad0a996f41a32a47dd453c2", User:"root", Host:"341b3bf0-76ac-11e9-b644-0a580a2c0404", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.6-6-geec7a74"}
root@ubuntu:/home/viewadmin/knative# 

do i need to use different Istio ver ?

To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/93ec01fd-e671-4cc2-82dd-ef2b99ece341%40googlegroups.com.

Nghia Tran

unread,
Sep 6, 2019, 4:33:00 PM9/6/19
to Knative Users
According to Istio API, headers is a valid field https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/#HTTPRouteDestination, however it wasn't valid in Istio 1.0.x versions.

I suspect we have a left over installation of some kind, so there is still an Istio 1.0 webhook.

Can you please run this command `kubectl get svc -n istio-system --show-labels` to see if there are chart labels indicating your Istio versions?

Thanks

Assaf Sauer

unread,
Sep 6, 2019, 5:26:58 PM9/6/19
to Nghia Tran, Knative Users
Hi , 

thanks , pls find output below:

root@ubuntu:/home/viewadmin/knative# kubectl get svc -n istio-system --show-labels

NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)                                                                                                                                      AGE       LABELS

istio-citadel            ClusterIP      10.100.200.185   <none>                      8060/TCP,15014/TCP                                                                                                                           143m      app=security,chart=security,heritage=Tiller,istio=citadel,release=release-name

istio-galley             ClusterIP      10.100.200.35    <none>                      443/TCP,15014/TCP,9901/TCP                                                                                                                   143m      app=galley,chart=galley,heritage=Tiller,istio=galley,release=release-name

istio-ingressgateway     LoadBalancer   10.100.200.21    10.40.14.112,100.64.128.9   15020:32067/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32760/TCP,15030:30207/TCP,15031:32221/TCP,15032:30318/TCP,15443:30958/TCP   149m      app=istio-ingressgateway,chart=gateways,heritage=Tiller,istio=ingressgateway,release=release-name

istio-pilot              ClusterIP      10.100.200.226   <none>                      15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       149m      app=pilot,chart=pilot,heritage=Tiller,istio=pilot,release=release-name

istio-policy             ClusterIP      10.100.200.61    <none>                      9091/TCP,15004/TCP,15014/TCP                                                                                                                 143m      app=mixer,chart=mixer,heritage=Tiller,istio=mixer,release=release-name

istio-sidecar-injector   ClusterIP      10.100.200.8     <none>                      443/TCP                                                                                                                                      143m      app=sidecarInjectorWebhook,chart=sidecarInjectorWebhook,heritage=Tiller,istio=sidecar-injector,release=release-name

istio-telemetry          ClusterIP      10.100.200.198   <none>                      9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       143m      app=mixer,chart=mixer,heritage=Tiller,istio=mixer,release=release-name

root@ubuntu:/home/viewadmin/knative# 


To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/ad575028-b84b-43c5-beb1-55712c061440%40googlegroups.com.

William Tam

unread,
Sep 9, 2019, 11:29:26 AM9/9/19
to Assaf Sauer, Nghia Tran, Knative Users
I wonder how is it working out?  I don't see istio version number in the labels and couldn't find it in the logs neither.

Assaf Sauer

unread,
Sep 9, 2019, 12:00:33 PM9/9/19
to William Tam, Nghia Tran, Knative Users
Hi William,
i cleaned up ISTIO installation , reinstall and my route config seems to shift traffic correctly.  Many Thanks for your support!!
just one question, the Revision HTTP requests graph is not working.  the revisions seems to be missing (all other reports working fine),  is it a bug ?  any idea ?

Thanks, 
Assaf

image.png
Reply all
Reply to author
Forward
0 new messages