Unable to access the deployed service

[Mukundha Madhavan Gnana Sekaran]

I am trying to deploy a service on

GKE version 1.10.7-gke.2

Istio version 1.0.2

Knative version v0.1.1

(https://github.com/knative/serving/releases/download/v0.1.1/release.yaml)

I see this in logs for the istio-proxy container in my pod

 [libprotobuf ERROR src/istio/mixerclient/report_batch.cc:83] Mixer Report failed with: UNKNOWN:1 error occurred:%0A%0A* evaluation failed at [requestlog.logentry.istio-system]'Timestamp': 'lookup failed: 'request.time''

I see these errors in the logs of queue-proxy container in my pod

{"level":"error","ts":"2018-10-08T12:18:46.555Z","logger":"queueproxy","caller":"queue/main.go:137","msg":"Stat sink not connected.","knative.dev/namespace":"default","knative.dev/configuration":"foodcart-api","knative.dev/revision":"foodcart-api-00001","knative.dev/pod":"foodcart-api-00001-deployment-6b98675677-vnqcw","stacktrace":"main.statReporter\n\t/usr/local/google/home/mattmoor/go/src/github.com/knative/serving/cmd/queue/main.go:137"}

{"level":"error","ts":"2018-10-08T12:18:46.600Z","logger":"queueproxy","caller":"queue/main.go:114","msg":"Retrying connection to autoscaler.{error 25 0  dial tcp 10.7.245.122:8080: connect: connection refused}","knative.dev/namespace":"default","knative.dev/configuration":"foodcart-api","knative.dev/revision":"foodcart-api-00001","knative.dev/pod":"foodcart-api-00001-deployment-6b98675677-vnqcw","stacktrace":"main.connectStatSink\n\t/usr/local/google/home/mattmoor/go/src/github.com/knative/serving/cmd/queue/main.go:114"}

{"level":"error","ts":"2018-10-08T12:18:47.555Z","logger":"queueproxy","caller":"queue/main.go:137","msg":"Stat sink not connected.","knative.dev/namespace":"default","knative.dev/configuration":"foodcart-api","knative.dev/revision":"foodcart-api-00001","knative.dev/pod":"foodcart-api-00001-deployment-6b98675677-vnqcw","stacktrace":"main.statReporter\n\t/usr/local/google/home/mattmoor/go/src/github.com/knative/serving/cmd/queue/main.go:137"}

{"level":"info","ts":"2018-10-08T12:18:47.635Z","logger":"queueproxy","caller":"queue/main.go:116","msg":"Connected to stat sink.","knative.dev/namespace":"default","knative.dev/configuration":"foodcart-api","knative.dev/revision":"foodcart-api-00001","knative.dev/pod":"foodcart-api-00001-deployment-6b98675677-vnqcw"}

2018/10/08 12:19:50 http: proxy error: context canceled

My calls are not reaching the user-container

I see `foodcart-api-00001-autoscaler-5475f565fd-76hc6` running in namespace 'knative-serving'

Can you pls help to troubleshoot this issue?

Thanks,

Mukundha

[Ben Browning]

Mukundha,

Knative Serving v0.1.1 was tested against Istio 0.8. You'll need to
install Istio 0.8 (as shown at
https://github.com/knative/docs/blob/08ab237cb330b30989044c710766db49784ddfb6/install/Knative-with-GKE.md#installing-istio)
with this Knative version.

The next Knative Serving release will be compatible with Istio 1.0.2.

Thanks,

Ben

> --
> You received this message because you are subscribed to the Google Groups "Knative Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
> To post to this group, send email to knativ...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/4c938070-8d9d-4691-87fd-1cc5fae9893e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Mukundha Madhavan Gnana Sekaran]

Thanks for your quick reply Ben, do you know when is the next release? 

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/CAOpbpxFAx95MYn10LuJ8Gaf-TOhvfm4K%2BcokUYWB_MNYgN0NWg%40mail.gmail.com.

[Matthew Moore]

+folks

Hmm, we should make sure 0.1.1 is compatible with Istio 1.x.y (at least until we get 0.2 out, shooting for 11/1).

I know we've backported at least one fix to this end (thus 0.1.0 -> 0.1.1).

Also, if you have any interest in trying this through the "add on" mechanism, please let us know.

-M

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/CAFU60J%2B65B_K72G72HU0iV%3DLOf4qaYuoS1OQp2xQoXm9BGCAvQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Matthew Moore

Container Development Uber-TL

Developer Infrastructure @ Google

[Mukundha Madhavan Gnana Sekaran]

Hi Matthew, What's the "add on" mechanism? will it allow me to have the latest version of Istio?

[Matthew Moore]

We believe that the version of Istio bundled with the addon (0.1.1) on the latest GKE version is 1.0.2.

IMO we can update the release branch to this as well, but my understanding was that there weren't major compat problems.  +Evan Anderson +Ville Aikas Any objections?

Certainly with testing on the release branch fixed (thanks, Ben!) it makes sense to test what folks are likely to run.

-M

[Mukundha Madhavan Gnana Sekaran]

Thats great, I will give it as try in serverless add-on and see if the errors go away

Thanks,

[Matthew Moore]

I think you need to signup: https://docs.google.com/forms/d/e/1FAIpQLSdG5cCIiHhkW7srw9MWvdiLEsLXwJES1R3lnKgAn-opy3_iuQ/viewform

[Ben Browning]

I wasn't aware anyone had tested the release branch with Istio 1.0.2.
So, if you have testing that shows it works, then ignore what I
originally said. I just saw what looks like a networking error and
untested combinations of Istio and Knative and thus was my
recommendation to switch to a tested combination.

I do agree that if we want to support Istio 1.0.x and Knative Serving
0.1.x releases then we should probably bump that in the release branch
and run tests verifying that combination works.

[Mukundha Madhavan Gnana Sekaran]

ok serverless add-on has Istio 1.0.2 

and my demo service worked fine in it,

not sure whats wrong with the manual installation, let me try from scratch again

Thanks,

[Mukundha Madhavan Gnana Sekaran]

Thanks everyone 

QQ: Do i need to enable auto sidecar injection?

In my manual setup I did enable auto sidecar injection and my pod ended up having these 3 containers - [user-container queue-proxy istio-proxy]

while, on serverless add-on, I did not enable sidecar injection and my pod only has these 2 - [user-container queue-proxy]

Thanks,

Mukundha

[Mukundha Madhavan Gnana Sekaran]

One more:

Even after I enabled auto sidecar injection, i don't see istio-proxy deployed in my pod for services deployed thro knative 

(apiVersion: serving.knative.dev/v1alpha1
kind: Service)

But other pods, automatically gets the istio-proxy sidecar

Is Knative overriding the istio behavior?

Here is my usecase: I am trying to get some istio policies applied on my knative services, is this possible? would you suggest any other approaches for what I am trying to do

This was working few weeks back [probably on a older version]

Thanks,

Mukundha

[Mukundha Madhavan Gnana Sekaran]

Thanks for your quick reply Zhimin

I see a lot of these continuously in the logs of istio-sidecar-injector pod

"2018-10-09T06:57:50.797294Z error Patch webhook failed: mutatingwebhookconfigurations.admissionregistration.k8s.io "istio-sidecar-injector" not found"

However, if i deploy a istio helloworld service, i see the sidecar proxy getting created [istio-proxy], but while deploying a knative service no istio-proxy, but a queue-proxy sidecar

Thanks,

On Tue, Oct 9, 2018 at 7:30 AM Zhimin Xiang <zhi...@google.com> wrote:

Hi Mukundha,

Could you please show me the log of istio-sidecar-injector pod? There is a known issue of istio-sidecar-injector in the Istio addon, which causes the failure of sidecar injection.

Thanks,

Zhimin

[Zhimin Xiang]

+Martin Ostrowski 

According to the log, looks like it is the known issue in GKE Istio addon. But I am surprised that istio sidecar got injected when deploying a Istio helloworld service.

Mukundha, which GKE version are you using? I will have a try tomorrow.

Thanks,

Zhimin

[Mukundha Madhavan Gnana Sekaran]

Thanks Zhimin, my GKE version is 1.10.7-gke.2

Pls ignore the sidecar for helloworld, just checked the source - it has sidecar configured in the deployment spec - sorry for the confusion. 

automatic sidecar injection is not working in all cases

Thanks,