knative eventing error "connection refused" in broker

[Guillaume Albini]

Hello,

I have installed knative with gloo (glooctl install knative --install-knative-version=0.10.0 --install-eventing).

I follow the documentation (https://knative.dev/docs/eventing/broker-trigger/) but i have an error when i push an event in the broker with 

curl -v "http://default-broker.default.svc.cluster.local/" \
  -X POST \
  -H "X-B3-Flags: 1" \
  -H "CE-SpecVersion: 0.2" \
  -H "CE-Type: dev.knative.foo.bar" \
  -H "CE-Time: 2018-04-05T03:56:24Z" \
  -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
  -H "CE-Source: dev.knative.example" \
  -H 'Content-Type: application/json' \
  -d '{ "much": "wow" }'


the error is inside the broker, see below : 

{"level":"warn","ts":1575211987.0189402,"logger":"fallback","caller":"http/transport.go:502",
"msg":"got an error from receiver fn","error":"Post http://my-service.default.svc.cluster.local: dial tcp 10.0.15.143:80: connect: connection refused"}                                                                                                                                                           │
{"level":"warn","ts":1575211987.0190103,"logger":"fallback","caller":"http/transport.go:594",
"msg":"error returned from invokeReceiver","error":"Post http://my-service.default.svc.cluster.local: dial tcp 10.0.15.143:80: connect: connection refused"}

Do you have any idea ?

The sample on the page https://knative.dev/docs/eventing/getting-started/ with native kubernetes deployment/service work correctly.

Best regards
Guillaume

[Victor Agababov]

Presuming the target service is running, do you have some mesh sidecars installed perhaps that restrict the connections?

[Guillaume Albini]

Hi Victor,

yes, the service is running and it can be call with public url.

I have juste create a cluster on GKE with the command:

gcloud beta container clusters create $CLUSTER_NAME \
  --machine-type=n1-standard-2 \
  --cluster-version=latest --zone=$CLUSTER_ZONE \
  --enable-stackdriver-kubernetes --enable-ip-alias \
  --enable-autoscaling --min-nodes=1 --max-nodes=10 \
  --enable-autorepair \
  --scopes cloud-platform

and then run glooctl install knative --install-knative-version=0.10.0 --install-eventing

I am not installed other mesh service.

see below the result of : kubectl get svc -n gloo-system

NAME                     TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                      AGE

clusteringress-proxy     LoadBalancer   10.0.4.209    34.76.12.189     80:30219/TCP,443:30592/TCP   2d2h

gloo                     ClusterIP      10.0.1.227    <none>           9977/TCP,9988/TCP,9966/TCP   2d2h

knative-external-proxy   LoadBalancer   10.0.4.217    104.199.11.220   80:31452/TCP,443:30715/TCP   2d2h

knative-internal-proxy   ClusterIP      10.0.15.143   <none>           80/TCP,443/TCP               2d2h

it's seem the knative-internal-proxy (10.0.15.143) not response (dial tcp 10.0.15.143:80: connect: connection refused)

other idea ?

[Nacho Cano]

This seems to be another problem, but also sending CE version 0.2 won't work. You will have to send 0.3 events.

IIRC documentation was updated but I do not see the updated version.

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/c13a1cb5-a228-482e-b7a6-8cfc8ffd024b%40googlegroups.com.

[Scott Weiss]

Hi Guillaume,

If Gloo has been properly configured, you should see a Proxy CRD that lives in the gloo-system namespace named `knative-external-proxy`.

If following the Hello World tutorial, the crd should look something like this:

apiVersion: gloo.solo.io/v1

kind: Proxy

metadata:

  creationTimestamp: "2019-12-02T18:25:08Z"

  generation: 6

  labels:

    created_by: gloo-knative-translator

  name: knative-external-proxy

  namespace: gloo-system

  resourceVersion: "4204"

  selfLink: /apis/gloo.solo.io/v1/namespaces/gloo-system/proxies/knative-external-proxy

  uid: 11f146a0-1531-11ea-a487-42010af00002

spec:

  listeners:

  - bindAddress: '::'

    bindPort: 80

    httpListener:

      virtualHosts:

      - domains:

        - helloworld-go.default.svc.cluster.local

        - helloworld-go.default.svc.cluster.local:80

        - helloworld-go.default.svc

        - helloworld-go.default.svc:80

        - helloworld-go.default

        - helloworld-go.default:80

        - helloworld-go.default.example.com

        - helloworld-go.default.example.com:80

        name: default.helloworld-go-0

        routes:

        - matchers:

          - regex: .*

          options:

            retries:

              numRetries: 3

              perTryTimeout: 600s

            timeout: 600s

          routeAction:

            multi:

              destinations:

              - destination:

                  kube:

                    port: 80

                    ref:

                      name: helloworld-go-km7tj

                      namespace: default
...

If the proxy is not present or contains no listeners, the proxy will not serve traffic. This will happen if there are no knative Ingress objects detected by gloo. 

You can make sure these exist by running 

kubectl get ingresses.networking.internal.knative.dev --all-namespaces

If they exist, and the Proxy looks to be correct configuration, we should make sure the URL returned by `glooctl proxy url` is accurate.

We can test Gloo directly by running the following:

# port forwarfd to the container
kubectl port-forward -n gloo-system deployment/knative-external-proxy 8080:80&
# test with a curl directly to the container
curl -v localhost:8080 # should return a 404

Please let me know the results of the above tests. Thanks!

[Guillaume Albini]

Hi Scott,

All your tests are ok but the problem is not to call directly a service but when i try to pass by the default-broker (eventing mode).

And the specversion set to 0.3 does not resolved the problem.

I try the next commands but i have the same error in portforwar logs.

kubectl port-forward -n gloo-system deployment/knative-internal-proxy 8080:80

Forwarding from 127.0.0.1:8080 -> 80

Forwarding from [::1]:8080 -> 80

Handling connection for 8080

E1202 21:04:19.520397   37788 portforward.go:400] an error occurred forwarding 8080 -> 80: error forwarding port 80 to pod 457fd763f614d56119ec8923d0d583ae98546f35a725761c7fecc12873864334, uid : exit status 1: 2019/12/02 20:04:19 socat[9347] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused

curl -v localhost:8080

* Rebuilt URL to: localhost:8080/

*   Trying ::1...

* TCP_NODELAY set

* Connected to localhost (::1) port 8080 (#0)

> GET / HTTP/1.1

> Host: localhost:8080

> User-Agent: curl/7.54.0

> Accept: */*

> 

* Empty reply from server

* Connection #0 to host localhost left intact

curl: (52) Empty reply from server

Thanks for your help.

[Guillaume Albini]

See below the yaml of knative-nternal-proxy

apiVersion: gloo.solo.io/v1

kind: Proxy

metadata:

  creationTimestamp: "2019-12-02T19:26:59Z"

  generation: 2

  labels:

    created_by: gloo-knative-translator

  name: knative-internal-proxy

  namespace: gloo-system

  resourceVersion: "1656"

  selfLink: /apis/gloo.solo.io/v1/namespaces/gloo-system/proxies/knative-internal-proxy

  uid: b59b9917-1539-11ea-bb05-42010a840076

spec: {}

status:

  reported_by: gloo

  state: 1

[Scott Weiss]

Hi Guillaume,

The above behavior is what I would expect. There are no routes created by Gloo to serve traffic, hence the connection refused. Do you have any knative ingress objects? 

`kubectl get ingresses.networking.internal.knative.dev --all-namespaces`

--

You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/ced802a6-d07a-468e-a621-ffd845ac0231%40googlegroups.com.

[Guillaume Albini]

Yes, I have knative ingress objects :

kubectl get ingresses.networking.internal.knative.dev --all-namespaces

NAMESPACE   NAME            READY   REASON

default     helloworld-go   True    

default     my-service      True    

To unsubscribe from this group and stop receiving emails from it, send an email to knativ...@googlegroups.com.

[Scott Weiss]

Can you share logs from the `ingress` deployment in gloo-system? It seems it is not picking up these ingresses

To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/0001f7ab-a8aa-43ca-9f3e-ac6564c15415%40googlegroups.com.

[Guillaume Albini]

Hello,

See in attachment the log of ingress pod.

Best regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/0001f7ab-a8aa-43ca-9f3e-ac6564c15415%40googlegroups.com.

[Scott Weiss]

Would you be able to share a Kservice that reproduces this issue? I see that the ingress controller sees the 2 knative ingresses, but for some reason is producing a Proxy result that contains no virtual hosts (we should expect 1 virtual host for each rule in each knative ingress).

To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/a5d12e0b-9395-4e4e-b436-4f6f4e48d19f%40googlegroups.com.