--
You received this message because you are subscribed to the Google Groups "submariner-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to submariner-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/a343d4b3-bcbc-4ce7-a42e-4ddb3d00983cn%40googlegroups.com.
Thank you- this appears to have been the problem. Both clusters are now able to join (confirmed via subctl show) and I am able to run tests.
Tests start running, but I run into issues with both connectivity and service-discovery suites.
I am not running gateway suite since I have only 1 node cluster.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/b7643ed9-16d8-4c3e-b340-3fc00ce1a5ebn%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CAOW2eyzGyZO-7uhhuW6oQAMPB7vYqQSwMCw6b_O3DObwrYDOOQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CAJADjR3KZ5Prew97QF5dPB8zA8tDWmKp48t6Pur3%2Ba%2BBa%2Ba9Bw%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/400eecbd-d2e4-47bb-8d4b-f5d6c14c5ebfn%40googlegroups.com.
Hi Aswin,Here is the lighthouse-agent log after a run of the test suite:john@green:~/.local/bin$ kubectl logs pod/submariner-lighthouse-agent-6bdff54f75-t798l --namespace submariner-operator
+ trap 'exit 1' SIGTERM SIGINT
+ LIGHTHOUSE_VERBOSITY=1
+ '[' '' == true ']'
+ DEBUG=-v=1
+ exec lighthouse-agent -v=1 -alsologtostderr
I0222 16:55:16.591705 1 main.go:34] AgentSpec: {green submariner-operator false}
W0222 16:55:16.592048 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0222 16:55:16.592378 1 main.go:51] Starting submariner-lighthouse-agent {green submariner-operator false}
I0222 16:55:38.304413 1 agent.go:167] Starting Agent controller
I0222 16:55:38.918260 1 agent.go:193] Agent controller started
E0223 01:28:59.605893 1 queue.go:83] Endpoints -> EndpointSlice: Failed to process object with key "nginx-test/nginx-ss": error creating &unstructured.Unstructured{Object:map[string]interface {}{"addressType":"IPv4", "apiVersion":"discovery.k8s.io/v1beta1", "endpoints":[]interface {}{map[string]interface {}{"addresses":[]interface {}{"10.71.1.65"}, "conditions":map[string]interface {}{"ready":true}, "hostname":"web-1", "topology":map[string]interface {}{"kubernetes.io/hostname":"gwork"}}}, "kind":"EndpointSlice", "metadata":map[string]interface {}{"labels":map[string]interface {}{"endpointslice.kubernetes.io/managed-by":"lighthouse-agent.submariner.io", "lighthouse.submariner.io/sourceCluster":"green", "lighthouse.submariner.io/sourceName":"nginx-ss", "lighthouse.submariner.io/sourceNamespace":"nginx-test", "multicluster.kubernetes.io/service-name":"nginx-ss-nginx-test-green"}, "name":"nginx-ss-green", "namespace":"nginx-test", "ownerReferences":[]interface {}{map[string]interface {}{"apiVersion":"lighthouse.submariner.io.v2alpha1", "controller":false, "kind":"ServiceImport", "name":"nginx-ss-nginx-test-green", "uid":"2d2339af-382c-4db1-8a0e-c6a2055e8064"}}}, "ports":[]interface {}{map[string]interface {}{"name":"web", "port":80, "protocol":"TCP"}}}}: endpointslices.discovery.k8s.io "nginx-ss-green" is forbidden: unable to create new content in namespace nginx-test because it is being terminated
E0223 01:28:59.617115 1 queue.go:83] Endpoints -> EndpointSlice: Failed to process object with key "nginx-test/nginx-ss": error creating &unstructured.Unstructured{Object:map[string]interface {}{"addressType":"IPv4", "apiVersion":"discovery.k8s.io/v1beta1", "endpoints":interface {}(nil), "kind":"EndpointSlice", "metadata":map[string]interface {}{"labels":map[string]interface {}{"endpointslice.kubernetes.io/managed-by":"lighthouse-agent.submariner.io", "lighthouse.submariner.io/sourceCluster":"green", "lighthouse.submariner.io/sourceName":"nginx-ss", "lighthouse.submariner.io/sourceNamespace":"nginx-test", "multicluster.kubernetes.io/service-name":"nginx-ss-nginx-test-green"}, "name":"nginx-ss-green", "namespace":"nginx-test", "ownerReferences":[]interface {}{map[string]interface {}{"apiVersion":"lighthouse.submariner.io.v2alpha1", "controller":false, "kind":"ServiceImport", "name":"nginx-ss-nginx-test-green", "uid":"2d2339af-382c-4db1-8a0e-c6a2055e8064"}}}, "ports":interface {}(nil)}}: endpointslices.discovery.k8s.io "nginx-ss-green" is forbidden: unable to create new content in namespace nginx-test because it is being terminated
john@green:~/.local/bin$
Hi Aswin,Thanks- I performed the manual verification as suggested, but am not certain I deleted the service export in the correct way, so I would like to confirm.After exporting the service, the remote cluster is able to resolve it correctly (cluster exporting service = 10.72.x.x, remote cluster is 10.62.x.x):bash-5.0# dig nginx.test.svc.clusterset.local
; <<>> DiG 9.16.6 <<>> nginx.test.svc.clusterset.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44242
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 16a67284d43d8a77 (echoed)
;; QUESTION SECTION:
;nginx.test.svc.clusterset.local. IN A
;; ANSWER SECTION:
nginx.test.svc.clusterset.local. 5 IN A 10.72.158.54
;; Query time: 0 msec
;; SERVER: 10.67.0.10#53(10.67.0.10)
;; WHEN: Tue Feb 23 14:59:12 UTC 2021
;; MSG SIZE rcvd: 119Then I attempted to delete the service exports CRD from the cluster exporting the service using kubectl delete serviceexports.multicluster.x-k8s.io --all. This seemed to have no effect though, as I was still able to resolve the service from the remote cluster. The remained true after waiting > 5 minutes to in case TTL was my problem.So then I simply deleted the entire test namespace from the cluster exporting the service. Once deleted, I retried the dig query from the remote cluster and I do observe the service is no longer resolvable from the remote cluster:bash-5.0# dig nginx.test.svc.clusterset.local
; <<>> DiG 9.16.6 <<>> nginx.test.svc.clusterset.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64806
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 2d076e73e6d6a967 (echoed)
;; QUESTION SECTION:
;nginx.test.svc.clusterset.local. IN A
;; Query time: 8 msec
;; SERVER: 10.67.0.10#53(10.67.0.10)
;; WHEN: Tue Feb 23 16:03:36 UTC 2021
;; MSG SIZE rcvd: 72
It may stop resolving (ignoring the TTL part) as soon as you delete theservice export, or the service. The failure for that to happen is what theE2E failures are indicating.Which version of k8s are you using?Can you share the steps/details to reproduce in the form of an issue in the lighthouse repo?Can you also post there the logs for the lighthouse-agent on all clusters around the time youcreate, .. then delete the service or service export manually?@Aswin Suryanarayanan Is there anything else we could need? ^
To view this discussion on the web visit https://groups.google.com/d/msgid/submariner-users/CADSDy2iFOXCN6RsMEOPpTnA93vmR%3DZwLt8tFMG%3DHC6yNherKoQ%40mail.gmail.com.
Hi all,For some reason I am having trouble with my responses getting deleted on the mailing list- so I just wanted to respond to the questions above:I'm using K8s v1.20.4, submariner v0.8.1I logged the issue as requested at the lighthouse repo: https://github.com/submariner-io/lighthouse/issues/470
@Aswin Suryanarayanan, from the cluster where I invoked the 'dig' command, I did not delete/update anything with ServiceImport, after removing the ServiceExport. dig was still able to resolve the service. It was after I deleted the test namespace from the exporting cluster that the dig request at the remote cluster was finally no longer able to resolve the service.
regards,John