CometD in kubernetes

[Jaya Gupta]

Hi,

I wanted to check if we can run CometD servers in kubernetes pods as part of deployment and service.
If yes, what should be the configuration for oort.url for multicastStrategy?

Thanks,

Jaya

[Jaya Gupta]

Hi,
I am giving some more context on how I have been trying to deploy cometd application in kubernetes since the time I posted the query.
I am creating a deployment which creates two replicas of the application and a service
I am using the following as my config:

```apiVersion: v1
kind: Service
metadata:
  name: test-service
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/backend-config: '{"ports": {"8080":"be"}}'

spec:
  type: NodePort
  ports:
    - protocol: "TCP"
      port: 8080
      targetPort: 8080
      nodePort: 30005
—
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "test-cookie"
    nginx.ingress.kubernetes.io/session-cookie-expires: "86400"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "86400"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
spec:
  rules:
    - http:
        paths:
          - path: /cometd
            pathType: ImplementationSpecific
            backend:
              service:
                name: test-service
                port:
                  number: 8080
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: be
spec:
  sessionAffinity:
    affinityType: "GENERATED_COOKIE"
    affinityCookieTtlSec: 86400```

I have 2 clients connected. Each pod gets some clients, but I am trying to send message from one client to another, I get seti location as empty. It looks like the connection are not getting shared between the pods.

Here are cometd logs:
```OORT Url :: http://10.92.1.23:8080/cometd
11:53:42.196 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Bound multicast receiver to 0.0.0.0/0.0.0.0:5577 with protocol family INET
11:53:42.198 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Joining multicast group with name:eth0 (eth0)
11:53:42.199 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Joined multicast group with name:eth0 (eth0)
11:53:42.199 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Joining multicast group with name:lo (lo)
11:53:42.200 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Skipped joining multicast group with name:lo (lo)
11:53:42.203 Oort-Multicast-Sender-1 [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Entering multicast sender thread
11:53:42.212 Oort-Multicast-Receiver-2 [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Entering multicast receiver thread on /0.0.0.0:5577
11:53:42.222 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Observing channel /seti/all
11:53:42.225 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Observing channel /seti/http_10_92_1_23_8080_cometd
11:53:42.225 main [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] org.cometd.oort.Seti[http_10_92_1_23_8080_cometd] started
11:53:42.330 main [DEBUG][o.c.o.O.h.connectors] Set local Info[connectors/0] (from http://10.92.1.23:8080/cometd): {}
11:53:42.331 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Observing channel /oort/objects/connectors
11:53:42.332 main [DEBUG][o.c.o.O.h.connectors] OortStringMap[connectors]@http://10.92.1.23:8080/cometd started
11:53:42.344 main [DEBUG][o.c.o.O.http_10_92_1_23_8080_cometd] Observing channel /preview/**
2021-12-20 11:53:42.345:INFO :oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@c86b9e3{ROOT,/,file:///var/lib/jetty/webapps/ROOT/,AVAILABLE}{/var/lib/jetty/webapps/ROOT}
2021-12-20 11:53:42.368:INFO :oejs.AbstractConnector:main: Started ServerConnector@57ee3b03{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2021-12-20 11:53:42.385:INFO :oejs.Server:main: Started Server@1afd44cb{STARTING}[11.0.7,sto=5000] @3655ms
12:13:50.699 qtp1782704802-19 [DEBUG][c.i.router.service.ConnectorService] Logged in session 'ed30b867-d578-4a43-b993-2749fa3afebd'@68dhjvhtj2lzbknx05xe48kqs
12:13:50.701 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Associations: 1
12:13:50.701 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Associated ServerSessionImpl@73ad9a29[68dhjvhtj2lzbknx05xe48kqs,HANDSHAKEN,cycle=0,last=2,expire=0] to user ed30b867-d578-4a43-b993-2749fa3afebd
12:13:50.702 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Broadcasting association addition for user ed30b867-d578-4a43-b993-2749fa3afebd
12:13:50.712 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Received broadcast message {data={replace=false, presence=true, userIds=[ed30b867-d578-4a43-b993-2749fa3afebd], oortURL=http://10.92.1.23:8080/cometd}, channel=/seti/all}
12:13:50.712 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Received presence message {replace=false, presence=true, userIds=[ed30b867-d578-4a43-b993-2749fa3afebd], oortURL=http://10.92.1.23:8080/cometd}
12:13:50.713 qtp1782704802-19 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Notifying presence listeners {replace=false, presence=true, userIds=[ed30b867-d578-4a43-b993-2749fa3afebd], oortURL=http://10.92.1.23:8080/cometd}
12:13:50.716 qtp1782704802-19 [DEBUG][o.c.o.O.h.connectors] Sharing map put {oort.info.url=http://10.92.1.23:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=ed30b867-d578-4a43-b993-2749fa3afebd connectorProperties={tenantId=tenant1, id=ed30b867-d578-4a43-b993-2749fa3afebd}], oort.map.key=ed30b867-d578-4a43-b993-2749fa3afebd}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry}
12:13:50.718 qtp1782704802-19 [DEBUG][o.c.o.O.h.connectors] Received broadcast {data={oort.info.url=http://10.92.1.23:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=ed30b867-d578-4a43-b993-2749fa3afebd connectorProperties={tenantId=tenant1, id=ed30b867-d578-4a43-b993-2749fa3afebd}], oort.map.key=ed30b867-d578-4a43-b993-2749fa3afebd}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry}, channel=/oort/objects/connectors}
12:13:50.718 qtp1782704802-19 [DEBUG][o.c.o.O.h.connectors] Generated version=1 for {oort.info.url=http://10.92.1.23:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=ed30b867-d578-4a43-b993-2749fa3afebd connectorProperties={tenantId=tenant1, id=ed30b867-d578-4a43-b993-2749fa3afebd}], oort.map.key=ed30b867-d578-4a43-b993-2749fa3afebd}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry, oort.info.version=1}
12:13:50.719 qtp1782704802-19 [DEBUG][o.c.o.O.h.connectors] Processing update, version=0, data={oort.info.url=http://10.92.1.23:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=ed30b867-d578-4a43-b993-2749fa3afebd connectorProperties={tenantId=tenant1, id=ed30b867-d578-4a43-b993-2749fa3afebd}], oort.map.key=ed30b867-d578-4a43-b993-2749fa3afebd}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry, oort.info.version=1}
12:13:50.721 qtp1782704802-19 [DEBUG][o.c.o.O.h.connectors] Local map oort.map.put of (ed30b867-d578-4a43-b993-2749fa3afebd=null->ConnectorInfo[ id=ed30b867-d578-4a43-b993-2749fa3afebd connectorProperties={ tenantId=tenant1, id=ed30b867-d578-4a43-b993-2749fa3afebd}])
12:13:51.164 qtp1782704802-20 [DEBUG][c.i.router.service.RouterService] Forwarding preview request : {data={tenantId=tenant1, channel=/preview/request, from_connector_id=ed30b867-d578-4a43-b993-2749fa3afebd, to_connector_id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b, depType=onPrem}, channel=/preview/request, id=38}
12:13:51.165 qtp1782704802-20 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Sending message to locations [SetiLocation[/seti/all]]
12:13:51.166 qtp1782704802-20 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Received broadcast message {data={channel=/preview/request, setiId=http_10_92_1_23_8080_cometd, data={data={tenantId=tenant1, channel=/preview/request, from_connector_id=ed30b867-d578-4a43-b993-2749fa3afebd, to_connector_id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b, depType=onPrem}, channel=/preview/request, id=38}, userId=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, channel=/seti/all}
12:13:51.166 qtp1782704802-20 [DEBUG][o.c.o.S.http_10_92_1_23_8080_cometd] Received message {channel=/preview/request, setiId=http_10_92_1_23_8080_cometd, data={data={tenantId=tenant2, channel=/preview/request, from_connector_id=ed30b867-d578-4a43-b993-2749fa3afebd, to_connector_id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, channel=/preview/request, id=38}, userId=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b} for locations []







OORT Url :: http://10.92.0.19:8080/cometd
11:53:42.670 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Bound multicast receiver to 0.0.0.0/0.0.0.0:5577 with protocol family INET
11:53:42.671 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Joining multicast group with name:eth0 (eth0)
11:53:42.672 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Joined multicast group with name:eth0 (eth0)
11:53:42.672 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Joining multicast group with name:lo (lo)
11:53:42.672 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Skipped joining multicast group with name:lo (lo)
11:53:42.694 Oort-Multicast-Receiver-2 [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Entering multicast receiver thread on /0.0.0.0:5577
11:53:42.681 Oort-Multicast-Sender-1 [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Entering multicast sender thread
11:53:42.705 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Observing channel /seti/all
11:53:42.711 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Observing channel /seti/http_10_92_0_19_8080_cometd
11:53:42.712 main [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] org.cometd.oort.Seti[http_10_92_0_19_8080_cometd] started
11:53:42.813 main [DEBUG][o.c.o.O.h.connectors] Set local Info[connectors/0] (from http://10.92.0.19:8080/cometd): {}
11:53:42.814 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Observing channel /oort/objects/connectors
11:53:42.815 main [DEBUG][o.c.o.O.h.connectors] OortStringMap[connectors]@http://10.92.0.19:8080/cometd started
11:53:42.825 main [DEBUG][o.c.o.O.http_10_92_0_19_8080_cometd] Observing channel /preview/**
2021-12-20 11:53:42.826:INFO :oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@c86b9e3{ROOT,/,file:///var/lib/jetty/webapps/ROOT/,AVAILABLE}{/var/lib/jetty/webapps/ROOT}
2021-12-20 11:53:42.848:INFO :oejs.AbstractConnector:main: Started ServerConnector@1ce93c18{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2021-12-20 11:53:42.861:INFO :oejs.Server:main: Started Server@1afd44cb{STARTING}[11.0.7,sto=5000] @4194ms
12:11:10.450 qtp1782704802-12 [DEBUG][c.i.router.service.ConnectorService] Logged in session '785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b'@6117xrw1csvhoh1rj3u1fbamqyk
12:11:10.451 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Associations: 1
12:11:10.451 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Associated ServerSessionImpl@3da767d3[6117xrw1csvhoh1rj3u1fbamqyk,HANDSHAKEN,cycle=0,last=2,expire=0] to user 785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b
12:11:10.452 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Broadcasting association addition for user 785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b
12:11:10.461 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Received broadcast message {data={replace=false, presence=true, userIds=[785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b], oortURL=http://10.92.0.19:8080/cometd}, channel=/seti/all}
12:11:10.461 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Received presence message {replace=false, presence=true, userIds=[785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b], oortURL=http://10.92.0.19:8080/cometd}
12:11:10.462 qtp1782704802-12 [DEBUG][o.c.o.S.http_10_92_0_19_8080_cometd] Notifying presence listeners {replace=false, presence=true, userIds=[785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b], oortURL=http://10.92.0.19:8080/cometd}
12:11:10.463 qtp1782704802-12 [DEBUG][o.c.o.O.h.connectors] Sharing map put {oort.info.url=http://10.92.0.19:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b connectorProperties={tenantId=tenant2, id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}], oort.map.key=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry}
12:11:10.465 qtp1782704802-12 [DEBUG][o.c.o.O.h.connectors] Received broadcast {data={oort.info.url=http://10.92.0.19:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b connectorProperties={tenantId=tenant2, id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}], oort.map.key=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry}, channel=/oort/objects/connectors}
12:11:10.466 qtp1782704802-12 [DEBUG][o.c.o.O.h.connectors] Generated version=1 for {oort.info.url=http://10.92.0.19:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b connectorProperties={  tenantId=tenant2, id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}], oort.map.key=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry,oort.info.version=1}
12:11:10.466 qtp1782704802-12 [DEBUG][o.c.o.O.h.connectors] Processing update, version=0, data={oort.info.url=http://10.92.0.19:8080/cometd, oort.info.object={oort.map.value=ConnectorInfo[ id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b connectorProperties={tenantId=tenant2, id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}], oort.map.key=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}, oort.info.name=connectors, oort.info.action=oort.map.put, oort.info.type=oort.map.entry, oort.info.version=1}
12:11:10.468 qtp1782704802-12 [DEBUG][o.c.o.O.h.connectors] Local map oort.map.put of (785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b=null->ConnectorInfo[ id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b connectorProperties={tenantId=tenant2, id=785fb6cf-c7bc-4e64-b3a7-d8cc56cdaf8b}])```

This is my StartupServlet:

```public class StartupServlet extends GenericServlet {
@Override
public void init() throws ServletException {
try {
BayeuxServer bayeux = (BayeuxServer)getServletContext().getAttribute(BayeuxServer.ATTRIBUTE);

String oortUrl = "http://" + System.getenv("MY_POD_IP") + ":8080/cometd";
LOGGER.info("OORT Url :: {} ", oortUrl);
Oort oort = new Oort(bayeux, oortUrl);
JSONContextClient jsonContextClient = new JSONContextClient();
oort.setJSONContextClient(jsonContextClient);
oort.setSecret("oort_objects");
oort.start();

OortMulticastConfigurer oortMulticastConfigurer = new OortMulticastConfigurer(oort);
oortMulticastConfigurer.start();

Seti seti = new Seti(oort);
seti.start();

bayeux.setSecurityPolicy(new SecurityPolicy(oort));

ServerAnnotationProcessor processor = new ServerAnnotationProcessor(bayeux, oort, seti);
processor.process("Service Name");
} catch (Exception ex) {
throw new ServletException(ex);
}
}

```

Any help with regards  to run the cometd servers in cluster mode in kubernetes would be helpful.

Thanks,
Jaya