CAS 6 - Dockerized Deployments on two VMs with ticket registry

[Maksim Kopeyka]

Hello,

I have a problems with configuration of ticket registry in my env.

This env contains several VMs and each VM has a docker container with CAS 6.0.3. The multicast doesn't work on this env.

I have similar env with keycloak and I configured JDBC_PING to use distributed cache over all nodes.

So I need something similar for CAS.

Hazelcast

I found this example based on hazelcast. Unfortunately I don't see hazelcast members in CAS log, i.e. it doesn't work with CAS 6.1.3

https://apereo.github.io/2019/05/13/cas61x-docker-hazelcast-mgmtcenter/#dockerized-cas-deployment

This example uses this properties:

cas.ticket.registry.hazelcast.cluster.public-address

However CAS documentation has this one:

${configurationKey}.cluster.publicAddress

I tried both without success.

Ehcache

I tried to configure it via JGroupsCacheManagerPeerProviderFactory TCPPING but every time I have got a null pointer during login to CAS 6.0.3:

cas-only-local-2 | -------------------------------------------------------------------
cas-only-local-2 | GMS: address=a0e029837788-38715, cluster=ticketRegistryCacheManager, physical address=192.168.96.3:40001
cas-only-local-2 | -------------------------------------------------------------------
cas-only-local | -------------------------------------------------------------------
cas-only-local | GMS: address=3a992373bbcb-29947, cluster=ticketRegistryCacheManager, physical address=192.168.80.3:40001
cas-only-local | -------------------------------------------------------------------
cas-only-local | =============================================================
cas-only-local | WHO: casuser
cas-only-local | WHAT: Supplied credentials: [UsernamePasswordCredential(username=casuser, source=null)]
cas-only-local | ACTION: AUTHENTICATION_SUCCESS
cas-only-local | APPLICATION: CAS
cas-only-local | WHEN: Mon Jan 20 22:04:55 UTC 2020
cas-only-local | CLIENT IP ADDRESS: 192.168.1.106
cas-only-local | SERVER IP ADDRESS: 192.168.80.3
cas-only-local | =============================================================
cas-only-local | =============================================================
cas-only-local | WHO: casuser
cas-only-local | WHAT: NULL_POINTER_EXCEPTION
cas-only-local | ACTION: TICKET_GRANTING_TICKET_NOT_CREATED
cas-only-local | APPLICATION: CAS
cas-only-local | WHEN: Mon Jan 20 22:04:55 UTC 2020
cas-only-local | CLIENT IP ADDRESS: 192.168.1.106
cas-only-local | SERVER IP ADDRESS: 192.168.80.3
cas-only-local | =============================================================
cas-only-local | 
cas-only-local | java.lang.NullPointerException: null
cas-only-local | at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.listRemoteCachePeers(RMISynchronousCacheReplicator.java:335) ~[ehcache-2.10.6.jar!/:2.10.6]
cas-only-local | at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:145) ~[ehcache-2.10.6.jar!/:2.10.6]
cas-only-local | at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementPut(R

Does somebody have any luck with Hazelcast or Ehcache on similar env?

Maybe somebody uses JPA Ticket Registry? Documentation says it's fairly unnecessary and complicated process. All CAS instances in my env. use the same DB.

[Andy Ng]

Hi Maksim,

Pretty sure:

cas.ticket.registry.hazelcast.cluster.public-address 

and 

cas.ticket.registry.hazelcast.cluster.publicAddress 

Both works the same, since spring property allows both camelCase and kebak-case.

And I did successfully use docker CAS and use Hazelcast  as ticketing system, however I am using it for demo so I just included a whole bunch of private IP so it works......

here's my CAS properties:

cas.ticket.registry.hazelcast.cluster.members=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5,172.20.0.6,172.20.0.7,172.20.0.8,172.20.0.9,172.20.0.10

cas.ticket.registry.hazelcast.cluster.instanceName=localhost

my project link as well so you can reference if want to: https://github.com/NgSekLong/SelectUrCAS/blob/master/source/ticket-registry/hazelcast/cas.yml

Cheers!

- Andy

[Maksim Kopeyka]

Hi Andy,

Your example is very helpful. Thank you.

I see how hazelcast tickets registry works on my local env. I turned off active container and another container continues to work with my active session without relogin.

However on my remote env. with CAS 6.1.3 it doesn't work in this way. Load balancer ask me to re-login If I turn off active container.

Each node with hazelcast sees other nodes. I see such messages on all nodes in case node1 is turned off:

WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java.net.SocketException {Connection refused to address node1/xx.xx.xx.xx:5701}, Error-Count: 5>
WARN [com.hazelcast.internal.cluster.impl.MembershipManager] - <[node2]:5701 [dev] [3.12.4] Member [node1]:5701 - b1fba639-dfff-4536-b5f4-a8681920594d is suspected to be dead for reason: No connection>
WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java.net.SocketException {Connection refused to address node1/xx.xx.xx.xx:5701}, Error-Count: 6>
WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java.net.SocketException {Connection refused to address node1/xx.xx.xx.xx:5701}, Error-Count: 7>
WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java.net.SocketException {Connection refused to address node1/xx.xx.xx.xx:57001}, Error-Count: 8>

This is my hazelcast settings:

cas.ticket.registry.hazelcast.cluster.members=node1:5701,node2:5701,node3:5701,node3:5701
cas.ticket.registry.hazelcast.cluster.asyncBackupCount=3
cas.ticket.registry.hazelcast.cluster.port=5701
cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false
cas.ticket.registry.hazelcast.cluster.instanceName=localhost
cas.ticket.registry.hazelcast.cluster.publicAddress=node1:5701
cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true

Why hazelcast doesn't share data across the cluster?

These messages I see on startup

WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] Could not find a matching address to start with! Picking one of non-loopback addresses.>
INFO [org.apereo.cas.util.CoreTicketUtils] - <Ticket registry encryption/signing is turned off. This MAY NOT be safe in a clustered production environment. Consider using other choices to handle encryption, signing and verification of ticket registry tickets, and verify the chosen ticket registry does support this behavior.>

[Ray Bon]

Maksim,

Hazelcast is distributed but not replicated. Thus, when a server goes down, the tickets on that server are lost. You have to relogin only if your ticket was on that server.

Hazelcast has some mechanism of determining which node has which ticket. It may also be possible to make hazelcast replicated but I have not tried.

Ray

P.S. you have two node3s in your config.

-- 

Ray Bon

Programmer Analyst

Development Services, University Systems

2507218831 | CLE 019 | rb...@uvic.ca

I respectfully acknowledge that my place of work is located within the ancestral, traditional and unceded territory of the Songhees, Esquimalt and WSÁNEĆ Nations.

[Maksim Kopeyka]

Hi Ray,

Seems to me Hazelcast doesn't distribute data across all nodes because each node doesn't have information about sessions on other nodes.

How to check data distribution?

[Ray Bon]

Maksim,

I do not know if there is a stand alone client for accessing hazelcast data. In the docs, https://docs.hazelcast.org/docs/4.0/manual/html-single/index.html, there is a section on clients and one on management. It looks like you would have to create an application yourself but someone has probably done that already.

Ray

[Maksim Kopeyka]

Ray,

I asked about CAS functionality to distribute tickets across nodes. I need specific CAS functionality based on Hazelcast and seems to me this functionality doesn't work as expected so I need to check it somehow. Maybe with some debug logging.

I have a cluster with several nodes of CAS with hazelcast ticket registry and I have a load balancer. If I turn off sticky sessions CAS asks me about username/password every time. This cluster works in the same way without hazelcast ticket registry. So seems to me this functionality doesn't work and I don't see any errors in the logs.

[Ray Bon]

Maksim,

There is this config setting

cas.ticket.registry.hazelcast.cluster.members=

Add IPs of all members to the list. https://apereo.github.io/cas/6.1.x/configuration/Configuration-Properties.html#hazelcast-ticket-registry and the link to common settings.

There are some hazelcast loggers in log4j2.xml

        <AsyncLogger name="com.hazelcast" level="${sys:hazelcast.log.level}" includeLocation="true" />

Ray

[David Curry]

Maksim,

If you don't want to ever lose tickets, then you would want all nodes to back up all other nodes. So if you have 3 member nodes, you would want 2 async backup nodes (asyncBackupCount) and also you'd probably want to disable the default sync backup (backupCount) node since it will block. Here are the settings we're running with in production (although this is CAS 5):

cas.ticket.registry.hazelcast.cluster.members:          cas01.newschool.edu,cas02.newschool.edu,cas03.newschool.edu,cas04.newschool.edu,cas05.newschool.edu
cas.ticket.registry.hazelcast.cluster.asyncBackupCount: 4
cas.ticket.registry.hazelcast.cluster.backupCount:      0
cas.ticket.registry.hazelcast.cluster.port:             5701
cas.ticket.registry.hazelcast.cluster.portAutoIncrement:        false
cas.ticket.registry.hazelcast.crypto.encryption.key:    xxxIoXN6SBU5bF+iAVTKgw==
cas.ticket.registry.hazelcast.crypto.signing.key:       xxxmEbPGT_MXg0JWYLTe4oFaOaklocCqlY2VuHBdAHuh0V6-PdQxmgi4tTA3CZZos8TUbzg-L9nYHJpA5RqcvA
cas.ticket.registry.hazelcast.crypto.enabled:           true

This works well for us behind an F5 load balancer; we do not use sticky sessions. We can (and do) reboot servers in the pool without anyone getting re-prompted to log in (just don't reboot them all at once).

The crypto stuff (last three lines) is not needed for this to work, but you (arguably) might want it in production. You can leave it off while getting things to work and enable it later.

One other thing -- did you remember to open 5701 in the firewall on all the servers?

--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/cfe74ef4f3aeb0c1dfc506a7257e659418e41f0e.camel%40uvic.ca.

[Maksim Kopeyka]

Hi David,

I tried similar config with 4 nodes:

cas.ticket.registry.hazelcast.cluster.members=${HAZELCAST_CLUSTER_MEMBERS}
cas.ticket.registry.hazelcast.cluster.asyncBackupCount=4
cas.ticket.registry.hazelcast.cluster.backupCount=0
cas.ticket.registry.hazelcast.cluster.port=5701
cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false
cas.ticket.registry.hazelcast.cluster.instanceName=localhost
cas.ticket.registry.hazelcast.cluster.publicAddress=${HAZELCAST_PUBLIC_ADDRESS}
cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true
cas.ticket.registry.hazelcast.crypto.enabled=false

I see this output on each node i.e. hazelcast creates a cluster and sees all nodes:

2020-02-06 21:20:49,235 INFO [com.hazelcast.internal.cluster.ClusterService] - <[ecdc-rant-affiliateidp-dev-1]:5701 [dev] [3.12.4]
Members {size:4, ver:4} [
        Member [wcdc-rant-affiliateidp-dev-1]:5701 - a245c93b-beb0-4929-b831-e40a323cad8b
        Member [ecdc-rant-affiliateidp-dev-2]:5701 - bcbcd799-8cb8-4e5d-8802-5d95d4015ffd
        Member [wcdc-rant-affiliateidp-dev-2]:5701 - 9d3f52c9-1475-462e-844a-1b534efdca73
        Member [ecdc-rant-affiliateidp-dev-1]:5701 - e9f81f52-7a99-4428-a402-5a2f48cba838 this
]
>

However tickets distribution doesn't work. Nodes 1, 2, 3 don't know about session on Node 4.

I don't see any errors in the logs related to hazelcast but this one appears time to time

2020-02-06 17:31:56,248 ERROR [org.apereo.cas.web.flow.executor.EncryptedTranscoder] - <Null input buffer>
java.lang.IllegalArgumentException: Null input buffer
        at javax.crypto.Cipher.doFinal(Unknown Source) ~[?:?]
        at org.apereo.cas.util.cipher.BaseBinaryCipherExecutor.decode(BaseBinaryCipherExecutor.java:92) ~[cas-server-core-util-api-6.1.3.jar!/:6.1.3]

To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

[David Curry]

I believe, if you have 4 members, that asyncBackupCount should be 3. Because a node doesn't back itself up. (Hazelcast might be smart enough to fix that itself, but I don't know.)

Also, I'm curious as to why you have tcpIpEnabled set to true? You'd be much better off, from a performance standpoint, setting it to false and using UDP. TCP blocks, UDP doesn't.

I've never run this in docker, so I don't know what, if anything, you need to do differently. But I would be at least a little suspicious of the interactions there -- it should work of course, but could you be missing something in the docker config, or the config on the host?

--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/8d82a0af-84cb-4e9e-96e9-7e1e688eba1d%40apereo.org.

[Maksim Kopeyka]

I used asyncBackupCount=3 before and behavior was the same.

Regarding UDP I may try to use it but it should work with tcpIpEnabled too so it's not an issue.

I guess this problem is related to docker environment somehow. I don't have an access to configuration of VM. Probably I have to expose some additional port(s) on docker. I found thread about dockerized env of CAS with hazelcast in this group but this env is based on docker swarm but my env doesn't support docker swarm.

I saw this property in the documentation of CAS 

# ${configurationKey}.cluster.outboundPorts[0]=45000

But container stops with exception if I add it to properties file:

Binding to target [Bindable@2dd2e270 type = org.apereo.cas.configuration.CasConfigurationProperties, value = 'provided', annotations = array<Annotation>[@org.springframework.boot.context.properties.ConfigurationProperties(ignoreInvalidFields=false, ignoreUnknownFields=false, prefix=cas, value=cas)]] failed:
    Property: cas.ticket.registry.hazelcast.cluster.outboundports[0]
    Value: 33000-33100
    Origin: class path resource [application-dev.properties]:82:56
    Reason: The elements [cas.ticket.registry.hazelcast.cluster.outboundports[0]] were left unbound.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/8d82a0af-84cb-4e9e-96e9-7e1e688eba1d%40apereo.org.