[cas-user] CAS 6.3.4 Hazelcast 4.1 Issue.

['Mallory, Erik' via CAS Community]

All I'm having a problem implementing the Hazelcast ticket store in CAS
6.3.4 which uses hazelcast-4.1

Currently I'm testing with a two node cluster fontended with a
netscaler. Each node has it's own /etc/cas/config/cas.properties so
each node as it's own hazelcast configuration.
Here is the relevant hazelcast configuration parameters:
cas.ticket.registry.hazelcast.page-size=500
cas.ticket.registry.hazelcast.cluster.tcpip-enabled=true
cas.ticket.registry.hazelcast.cluster.map-merge-policy=PUT_IF_ABSENT
cas.ticket.registry.hazelcast.cluster.instance-name=cas-dev
cas.ticket.registry.hazelcast.cluster.members=10.0.79.38,10.0.79.37
cas.ticket.registry.hazelcast.cluster.eviction-policy=LRU
cas.ticket.registry.hazelcast.cluster.max-no-heartbeat-seconds=300
cas.ticket.registry.hazelcast.cluster.logging-type=slf4j
cas.ticket.registry.hazelcast.cluster.port=5701
cas.ticket.registry.hazelcast.cluster.max-size=85
cas.ticket.registry.hazelcast.cluster.backup-count=1
cas.ticket.registry.hazelcast.cluster.async-backup-count=0
cas.ticket.registry.hazelcast.cluster.max-size-
policy=USED_HEAP_PERCENTAGE
cas.ticket.registry.hazelcast.cluster.timeout=5

IN my testing I found that the tickets were not being replicated the
other host. I'd use the netscaler to switch between the backend CAS
nodes, log in to one, fail over to the other node and attempt to access
cas, and I was redirected to the login screen.

After restarting the cas services on both nodes and tailing out the cas
log I noticed the following error:

Cannot add a dynamic configuration

'MapConfig{name='serviceTicketsCache', inMemoryFormat=BINARY',
metadataPolicy=CREATE_ON_
UPDATE, backupCount=1, asyncBackupCount=0, timeToLiveSeconds=0,
maxIdleSeconds=500, readBackupData=false, evictionConfig=Evict
ionConfig{size=85, maxSizePolicy=USED_HEAP_PERCENTAGE,
evictionPolicy=LRU, comparatorClassName=null, comparator=null}, merkleT
ree=MerkleTreeConfig{enabled=false, depth=10},
eventJournal=EventJournalConfig{enabled=false, capacity=10000,
timeToLiveSecond
s=0}, hotRestart=HotRestartConfig{enabled=false, fsync=false},
nearCacheConfig=null, mapStoreConfig=MapStoreConfig{enabled=fal
se, className='null', factoryClassName='null', writeDelaySeconds=0,
writeBatchSize=1, implementation=null, factoryImplementation=null,
properties={}, initialLoadMode=LAZY, writeCoalescing=true},
mergePolicyConfig=MergePolicyConfig{policy='com.hazelcast.spi.merge.Lat
estUpdateMergePolicy', batchSize=100}, wanReplicationRef=null,
entryListenerConfigs=null, indexConfigs=null, attributeConfigs=null,
splitBrainProtectionName=null, queryCacheConfigs=null,
cacheDeserializedValues=INDEX_ONLY}'

as there is already a conflicting configuration

'MapConfig{name='serviceTicketsCache', inMemoryFormat=BINARY',
metadataPolicy=CREATE_ON_UPDATE, backupCount=1, asyncBackupCount=0,
timeToLiveSeconds=0, maxIdleSeconds=10, readBackupData=false,
evictionConfig=EvictionConfig{size=85,
maxSizePolicy=USED_HEAP_PERCENTAGE, evictionPolicy=LRU,
comparatorClassName=null, comparator=null},
merkleTree=MerkleTreeConfig{enabled=false, depth=10},
eventJournal=EventJournalConfig{enabled=false, capacity=10000,
timeToLiveSeconds=0}, hotRestart=HotRestartConfig{enabled=false,
fsync=false}, nearCacheConfig=null,
mapStoreConfig=MapStoreConfig{enabled=false, className='null',
factoryClassName='null', writeDelaySeconds=0, writeBatchSize=1,
implementation=null, factoryImplementation=null, properties={},
initialLoadMode=LAZY, writeCoalescing=true},
mergePolicyConfig=MergePolicyConfig{policy='com.hazelcast.spi.merge.Lat
estUpdateMergePolicy', batchSize=100}, wanReplicationRef=null,
entryListenerConfigs=null, indexConfigs=null, attributeConfigs=null,
splitBrainProtectionName=null, queryCacheConfigs=null,
cacheDeserializedValues=INDEX_ONLY}'>

So off to google I go and I find
https://github.com/hazelcast/hazelcast/issues/12222
and I add -Dhazelcast.dynamicconfig.ignore.conflicts=true for giggles
and to see something at least boot.

So now both services start up but I'm ignoring the dynamic config
conflicts. My testing fails it would appear that hazelcast is not able
to share the tgt between nodes.

Any help would be greatly appreciated.


--
Erik Mallory
Server Analyst
Wichita State University

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/eb34bc51bfd8f5db71a2b000f1b362491c243cbd.camel%40wichita.edu.

[Ray Bon]

Erik,

Hazelcast is not a replicated cache by default, just distributed. I understand there is a backup/restore system but you would need at least three servers to test it.

The only config I have are these:

cluster.members

cluster.instanceName

crypto.signing.key

crypto.encryption.key

crypto.enabled=true

which is in a shared file.

I have two cas'es running on my local and have not seen that error. You can tell if the hazelcast servers are communicating if your tickets are validated on a different server than they were created. Set the load balancer to round robin.

I do not see a cas property for replication. Is it an option for cas? See hazelcast docs, https://docs.hazelcast.com/imdg/4.1/data-structures/replicated-map.html

Ray

On Thu, 2021-07-01 at 19:41 +0000, 'Mallory, Erik' via CAS Community wrote:

Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/7f06f02da2b65ccfeb8f8bb1868ffd23eac32ef1.camel%40uvic.ca.

[Baron Fujimoto]

I'm also confused by this. What does distributed mean, if not replicated? We have Hazelcast enabled in our current 5.0 deployment. It definitely replicates the TGTs across the nodes. We rely on this for high availability (HA) of our cas service in case one of the cluster nodes goes down.

This TGT replication does not appear to be occurring with 6.3.5. How can we enable the replication of TGTs  among nodes?

We've currently configured Hazelcast for 6.3 with the following dependency in build.gradle:

    implementation "org.apereo.cas:cas-server-support-hazelcast-ticket-registry:${casServerVersion}"

And in cas.properties:

cluster.members
cluster.instanceName

We're not using crypto on the keys. The following is logged when CAS starts up:

WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [4.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [4.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.>
WARN [com.hazelcast.cp.CPSubsystem] - <[cas.example.edu]:5701 [dev] [4.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.>

The DNS warnings I understand and accept. I don't know what the CP subsystem is nor what, if anything, to do about it. We also get subsequent logs about the lack of crypto on the tickets, but that also doesn't seem relevant to the lack of TGT replication among nodes.

--

Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum desendus pantorum

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAAjLUL3oSJ9Ho%3Dij4sFdr6E8%2BShxYRLf8oieSLgJEd-YZLiU8A%40mail.gmail.com.

[Mark H. Wood]

On Wed, Jul 07, 2021 at 02:49:32PM -1000, Baron Fujimoto wrote:
> I'm also confused by this. What does distributed mean, if not replicated?

I understand "replicated" to mean that each associated instance
contains the complete set of cache entries locally. There would be a
great deal of communication required to maintain consistency, but the
cost of cache queries is small.

Another form of distribution is variously called "sharded,",
"partitioned," etc. Associated instances would hold subsets of the
complete cache content, and their association would mean that one
instance can ask the others if any have hits where it has a miss, and
to supply one. This isn't as fast as having a complete set in each
instance, but it reduces the communication load on insertion. It may
be appropriate where the cost of acquiring an uncached object is
sufficiently higher than the cost of asking for help from another
subset and awaiting a reply.

If the cost of uncached objects is quite high, it can also make sense
to run multiple cache instances atop a shared backing store, which has
its own cost.

> > <https://groups.google.com/a/apereo.org/d/msgid/cas-user/7f06f02da2b65ccfeb8f8bb1868ffd23eac32ef1.camel%40uvic.ca?utm_medium=email&utm_source=footer>
> > .

> >
>
>
> --
> Baron Fujimoto <ba...@hawaii.edu> :: UH Information Technology Services
> minutas cantorum, minutas balorum, minutas carboratum desendus pantorum
>
> --
> - Website: https://apereo.github.io/cas
> - Gitter Chatroom: https://gitter.im/apereo/cas
> - List Guidelines: https://goo.gl/1VRrw7
> - Contributions: https://goo.gl/mh7qDG
> ---
> You received this message because you are subscribed to the Google Groups "CAS Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
> To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAAjLUL3oSJ9Ho%3Dij4sFdr6E8%2BShxYRLf8oieSLgJEd-YZLiU8A%40mail.gmail.com.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/YOcDSh7IzCPHaVo/%40IUPUI.Edu.

[Baron Fujimoto]

Thanks for that clarification of replicated vs distributed. That was helpful. So if I understand correctly, CAS's use of Hazelcast, if distributed but not replicated, means that the loss of a node in a cluster means the remaining nodes may not actually have the desired content from the lost node.

Also, FWIW, I think I resolved our problem where we weren't seeing any of the tickets distributed to other nodes. I believe Hazelcast uses port 5701 by default for this. These were new VMs and they require each port to be explicitly added. (New host, who dis?) Seems to be working as expected now. Sorry, PEBKAC, mea culpa, etc... 

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAAjLUL28X_TKoXSipPd2gUf_aOr3R6O_WwVxrMOF2n_-5j1GNg%40mail.gmail.com.

[King, Robert]

If I had a nickel for every “is it plugged in” I have missed.  ;)

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/c063704f233f48c8ac5eb29b3edab50b%40mun.ca.

['Mallory, Erik' via CAS Community]

Thanks to all who replied. I got side tracked by other issues and when
I returned to this one I found that some services worked consistently
and one (cas-managment version 6.1.x) was problematic. So it turns out
that hazelcast was working.
Thanks for the clarification on distributed vs replicated. I had not
made the distinction.
Warm regards,

--
Erik Mallory
Server Analyst
Wichita State University

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/ca069fffb75fb1f4f0f36ee2264300633d8f6068.camel%40wichita.edu.

[Baba Ndiaye]

Thank you for your answer

I would like to know if we should configure hazelcast on each node or just one node I also want to know cas.ticket.registry.hazelcast.cluster.instanceName this will be our node name that we must put on the platforms (Moodle for example) and that behind it are the cas1.mydomain.com and cas2.mydomain.com nodes that respond.
Thank you for answering me I am a beginner on CAS Apereo

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/f8fbf1e4-0931-4f45-b7ea-0d1c54e7bacen%40apereo.org.