SYM_ENCRYPT not working on Kubernetes

[Chintan Mohan Rohila]

Hello all,

We need an urgent help for configuring a secure JGroups intra-cluster communication on our application running on Kubernetes.

Protocol stack used is:
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP_NIO2 bind_port="8800" diagnostics_port="10599"
port_range="950" recv_buf_size="${tcp.recv_buf_size:230k}"
send_buf_size="${tcp.send_buf_size:230k}" max_bundle_size="64K"
sock_conn_timeout="300" thread_pool.min_threads="0"
thread_pool.max_threads="980" thread_pool.keep_alive_time="70000" />
<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${KUBERNETES_NAMESPACE}" labels="${KUBERNETES_LABELS}" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="80000" interval="18000" timeout_check_interval="18000" />
<VERIFY_SUSPECT />
<SYM_ENCRYPT
sym_algorithm="AES"
sym_keylength="128"
        keystore_name="${jgroups.keystore_name}"
        store_password="${jgroups.store_password}"
        alias="${jgroups.alias}"/>
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<BARRIER />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="8000"
leave_timeout="18000" view_ack_collection_timeout="18000" />
<AUTH
        auth_class="org.jgroups.auth.MD5Token"
        auth_value="${jgroups.auth_value}"
        token_hash="MD5"/>    
<UFC />
<MFC />
<FRAG2 />
<pbcast.STATE_TRANSFER />
</config>

Keystore used:
keytool -genseckey -alias mykey -keypass xxxxxxxxxxxxxxx -storepass xxxxxxxxxxxxxxx -keyalg AES -keysize 128 -keystore jgroup.keystore -storetype JCEKS

With the above configuration views are not able to get all the members from all the pods and following exception is coming. Please suggest if something can be fixed in this configuration or else ASYM protocol can be used and how?

Exception:

[org.jgroups.protocols.SYM_ENCRYPT]eric-bss-ec-vs50-erxxhar-2-29765: rejected decryption of unicast message from non-member eric-bss-ec-vs50-erxxhar-1-35959

Probe output:
java -cp /opt/vs/bundles/org.jgroups_4.2.10.Final-E002.jar org.jgroups.tests.Probe -port 10599 -bind_addr 192.168.134.114 jmx=SYM_ENCRYPT

local_addr=eric-bss-ec-vs50-erxxhar-0-24353 [ip=192.168.134.114:8800, version=4.2.10.Final (Julier), cluster=TRAFFIC, 1 mbr(s)]
SYM_ENCRYPT={after_creation_hook=null, asym_algorithm=RSA, asym_keylength=2048, cipher_pool_size=8, encrypt_entire_message=false, ergonomics=true, id=58, key_map_max_size=20, keystore_name=/var/opt/vs/SecureInterface/jgroups/jgroup.keystore, keystore_type=JCEKS, level=DEBUG, provider=null, sign_msgs=false, stats=true, sym_algorithm=AES, sym_iv_length=0, sym_keylength=128, use_adler=false, version=7D1160B53765281EECA6EDEFE6436BB3}

Note: If I just remove the SYM_ENCRYPT protocol, JGroups cluster works just fine with all the completed views.

Any help on this is highly appreciated.

Best regards.

Chintan Rohila

[Bela Ban]

I get an exception with keystore jgroups.keystore and generating it the
way you described below:

Exception in thread "main" java.io.IOException:
DerInputStream.getLength(): lengthTag=78, too big.
at
java.base/sun.security.util.DerInputStream.getLength(DerInputStream.java:606)
at java.base/sun.security.util.DerValue.init(DerValue.java:390)
at java.base/sun.security.util.DerValue.<init>(DerValue.java:331)
at java.base/sun.security.util.DerValue.<init>(DerValue.java:344)
at
java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1993)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at
org.jgroups.protocols.SYM_ENCRYPT.readSecretKeyFromKeystore(SYM_ENCRYPT.java:107)
at org.jgroups.protocols.SYM_ENCRYPT.init(SYM_ENCRYPT.java:88)
at
org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:806)
at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:442)
at org.jgroups.JChannel.init(JChannel.java:894)
at org.jgroups.JChannel.<init>(JChannel.java:124)
at org.jgroups.JChannel.<init>(JChannel.java:105)
at org.jgroups.demos.Chat.start(Chat.java:34)
at org.jgroups.demos.Chat.main(Chat.java:106)


When I use
<SYM_ENCRYPT
provider="SunJCE"
sym_algorithm="AES/CBC/PKCS5Padding"
sym_iv_length="16"
keystore_name="/Users/bela/JGroups/keystore/defaultStore.keystore"
store_password="changeit" alias="myKey"/>,

this works. The keystore is generated via 'ant make-keystore'.

I used Java 11 and 21, and this fails in both cases.

On 30.10.23 17:44, Chintan Mohan Rohila wrote:
> Hello all,
>
> We need an urgent help for configuring a secure JGroups intra-cluster
> communication on our application running on Kubernetes.
>

> _*Protocol stack used is:*_

> _*Keystore used:*_

> keytool -genseckey -alias mykey -keypass xxxxxxxxxxxxxxx -storepass
> xxxxxxxxxxxxxxx -keyalg AES -keysize 128 -keystore jgroup.keystore
> -storetype JCEKS
>
> With the above configuration views are not able to get all the members
> from all the pods and following exception is coming. Please suggest if
> something can be fixed in this configuration or else ASYM protocol can
> be used and how?
>

> _*Exception:*_

> [org.jgroups.protocols.SYM_ENCRYPT]eric-bss-ec-vs50-erxxhar-2-29765:
> rejected decryption of unicast message from non-member
> eric-bss-ec-vs50-erxxhar-1-35959
>

> _*Probe output:*_

> java -cp /opt/vs/bundles/org.jgroups_4.2.10.Final-E002.jar
> org.jgroups.tests.Probe -port 10599 -bind_addr 192.168.134.114
> jmx=SYM_ENCRYPT
>
> local_addr=eric-bss-ec-vs50-erxxhar-0-24353 [ip=192.168.134.114:8800,
> version=4.2.10.Final (Julier), cluster=TRAFFIC, 1 mbr(s)]
> SYM_ENCRYPT={after_creation_hook=null, asym_algorithm=RSA,
> asym_keylength=2048, cipher_pool_size=8, encrypt_entire_message=false,
> ergonomics=true, id=58, key_map_max_size=20,
> keystore_name=/var/opt/vs/SecureInterface/jgroups/jgroup.keystore,
> keystore_type=JCEKS, level=DEBUG, provider=null, sign_msgs=false,
> stats=true, sym_algorithm=AES, sym_iv_length=0, sym_keylength=128,
> use_adler=false, version=7D1160B53765281EECA6EDEFE6436BB3}
>

> /Note: If I just remove the SYM_ENCRYPT protocol, JGroups cluster works
> just fine with all the completed views./

>
> Any help on this is highly appreciated.
>
> Best regards.
> Chintan Rohila
>

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/7fe11173-ec4d-4fb8-b2f8-3b5ee3dcc7f2n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/7fe11173-ec4d-4fb8-b2f8-3b5ee3dcc7f2n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Bela Ban | http://www.jgroups.org

[Bela Ban]

Correction: this works with Java 8 and JGroups 4.x (4.2.24).

Can you reproduce this? Do you have a sample image that I could try out?

A few things to note:

* TCP_NIO2.port_range=950? Why so big? Do you intend to run 900+
instances on the same box?

* GMS.join_timeout=8000: the first instance waits for 8s, this is not
needed as it slows down startup

* AUTH: this is not needed and MD5Token was removed in 5.x

* TCP_NIO2.thread_pool.max_threads="980"? Why so big?


The error message is caused by the cluster not forming. The non-member
has a different sym_version and is not in the membership, and so we
reject the message from it.

I suggest (if you can reproduce this) setting TRACE level for KUBE_PING,
SYM_ENCRYPT and GMS, and posting the logs for the 2 members.

[Chintan Mohan Rohila]

Hi Bela,

Do you have the keytool cmd to create the "/Users/bela/JGroups/keystore/defaultStore.keystore" that you have used in your following working example?

<SYM_ENCRYPT
provider="SunJCE"
sym_algorithm="AES/CBC/PKCS5Padding"
sym_iv_length="16"
keystore_name="/Users/bela/JGroups/keystore/defaultStore.keystore"
store_password="changeit" alias="myKey"/>,

[Bela Ban]

Yes, use 'ant make-keystore'. It uses class

org.jgroups.demos.KeyStoreGenerator

> [ip=192.168.134.114:8800 <http://192.168.134.114:8800>,

> an email to jgroups-dev...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/c0ed424d-b03f-4f1d-af5c-ae010bb972dfn%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/c0ed424d-b03f-4f1d-af5c-ae010bb972dfn%40googlegroups.com?utm_medium=email&utm_source=footer>.