outdated certs,

[Carlos Pasqualini]

Hi!

I forgot to renew my cluster's certificates, and found that they had been outdated on Sept 25...

Our cluster is using debian's bullseye system on all (4 four) nodes

I tried as documentation, with gnt-cluster cert-renew ... seems without success
(fails because 1811 needs valid handshake, will fail because i don't have any valid certs)

then tried manually, with openssl, more or less same result

On ganeti-luxid, strated using:

sudo -u gnt-masterd ganeti-luxid --no-voting --yes-do-it -f -d

each servers prints an error that ends with:
CurlLayerError "code: CurlSSLCACert, explanation: SSL certificate problem: self signed certificate"


Do we have a documented way of regenerating all certs manually / by SSH?


what I tried mannually so far:

# on Master node (or ate least, on the node that used to be the master)

CA_PATH="/var/lib/ganeti"
MASTER_FQDN="a.example.com"
NODES=("b.example.com" "c..example.com" "d.example.com")
CLUSTER_NAME="ganeti.example.com"
DAYS_VALID=3650  # 10 year
SERIAL_MASTER=1    # (client.pem)
SERIAL_RAPI=2      # (rapi.pem)
SERIAL_SPICE=3     # (spice.pem)

cd $CA_PATH
openssl genrsa -out server.key 2048
openssl req -new -x509 -days $DAYS_VALID -key server.key -out server.cert -subj "/CN=$CLUSTER_NAME/OU=Ganeti Cluster Root CA"
cat server.key server.cert > server.pem
cp server.cert ca.pem

openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=$MASTER_FQDN/OU=Ganeti Node"
openssl x509 -req -days $DAYS_VALID -in client.csr -CA server.pem -CAkey server.key -set_serial $SERIAL_MASTER -out client.cert
cat client.key client.cert > client.pem

openssl genrsa -out rapi.key 2048
openssl req -new -key rapi.key -out rapi.csr -subj "/CN=$MASTER_FQDN/OU=Ganeti RAPI"
openssl x509 -req -days $DAYS_VALID -in rapi.csr -CA server.pem -CAkey server.key -set_serial $SERIAL_RAPI -out rapi.cert
cat rapi.key rapi.cert > rapi.pem

openssl genrsa -out spice.key 2048
openssl req -new -key spice.key -out spice.csr -subj "/CN=Ganeti Spice/OU=Spice"
openssl x509 -req -days $DAYS_VALID -in spice.csr -CA server.pem -CAkey server.key -set_serial $SERIAL_SPICE -out spice.cert
cat spice.key spice.cert > spice.pem
cp ca.pem spice-ca.pem

/etc/init.d/ganeti start

#####################
# Nodes
START_SERIAL=10

for NODE_FQDN in "${NODES[@]}"; do
SERIAL=$START_SERIAL
ssh root@$NODE_FQDN "systemctl stop ganeti; killall ganeti-noded ganeti-confd ganeti-mond 2>/dev/null"
ssh root@$NODE_FQDN bash << EOF
cd $CA_PATH
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=$NODE_FQDN/OU=Ganeti Node"
EOF

scp root@$NODE_FQDN:$CA_PATH/client.csr /tmp/$NODE_FQDN.csr
openssl x509 -req -days 3650 -in /tmp/$NODE_FQDN.csr -CA $CA_PATH/server.pem -CAkey $CA_PATH/server.key -set_serial $SERIAL -out /tmp/$NODE_FQDN.cert
scp /tmp/$NODE_FQDN.cert root@$NODE_FQDN:$CA_PATH/
scp $CA_PATH/ca.pem root@$NODE_FQDN:$CA_PATH/
ssh root@$NODE_FQDN bash << EOF
cd $CA_PATH
cat client.key $NODE_FQDN.cert > client.pem
rm $NODE_FQDN.cert client.csr
EOF

ssh root@$NODE_FQDN "systemctl start ganeti"
START_SERIAL=$((START_SERIAL + 1))
done

seems to be working the execution, but it does not work
my test is, from master node:

echo | openssl s_client -connect b.example.com:1811 -state

and responds:

CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
SSL_connect:TLSv1.3 read encrypted extensions
SSL_connect:SSLv3/TLS read server certificate request
depth=0 CN = ganeti.example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = ganeti.example.com
verify error:num=26:unsupported certificate purpose
verify return:1
depth=0 CN = ganeti.example.com
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:TLSv1.3 read server certificate verify
SSL_connect:SSLv3/TLS read finished
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write finished
---
Certificate chain
 0 s:CN = ganeti.example.com
   i:CN = ganeti.example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
[a server certificate here, but not matching client.pem]
-----END CERTIFICATE-----
subject=CN = ganeti.example.com

issuer=CN = ganeti.example.com

---
Acceptable client certificate CA names
CN = ganeti.example.com
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1480 bytes and written 431 bytes
Verification error: unsupported certificate purpose
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 26 (unsupported certificate purpose)
---
SSL3 alert read:fatal:unknown
140702110356800:error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:../ssl/record/rec_layer_s3.c:1571:SSL alert number 116

so I don't understand from where it is loading the certificate, or if it is something wrong on my steps.

Any help will be welcome, specific documentation on restore procedure would be awesome

thanks!

[Sascha Lucas]

Hi Carlos,

On Mon, 6 Oct 2025, Carlos Pasqualini wrote:

> I forgot to renew my cluster's certificates, and found that they had been
> outdated on Sept 25...
>
> Our cluster is using debian's bullseye system on all (4 four) nodes

That should be Ganeti-3.0.2, right?

> I tried as documentation, with gnt-cluster cert-renew ... seems without
> success
> (fails because 1811 needs valid handshake, will fail because i don't have
> any valid certs)

The man page for gnt-cluster lists the following command (not cert-renew):

$ gnt-cluster renew-crypto --new-cluster-certificate --new-node-certificates

But in the end, it first tries to talk to ganeti, before "This command
will stop all Ganeti daemons in the cluster and start them again once the
new certificates and keys are replicated".

> then tried manually, with openssl, more or less same result

There is an very old wiki regarding SSL[1], mentioning even without SSL.

Please not the typo -> ssconf_master_candidate_certs should be
ssconf_master_candidates_certs.

I quickly tested running without SSL, which seems to work far enough to
run "gnt-cluster renew-crypto".

There is also the "faketime" package, which you can use to turn back time
selectively for the ganeti daemons. In /etc/init.d/ganeti, start_action()
add:

faketime -f -30d $DAEMON_UTIL start "$@"

HTH, Sascha.

[1] https://github.com/ganeti/ganeti/wiki/Ganeti-and-SSL#run-ganeti-without-client-certificates