Gerrit deployment with Operator fails

[Igors Sersts]

Hi all,
I am trying to deploy Gerrit with Operator on AKS (private cluster). What I try to achieve:
Deploy Gerrit (primary) as 1 pod
No replica instance
No optional plugins, ingress, refdb, fluentBit
Use as much default values as possible.
What I did:
I have installed gerrit-operator (https://gerrit.googlesource.com/k8s-gerrit/+/HEAD/Documentation/operator.md#gerrit_operator):
% kubectl get crd | grep gerrit
gerritclusters.gerritoperator.google.com 2024-07-12T12:46:56Z
gerritnetworks.gerritoperator.google.com 2024-07-12T12:46:55Z
gerrits.gerritoperator.google.com 2024-07-12T12:46:56Z
gitgcs.gerritoperator.google.com 2024-07-12T12:46:55Z
incomingreplicationtasks.gerritoperator.google.com 2024-07-12T12:46:55Z
receivers.gerritoperator.google.com 2024-07-12T12:46:56Z
Then I have applyed gerritcluster.yaml (check the attachment):
% kubectl apply -f gerritcluster.yaml
gerritcluster.gerritoperator.google.com/gerrit created
Kubernetes resources are created succesfully:
% kubectl get all -n gerrit-poc-v3-operator
NAME READY STATUS RESTARTS AGE
pod/gerrit-v3-0 0/1 Running 13 (12s ago) 29m
pod/github-28688280-hb7kv 0/1 CrashLoopBackOff 4 (45s ago) 2m20s
pod/receiver-84685cbf78-xkc8h 1/1 Running 0 29m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/gerrit-v3 ClusterIP None <none> 80/TCP,29418/TCP 29m
service/gerrit-v3-service ClusterIP 10.0.107.211 <none> 80/TCP,29418/TCP 29m
service/receiver ClusterIP 10.0.166.205 <none> 80/TCP 29m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/receiver 1/1 1 1 29m
NAME DESIRED CURRENT READY AGE
replicaset.apps/receiver-84685cbf78 1 1 1 29m
NAME READY AGE
statefulset.apps/gerrit-v3 0/1 29m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/github 0 * * * * False 1 2m21s 29m
NAME COMPLETIONS DURATION AGE
job.batch/github-28688280 0/1 2m21s 2m21s
gerrit-init initialized successfully:
[2024-07-18 09:33:29,040] INFO Requiring plugins (ClusterMode: HIGH_AVAILABILITY): ['healthcheck']
[2024-07-18 09:33:29,040] INFO Requiring libs (ClusterMode: HIGH_AVAILABILITY): []
[2024-07-18 09:33:29,044] INFO Created plugin installation directory: /var/gerrit/plugins
[2024-07-18 09:33:29,045] INFO Created lib installation directory: /var/gerrit/lib
[2024-07-18 09:33:29,048] INFO Installing plugin healthcheck from container to /var/gerrit/plugins/healthcheck.jar.
[2024-07-18 09:33:29,048] INFO Installing packaged plugin delete-project.
[2024-07-18 09:33:29,053] INFO Gerrit is not yet installed. Initializing new site.
[2024-07-18 09:33:29,053] INFO Existing gerrit.config found.
Auto-configured "receive.autogc = false" to disable auto-gc after git-receive-pack.
Generating SSH host key ... rsa... ed25519... ecdsa 256... ecdsa 384... ecdsa 521... done
Initialized /var/gerrit

Collecting accounts: 1
Collecting accounts: 1

Reindexing accounts: 100% (1/1)
Reindexing accounts: 100% (1/1)

Reindexing groups: 50% (1/2)
Reindexing groups: 100% (2/2)
Reindexing groups: 100% (2/2)

Reindexing changes: Slicing projects: 50% (1/2)
Reindexing changes: Slicing projects: 100% (2/2)
Reindexing changes: Slicing projects: 100% (2/2), done

Reindexing projects: 50% (1/2)
Reindexing projects: 100% (2/2)
Reindexing projects: 100% (2/2)
[2024-07-18 09:33:42,077] INFO Skipping reindexing.

But gerrit it self is throwing error that it can not locate SSH keys:

[2024-07-18T10:13:32.439Z] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
      \_ for 6th parameter hostKeyProvider
  at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
      \_ for 1st parameter authenticator
  at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
  while locating CachingPublicKeyAuthenticator
  at SshDaemon.<init>(SshDaemon.java:168)
      \_ for 3rd parameter userAuth
  at SshDaemon.class(SshDaemon.java:140)
  while locating SshDaemon
  at SshModule.configure(SshModule.java:76)
  while locating SshInfo

2) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
      \_ for 6th parameter hostKeyProvider
  at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
      \_ for 1st parameter authenticator
  at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
  while locating CachingPublicKeyAuthenticator
  while locating PublickeyAuthenticator

3) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
      \_ for 6th parameter hostKeyProvider
  at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
      \_ for 1st parameter authenticator
  at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
  while locating CachingPublicKeyAuthenticator
  at SshDaemon.<init>(SshDaemon.java:168)
      \_ for 3rd parameter userAuth
  at SshDaemon.class(SshDaemon.java:140)
  while locating SshDaemon
  while locating LifecycleListener annotated with @UniqueAnnotations$Internal(71)

4) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  while locating KeyPairProvider

5) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
      \_ for 6th parameter hostKeyProvider
  at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
      \_ for 1st parameter authenticator
  at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
  while locating CachingPublicKeyAuthenticator

6) No SSH keys under /var/gerrit/etc
  while locating HostKeyProvider
  at SshHostKeyModule.configure(SshHostKeyModule.java:25)
  at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
      \_ for 6th parameter hostKeyProvider
  at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
      \_ for 1st parameter authenticator
  at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
  while locating CachingPublicKeyAuthenticator
  at SshDaemon.<init>(SshDaemon.java:168)
      \_ for 3rd parameter userAuth
  at SshDaemon.class(SshDaemon.java:140)
  while locating SshDaemon

6 errors

======================
Full classname legend:
======================
CachingPublicKeyAuthenticator: "com.google.gerrit.sshd.CachingPublicKeyAuthenticator"
DatabasePubKeyAuth:            "com.google.gerrit.sshd.DatabasePubKeyAuth"
HostKeyProvider:               "com.google.gerrit.sshd.HostKeyProvider"
KeyPairProvider:               "org.apache.sshd.common.keyprovider.KeyPairProvider"
LifecycleListener:             "com.google.gerrit.extensions.events.LifecycleListener"
PublickeyAuthenticator:        "org.apache.sshd.server.auth.pubkey.PublickeyAuthenticator"
SshDaemon:                     "com.google.gerrit.sshd.SshDaemon"

Any ideas what I am missing here? Thanks!

[Igors Sersts]

This discussion did not appear in the list, so I thought I hadn't posted it and created a new one. This discussion can be deleted.