Gerrit Operator deployment - No SSH keys under /var/gerrit/etc

[Igors Sersts]

Hi all,

Trying to deploy minimal Gerrit setup with Gerrit Operator - no optional plugins, ingress, refDb, fluentBit.

Folowing this docs installed operator:

% kubectl get crd | grep gerrit
gerritclusters.gerritoperator.google.com                     2024-07-12T12:46:56Z
gerritnetworks.gerritoperator.google.com                     2024-07-12T12:46:55Z
gerrits.gerritoperator.google.com                            2024-07-12T12:46:56Z
gitgcs.gerritoperator.google.com                             2024-07-12T12:46:55Z
incomingreplicationtasks.gerritoperator.google.com           2024-07-12T12:46:55Z
receivers.gerritoperator.google.com                          2024-07-12T12:46:56Z

Applied gerritcluster resource defined as following:

apiVersion: "gerritoperator.google.com/v1beta10"

kind: GerritCluster

metadata:

name: gerrit

namespace: gerrit-poc-v3-operator

spec:

containerImages:

imagePullSecrets: []

imagePullPolicy: Always

gerritImages:

registry: docker.io

org: k8sgerrit

tag: latest

busyBox:

registry: docker.io

tag: latest

storage:

storageClasses:

readWriteOnce: default

readWriteMany: cluster-nfs-2

nfsWorkaround:

enabled: false

chownOnStartup: false

idmapdConfig: |-

[General]

Verbosity = 0

Domain = localdomain.com

[Mapping]

Nobody-User = nobody

Nobody-Group = nogroup

sharedStorage:

externalPVC:

enabled: false

claimName: ""

size: 10Gi

volumeName: ""

# selector:

# matchLabels:

# volume-type: ssd

# aws-availability-zone: us-east-1

pluginCache:

enabled: true

ingress:

enabled: false

host: example.com

annotations: {}

tls:

enabled: false

secret: ""

ambassador:

id: []

createHost: false

istio:

gatewaySelector:

istio: ingressgateway

refdb:

database: NONE

spanner:

projectName: ""

instance: ""

database: ""

zookeeper:

connectString: ""

rootNode: ""

fluentBitSidecar:

enabled: false

image: fluent/fluent-bit:latest

config: |-

[OUTPUT]

Name stdout

Match *

[FILTER]

Name modify

Match *

Add k8s.pod.name ${POD_NAME}

serverId: "gerrit-1"

gerrits:

- metadata:

name: gerrit-v3

labels:

app: gerrit-v3

spec:

serviceAccount: gerrit

tolerations: []

# - key: key1

# operator: Equal

# value: value1

# effect: NoSchedule

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: storagetier

operator: In

values:

- Premium_LRS

topologySpreadConstraints: []

# - maxSkew: 1

# topologyKey: zone

# whenUnsatisfiable: DoNotSchedule

# labelSelector:

# matchLabels:

# foo: bar

priorityClassName: ""

replicas: 1

updatePartition: 0

resources:

requests:

cpu: 1

memory: 1Gi

limits:

cpu: 1

memory: 2Gi

startupProbe:

initialDelaySeconds: 0

periodSeconds: 10

timeoutSeconds: 1

successThreshold: 1

failureThreshold: 3

readinessProbe:

initialDelaySeconds: 0

periodSeconds: 10

timeoutSeconds: 1

successThreshold: 1

failureThreshold: 3

livenessProbe:

initialDelaySeconds: 0

periodSeconds: 10

timeoutSeconds: 1

successThreshold: 1

failureThreshold: 3

gracefulStopTimeout: 30

service:

# type: NodePort

type: ClusterIP

httpPort: 80

sshPort: 29418

mode: PRIMARY

debug:

enabled: true

suspend: false

site:

size: 10Gi

plugins: []

# Installs a packaged plugin

# - name: delete-project

# Plugin with data files

# - name: hooks

# data:

# secretRef: hooks

# executable: false

# Downloads and installs a plugin

# - name: javamelody

# url: https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.6/job/plugin-javamelody-bazel-master-stable-3.6/lastSuccessfulBuild/artifact/bazel-bin/plugins/javamelody/javamelody.jar

# sha1: 40ffcd00263171e373a24eb6a311791b2924707c

# If the `installAsLibrary` option is set to `true` the plugin's jar-file will

# be symlinked to the lib directory and thus installed as a library as well.

# - name: saml

# url: https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.6/job/plugin-saml-bazel-master-stable-3.6/lastSuccessfulBuild/artifact/bazel-bin/plugins/saml/saml.jar

# sha1: 6dfe8292d46b179638586e6acf671206f4e0a88b

# installAsLibrary: true

libs: []

# - name: global-refdb

# url: https://example.com/global-refdb.jar

# sha1: 3d533a536b0d4e0184f824478c24bc8dfe896d06

configFiles:

gerrit.config: |-

[gerrit]

serverId = gerrit-1

disableReverseDnsLookup = true

[index]

type = LUCENE

[auth]

type = DEVELOPMENT_BECOME_ANY_ACCOUNT

[httpd]

requestLog = true

gracefulStopTimeout = 1m

[transfer]

timeout = 120 s

[user]

name = Gerrit Code Review

email = ger...@example.com

anonymousCoward = Unnamed User

[container]

javaOptions = -Xms200m

javaOptions = -Xmx2g

# envVars:

# - name: GOOGLE_APPLICATION_CREDENTIALS

# value: /var/gerrit/etc/gcp-credentials.json

secretRef: gerrit-secure-config

# secretRef: ""

receiver:

metadata:

name: receiver

labels:

app: receiver

spec:

tolerations: []

# - key: key1

# operator: Equal

# value: value2

# effect: NoSchedule

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: storagetier

operator: In

values:

- Premium_LRS

topologySpreadConstraints: []

# - maxSkew: 1

# topologyKey: zone

# whenUnsatisfiable: DoNotSchedule

# labelSelector:

# matchLabels:

# foo: bar

priorityClassName: ""

replicas: 1

maxSurge: 1

maxUnavailable: 1

resources:

requests:

cpu: 1

memory: 1Gi

limits:

cpu: 1

memory: 2Gi

readinessProbe:

initialDelaySeconds: 0

periodSeconds: 10

timeoutSeconds: 1

successThreshold: 1

failureThreshold: 3

livenessProbe:

initialDelaySeconds: 0

periodSeconds: 10

timeoutSeconds: 1

successThreshold: 1

failureThreshold: 3

service:

# type: NodePort

type: ClusterIP

httpPort: 80

credentialSecretRef: htpasswd

scheduledTasks:

incomingReplication:

- metadata:

name: github

spec:

schedule: "0 * * * *"

config:

remotes:

- name: github

url: https://github.com

fetch:

- remoteRepo: fabric8io/kubernetes-client

- remoteRepo: operator-framework/java-operator-sdk

localRepo: josdk

refSpec: "+refs/heads/master:refs/heads/github/master"

secretRef: github-repl-secret

Gerrit initialization ran without errors:

[2024-07-18 09:33:29,040] INFO Requiring plugins (ClusterMode: HIGH_AVAILABILITY): ['healthcheck']

[2024-07-18 09:33:29,040] INFO Requiring libs (ClusterMode: HIGH_AVAILABILITY): []

[2024-07-18 09:33:29,044] INFO Created plugin installation directory: /var/gerrit/plugins

[2024-07-18 09:33:29,045] INFO Created lib installation directory: /var/gerrit/lib

[2024-07-18 09:33:29,048] INFO Installing plugin healthcheck from container to /var/gerrit/plugins/healthcheck.jar.

[2024-07-18 09:33:29,048] INFO Installing packaged plugin delete-project.

[2024-07-18 09:33:29,053] INFO Gerrit is not yet installed. Initializing new site.

[2024-07-18 09:33:29,053] INFO Existing gerrit.config found.

Auto-configured "receive.autogc = false" to disable auto-gc after git-receive-pack.

Generating SSH host key ... rsa... ed25519... ecdsa 256... ecdsa 384... ecdsa 521... done

Initialized /var/gerrit

Collecting accounts: 1

Collecting accounts: 1

Reindexing accounts: 100% (1/1)

Reindexing accounts: 100% (1/1)

Reindexing groups: 50% (1/2)

Reindexing groups: 100% (2/2)

Reindexing groups: 100% (2/2)

Reindexing changes: Slicing projects: 50% (1/2)

Reindexing changes: Slicing projects: 100% (2/2)

Reindexing changes: Slicing projects: 100% (2/2), done

Reindexing projects: 50% (1/2)

Reindexing projects: 100% (2/2)

Reindexing projects: 100% (2/2)

[2024-07-18 09:33:42,077] INFO Skipping reindexing.

But Gerrit itself throws errors:

[2024-07-18T10:57:13.450Z] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No SSH keys under /var/gerrit/etc

while locating HostKeyProvider

at SshHostKeyModule.configure(SshHostKeyModule.java:25)

at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)

\_ for 6th parameter hostKeyProvider

at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)

\_ for 1st parameter authenticator

at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)

while locating CachingPublicKeyAuthenticator

at SshDaemon.<init>(SshDaemon.java:168)

\_ for 3rd parameter userAuth

at SshDaemon.class(SshDaemon.java:140)

while locating SshDaemon

at SshModule.configure(SshModule.java:76)

while locating SshInfo

I have inspected gerrit site pv and did not locate any ssh keys in /var/gerrit/etc/. 

Any ideas what am I missing? Thanks!

[David Åkerman]

I have no experience with Gerrit Operator. But when doing a Gerrit init the keys are by default put in gerrit/etc. According to your first log dump it looks like Gerrit generated the keys successfully. Have you checked if the keys where put somewhere else?

Best regards,
David

[Thomas Dräbing]

You will have to provide manually created SSH keys in the secret referenced under `spec.gerrits[].spec.secretRef`. The reason is that if you scale Gerrit, each Gerrit instance would generate its own SSH keys and thus have different SSH host keys causing clients to see different host keys and being unable to verify the server's identity.

Best Regards,

Thomas

 

Best regards,
David

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bb0b90aa-db7d-45ed-8c05-d67be21218ban%40googlegroups.com.

[Igors Sersts]

Thanks David for your response. 

Thanks Thomas, yes, it solves the issue. Initialy I missunderstood the description for the key in API docs. I guess mark "optional" confused me :)