Gerrit installation inside kubernetes cluster

[Mk]

Is it possible to install gerrit(3.7.0) in kubernetes cluster (bare metal based) with NFS(Persistent Volume) using manifest files instead of Helm chart?

[Emil Dabrowski]

Sure thing, no problem with that. The container images are on Docker hub. Helm is just a templating engine that spits out Kubernetes manifests. You'll need a Statefulset, Service and Ingress - and optionally a PersistentVolumeClaim if you don't use volumeClaimTemplates in the Statefulset.

[Luca Milanesio]

On 13 Feb 2023, at 14:30, 'Emil Dabrowski' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Sure thing, no problem with that. The container images are on Docker hub. Helm is just a templating engine that spits out Kubernetes manifests. You'll need a Statefulset, Service and Ingress - and optionally a PersistentVolumeClaim if you don't use volumeClaimTemplates in the Statefulset.

I guess you’ll also need to use the high-availability plugin (see [1]).

Luca.

[1] https://gerrit.googlesource.com/plugins/high-availability/

måndag 13 februari 2023 kl. 13:18:12 UTC+1 skrev Mk:

Is it possible to install gerrit(3.7.0) in kubernetes cluster (bare metal based) with NFS(Persistent Volume) using manifest files instead of Helm chart?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8e4778e4-91f0-45ec-bc48-bc9be51ff52fn%40googlegroups.com.

[Mk]

Thanks Emil and Luca for the information. Basically I’m trying to deploy gerrit as pod inside my cluster using yaml files, further would like to access the cluster gerrit service outside like (Users --> HAproxy(Front End --> Back End) --> Kubernetes Cluster(Nginx Ingress --> Service --> Gerrit POD)

 My gerrit deployment.yml as follows,

 apiVersion: apps/v1

kind: Deployment

metadata:

  name: gerrit

  namespace: gerrit

spec:

  replicas: 1

  selector:

    matchLabels:

      app: gerrit

  template:

    metadata:

      labels:

        app: gerrit

    spec:

      containers:

      - name: gerrit

        image: gerritcodereview/gerrit:3.7.0

        ports:

        - containerPort: 8080

        - containerPort: 29418

        volumeMounts:

        - name: gerrit-data

          mountPath: /var/gerrit/review_site

      volumes:

      - name: gerrit-data

        persistentVolumeClaim:

          claimName: gerrit-pvc

 My gerrit pod running fine could see below log information.

 [2023-02-13T14:43:16.885+05:30] [main] INFO  org.eclipse.jetty.server.Server : Started @6275ms

[2023-02-13T14:43:16.887+05:30] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.7.0 ready

[2023-02-13T14:43:16.915+05:30] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@50d5931d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5a2df86d[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@54ea1d58]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories...

[2023-02-13T14:43:16.920+05:30] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@50d5931d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5a2df86d[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@54ea1d58]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories... Done

[2023-02-13T14:44:11.089+05:30] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : 70 plugins successfully pre-loaded

Nginx ingress resource running status as follows.

$ kubectl describe ingress gerrit-ingress -n gerrit
Name:             gerrit-ingress
Labels:           <none>
Namespace:        gerrit
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  gerrit.example.com
                   /gerrit   gerrit-service:80 (10.243.3.20:8080)
Annotations:       <none>
Events:        <none>

When i access the gerrit URl - http://gerrit.example.com/gerrit on browser it says " Not Found" message.

 $ curl http://gerrit.example.com/gerrit -v
*   Trying 10.11.152.61...
* TCP_NODELAY set
* Connected to gerrit.example.com (10.11.152.61) port 80 (#0)
> GET /gerrit HTTP/1.1
> Host: gerrit.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< server: nginx/1.23.3
< date: Tue, 14 Feb 2023 03:52:51 GMT
< content-type: text/plain;charset=iso-8859-1
< content-length: 9
< x-frame-options: DENY
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
<
* Connection #0 to host gerrit.example.com left intact

In the gerrit pod container i can see

$ kubectl exec -it gerrit-57776b8d8-xt548 -n gerrit -- ls -lsa /var/gerrit/
total 88
8 drwxr-xr-x 1 gerrit gerrit 4096 Feb 13 16:31 .
8 drwxr-xr-x 1 root   root   4096 Nov 21 18:25 ..
4 -rw------- 1 gerrit gerrit  221 Feb 13 17:38 .bash_history
8 drwxr-xr-x 1 gerrit gerrit 4096 Nov 21 18:25 .config
0 -rw-r--r-- 1 gerrit gerrit    0 Feb 13 14:43 .firstTimeRedirect
8 drwxr-xr-x 1 gerrit gerrit 4096 Nov 21 18:25 .gerritcodereview
8 drwxr-xr-x 1 gerrit gerrit 4096 Nov 21 18:25 bin
4 drwxr-xr-x 2 gerrit gerrit 4096 Feb 13 14:43 cache
4 drwxr-xr-x 1 gerrit gerrit 4096 Feb 13 14:43 data
4 drwxr-xr-x 2 gerrit gerrit 4096 Feb 13 14:43 db
4 drwxr-xr-x 3 gerrit gerrit 4096 Feb 13 14:43 etc
4 drwxr-xr-x 4 gerrit gerrit 4096 Feb 13 14:43 git
4 drwxr-xr-x 6 gerrit gerrit 4096 Feb 13 14:43 index
4 drwxr-xr-x 2 gerrit gerrit 4096 Nov 21 18:25 lib
4 drwxr-xr-x 1 gerrit gerrit 4096 Feb 13 14:43 logs
4 drwxr-xr-x 1 gerrit gerrit 4096 Feb 13 14:43 plugins
0 drwxr-sr-x 7 root   nobody  152 Feb 13 14:27 review_site
4 drwxr-xr-x 2 gerrit gerrit 4096 Nov 21 18:25 static
4 drwx------ 1 gerrit gerrit 4096 Feb 13 14:43 tmp

In deployment yml file if i change the "mountPath: /var/gerrit/review_site" to  "mountPath: /var/gerrit"  it fails with below errors.

$ kubectl logs -f gerrit-79f5b86f85-4nsks -n gerrit
Initializing Gerrit site ...

Error: Unable to access jarfile /var/gerrit/bin/gerrit.war

please let me the method I'm trying will work or not? If work, do let me know what is wrong with my approach and configurations?

[Thomas Dräbing]

Why do you not use the charts provided in the k8s-gerrit repository? If you have improvements, please feel free to push those! If you can't use helm with your cluster, you could always use `helm template` to compute the resource files and use those to deploy with kubectl.

On Tue, 14 Feb 2023 at 05:04, Mk <moha...@gmail.com> wrote:

Thanks Emil and Luca for the information. Basically I’m trying to deploy gerrit as pod inside my cluster using yaml files, further would like to access the cluster gerrit service outside like (Users --> HAproxy(Front End --> Back End) --> Kubernetes Cluster(Nginx Ingress --> Service --> Gerrit POD)

 My gerrit deployment.yml as follows,

 apiVersion: apps/v1

kind: Deployment

Gerrit has a lot of state to manage. Consider using a StatefulSet. 

Have you configured Gerrit properly to expect the /gerrit as the root path? If not, Gerrit would think that there should be a repository called gerrit that does not exist on a fresh site. In such cases Gerrit would return 'Not found'.

You try to initialize Gerrit using the war-file that is expected to be in the mounted volume, but it is empty I guess. To initialize Gerrit use the war file from a different path. Gerrit init will copy the war-file into the desired site.

 

please let me the method I'm trying will work or not? If work, do let me know what is wrong with my approach and configurations?

On Monday, February 13, 2023 at 10:43:19 PM UTC+5:30 Luca Milanesio wrote:

On 13 Feb 2023, at 14:30, 'Emil Dabrowski' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Sure thing, no problem with that. The container images are on Docker hub. Helm is just a templating engine that spits out Kubernetes manifests. You'll need a Statefulset, Service and Ingress - and optionally a PersistentVolumeClaim if you don't use volumeClaimTemplates in the Statefulset.

I guess you’ll also need to use the high-availability plugin (see [1]).

Note, that this would only allow two sites. To scale further, the multisite plugin would be needed.

 

Luca.

[1] https://gerrit.googlesource.com/plugins/high-availability/

måndag 13 februari 2023 kl. 13:18:12 UTC+1 skrev Mk:

Is it possible to install gerrit(3.7.0) in kubernetes cluster (bare metal based) with NFS(Persistent Volume) using manifest files instead of Helm chart?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8e4778e4-91f0-45ec-bc48-bc9be51ff52fn%40googlegroups.com.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f66b58c9-6402-452d-a396-b468fa1d7b69n%40googlegroups.com.

[Emil D]

>Error: Unable to access jarfile /var/gerrit/bin/gerrit.war

You can not mount the whole PVC to /var/gerrit as you then override the contents of /var/gerrit/bin. You must mount a PVC to each subdirectory (except bin). You can use 1x PVC bu using "subPath" like this:

volumeMounts:

- name: my-pvc

subPath: .ssh

mountPath: /var/gerrit/.ssh

- name: my-pvc

subPath: cache

mountPath: /var/gerrit/cache

- name: my-pvc

subPath: data

mountPath: /var/gerrit/data

- name: my-pvc

subPath: db

mountPath: /var/gerrit/db

- name: my-pvc

subPath: etc

mountPath: /var/gerrit/etc

- name: my-pvc

subPath: index

mountPath: /var/gerrit/index

- name: my-pvc

subPath: git

mountPath: /var/gerrit/git

- name: my-pvc

subPath: lib

mountPath: /var/gerrit/lib

- name: my-pvc

subPath: logs

mountPath: /var/gerrit/logs

- name: my-pvc

subPath: plugins

mountPath: /var/gerrit/plugins

>When i access the gerrit URl - http://gerrit.example.com/gerrit on browser it says " Not Found" message.

Why have you added /gerrit at the end of the URL? It's possible to serve Gerrit from a sub path, but requires additional configuration which I no longer remember how I did when I tried it. You will find it in the docs.

[Emil D]

>I guess you’ll also need to use the high-availability plugin

@Luca: Assuming that he wants to run more than 1 replica

>Why do you not use the charts provided in the k8s-gerrit repository? If you have improvements, please feel free to push those!

FYI; My team looked into the k8s-gerrit repo but we didn't like it. E.g. .metadata.labels are non-conventional (instead, check the generated templates of helm create foo). Additionally the requirement for ReadWriteMany doesn't suit us. I also wonder; why all the special container images instead of improving the upstream container? We've created our own chart, and hope to build an HA where each replica in the Statefulset has its own PVC.

Speaking of pushing improvements... OSS contributions are very difficult for us, especially when it comes to Gerrit, since Google requires a signed agreement between Google and our company which becomes burdensome for our little team that just wants to build good infra :-)

Kind regards,

Emil

[Thomas Dräbing]

On Tue, 14 Feb 2023 at 10:36, 'Emil D' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

>I guess you’ll also need to use the high-availability plugin

@Luca: Assuming that he wants to run more than 1 replica

>Why do you not use the charts provided in the k8s-gerrit repository? If you have improvements, please feel free to push those!

FYI; My team looked into the k8s-gerrit repo but we didn't like it. E.g. .metadata.labels are non-conventional (instead, check the generated templates of helm create foo).

Yes, the labels in the charts were not chosen well initially :-(. That is something that I wanted to change for some time, but never came around doing so. We are currently working on an operator that will supersede the helm charts. The resources will use the recommended standard label format then.

 

Additionally the requirement for ReadWriteMany doesn't suit us.

Getting rid of RWM is also our goal (at least for the primary Gerrit), since this doesn't work properly due to the latencies. We are in the process of creating a setup based on the multisite plugin. Then each Gerrit in the StatefulSet would have its own site.

For Gerrit Replicas we plan to keep the RWM volume for the repositories for now, since we have some good experience there, but of course it would be possible to make this configurable.

 

I also wonder; why all the special container images instead of improving the upstream container?

For the runtime containers for Gerrit, this is mostly historical. It previously contained some code that was specific to the environment provided by Kubernetes and the helm-chart. This is now all moved to the gerrit-init container, which is definitely required to automate some decisions that usually a human would do. With some work the gerrit-init container would also be useful when running Gerrit with docker-compose, but as we did not require that, we just never worked on it.

In general, it was just convenient for us so far to have specifically tailored container images. I wouldn't be against moving to use the docker-gerrit images in the future, if we can run them without sudo and root-user. For me this is right now just not a priority, since I never got that requirement. 

To be honest the project also suffers from the fact that most contributions come from our team and thus of course are focused on our requirements and priorities. This is definitely something that I would like to change, since this would greatly improve the project IMO. At this point, thanks to everybody who already contributed changes!

 

We've created our own chart, and hope to build an HA where each replica in the Statefulset has its own PVC.

This is what we also want to achieve using the multisite plugin and an operator to manage all the required components. We hope to have a first version of this around summer this year.

 

Speaking of pushing improvements... OSS contributions are very difficult for us, especially when it comes to Gerrit, since Google requires a signed agreement between Google and our company which becomes burdensome for our little team that just wants to build good infra :-)

That is unfortunate. I think your insights would be very valuable.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7654228c-4972-42d0-adb7-18b2d44d0647n%40googlegroups.com.

[Mk]

Hello Emil,

I have deployed kind as "Deployment" instead of " StatefulSet" with "volumeMounts" along with "subPath" which you have mentioned above. Now i can see the data on my NFS path.

Further when i tried to access the gerrit url on browser.  only blank page it shows. also on cli using curl it shows below message

$ curl -L http://gerrit.example.com/gerrit -v

*   Trying 10.11.152.61...
* TCP_NODELAY set
* Connected to gerrit.example.com (10.11.152.61) port 80 (#0)
> GET /gerrit HTTP/1.1
> Host: gerrit.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>

< HTTP/1.1 302 Found
< server: nginx/1.23.3
< date: Wed, 15 Feb 2023 03:45:49 GMT
< transfer-encoding: chunked
< location: http://gerrit.example.com/gerrit/
<
* Ignoring the response-body

* Connection #0 to host gerrit.example.com left intact

* Issue another request to this URL: 'http://gerrit.example.com/gerrit/'
* Found bundle for host gerrit.example.com: 0x557519ceda60 [can pipeline]
* Re-using existing connection! (#0) with host gerrit.example.com

* Connected to gerrit.example.com (10.11.152.61) port 80 (#0)

> GET /gerrit/ HTTP/1.1

> Host: gerrit.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>

< HTTP/1.1 200 OK
< server: nginx/1.23.3
< date: Wed, 15 Feb 2023 03:45:49 GMT
< content-type: text/html;charset=utf-8
< content-length: 4732
< x-frame-options: DENY
< set-cookie: XSRF_TOKEN=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0
< expires: Thu, 01 Jan 1970 00:00:00 GMT
<
<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<meta name="description" content="Gerrit Code Review">
<meta name="referrer" content="never">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=0">
<noscript>To use PolyGerrit, please enable JavaScript in your browser settings, and then refresh this page.</noscript><script>window.polymerSkipLoadingFontRoboto = true; window.CLOSURE_NO_DEPS = true; window.DEFAULT_DETAIL_HEXES = {}; window.PRELOADED_QUERIES = {};window.INITIAL_DATA = JSON.parse('\x7b\x22\/config\/server\/version\x22: \x223.7.0\x22, \x22\/config\/server\/info\x22: \x7b\x22accounts\x22:\x7b\x22visibility\x22:\x22ALL\x22,\x22default_display_name\x22:\x22FULL_NAME\x22\x7d,\x22auth\x22:\x7b\x22auth_type\x22:\x22LDAP\x22,\x22editable_account_fields\x22:\x5b\x22REGISTER_NEW_EMAIL\x22\x5d,\x22git_basic_auth_policy\x22:\x22HTTP\x22\x7d,\x22change\x22:\x7b\x22allow_blame\x22:true,\x22update_delay\x22:300,\x22mergeability_computation_behavior\x22:\x22NEVER\x22,\x22enable_attention_set\x22:true\x7d,\x22download\x22:\x7b\x22schemes\x22:\x7b\x7d,\x22archives\x22:\x5b\x22tgz\x22,\x22tar\x22,\x22tbz2\x22,\x22txz\x22\x5d\x7d,\x22gerrit\x22:\x7b\x22all_projects\x22:\x22All-Projects\x22,\x22all_users\x22:\x22All-Users\x22,\x22doc_search\x22:true\x7d,\x22note_db_enabled\x22:true,\x22plugin\x22:\x7b\x22js_resource_paths\x22:\x5b\x22plugins\/codemirror-editor\/static\/codemirror_editor.js\x22,\x22plugins\/delete-project\/static\/gr-delete-repo.js\x22\x5d\x7d,\x22sshd\x22:\x7b\x7d,\x22suggest\x22:\x7b\x22from\x22:0\x7d,\x22user\x22:\x7b\x22anonymous_coward_name\x22:\x22Name of user not set\x22\x7d,\x22receive\x22:\x7b\x22enable_signed_push\x22:false\x7d,\x22submit_requirement_dashboard_columns\x22:\x5b\x5d\x7d, \x22\/config\/server\/top-menus\x22: \x5b\x5d\x7d');window.ENABLED_EXPERIMENTS = JSON.parse('\x5b\x22UiFeature__patchset_comments\x22,\x22UiFeature__submit_requirements_ui\x22\x5d');</script>
<link rel="icon" type="image/x-icon" href="/favicon.ico">
<link rel="preload" href="/fonts/opensans-latin-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/opensans-latin-600.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/opensans-latin-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/opensans-latin-ext-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/opensans-latin-ext-600.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/opensans-latin-ext-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-500.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-ext-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-ext-500.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-latin-ext-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-500.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-ext-400.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-ext-500.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/roboto-mono-latin-ext-700.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="/fonts/material-icons.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" as="style" href="/styles/fonts.css">
<link rel="preload" as="style" href="/styles/material-icons.css">
<link rel="preload" as="style" href="/styles/main.css">
<script src="/bower_components/webcomponentsjs/webcomponents-lite.js"></script>
<link rel="stylesheet" href="/styles/fonts.css">
<link rel="stylesheet" href="/styles/material-icons.css">
<link rel="stylesheet" href="/styles/main.css">
<body unresolved>
<gr-app id="pg-app"></gr-app>
<script src="/elements/gr-app.js" crossorigin="anonymous"></script>

* Connection #0 to host gerrit.example.com left intact

Additionally do i need to configure anything in my gerrit.config file to access my gerrit?

[Emil D]

Hi!

Try using these environment variables

CANONICAL_WEB_URL: https://subdomain.example.com/

HTTPD_LISTEN_URL: proxy-http://*:8080

[Mk]

With the above environment variables able to access cluster gerrit :)

Thanks a lot.

[Emil D]

Happy I could help! Enjoy your weekend.

[Dmytro Sydorov]

Hi @Thomas Dräbing,

> We are currently working on an operator that will supersede the helm charts
is there any public information about the operator, ETA, documentation, and so on? We are considering the migration to k8s, but the current implementation of the Helm chart is not flexible and brings multiple problems.

[Thomas Dräbing]

Hi Dmytro,

On Mon, Mar 6, 2023, 14:43 'Dmytro Sydorov' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hi @Thomas Dräbing,

> We are currently working on an operator that will supersede the helm charts
is there any public information about the operator, ETA, documentation, and so on?

The operator is developed in the same repository as the helm charts [1], in the `operator` subdirectory. There is also documentation present there.

I can't give you an exact ETA, but we plan to have it ready this year and will then also properly announce it.

We are considering the migration to k8s, but the current implementation of the Helm chart is not flexible and brings multiple problems.

Could you tell me what flexibility you would like to have and which problems you see? This would also help a lot in developing the operator.

Contributions fixing those issues are of course welcome!

Thanks,

Thomas

[1] https://gerrit.googlesource.com/k8s-gerrit

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d9237aa9-7c2c-4ead-bfd7-afe9b5936370n%40googlegroups.com.

[Dmytro Sydorov]

Hi Thomas,

> Could you tell me what flexibility you would like to have and which problems you see? This would also help a lot in developing the operator.

I have tried to install Gerrit to my k8s playground on AWS (EKS) and make it HA. Here I faced multiple issues. 

1. Expose endpoints. We must expose Gerrit web and SSH endpoints that listen to different ports and protocols but use the same domain. Ingress configuration is a bit tricky in this case, just a plain Service resource with annotations fits better, but the Helm chart does not expect any annotations: https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/helm-charts/gerrit/templates/gerrit.service.yaml. So I have to maintain the service outside of the Helm chart.

2. Storage. For HA I need some RWX storage, so in this case, AWS EFS is the only option. It supports dynamic provisioning (which is not something I really like) and static provisioning. Dynamic provisioned adds a root folder name === PVC name (/pvc-74a498d6-3929-47e8-8c02-078c1ece4d78) and it adds unnecessary complexity for the disaster recovery scenario where we need to patch storage configuration manually.

With static provisioning, you can specify the file system ID and access point IDs for the PV, and mount exactly the same access point to all pods. But again, it's not supported by the Helm chart, I can specify only the name and the size, but not a custom config. https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/helm-charts/gerrit/templates/storage.pvc.yaml#11
3. Secret and values.yaml. I can't create an external secret and mount it to the pod(s). I have to put everything into values.yaml, which makes this data not really secret.

4. HA. To be fair, it's not really a Helm issue, but still. It's not supported by default. I can set `replicas: 2` but it will not help. We have to add some additional plugins instead of just some external endpoint of Redis or Memcached. Sharing some data between pods and in-memory databases does not sound robust to me. In our case, we use spot instances for k8s workloads, so there is a chance that all pods can die at the same time. I don't want to trust in-memory storage here. The configuration of the HA plugin does not look like something that can be used in a dynamic environment like Kubernetes, so I can't use HPA or something like this to scale pods horizontally.

Thanks,
Dmytro

[Thomas Dräbing]

Hi Dmytro,

thanks for the feedback. See my comments below

On Tue, 11 Apr 2023 at 12:16, 'Dmytro Sydorov' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hi Thomas,

> Could you tell me what flexibility you would like to have and which problems you see? This would also help a lot in developing the operator.

I have tried to install Gerrit to my k8s playground on AWS (EKS) and make it HA. Here I faced multiple issues. 

1. Expose endpoints. We must expose Gerrit web and SSH endpoints that listen to different ports and protocols but use the same domain. Ingress configuration is a bit tricky in this case, just a plain Service resource with annotations fits better, but the Helm chart does not expect any annotations: https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/helm-charts/gerrit/templates/gerrit.service.yaml. So I have to maintain the service outside of the Helm chart.

Using an ingress with SSH is indeed tricky. We decided to use istio for that and other reasons (only implemented for the gerrit-replica chart for now). If annotations in the service would already help you, this would only be a small addition to the helm chart. Let me add that for you.

 

2. Storage. For HA I need some RWX storage, so in this case, AWS EFS is the only option. It supports dynamic provisioning (which is not something I really like) and static provisioning. Dynamic provisioned adds a root folder name === PVC name (/pvc-74a498d6-3929-47e8-8c02-078c1ece4d78) and it adds unnecessary complexity for the disaster recovery scenario where we need to patch storage configuration manually.

With static provisioning, you can specify the file system ID and access point IDs for the PV, and mount exactly the same access point to all pods. But again, it's not supported by the Helm chart, I can specify only the name and the size, but not a custom config. https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/helm-charts/gerrit/templates/storage.pvc.yaml#11

There is an option to configure an external PVC instead of letting helm manage it: `gitRepositoryStorage.externalPVC`. I did that because I didn't like to manage the storage with helm. It was just too easy to break storage during upgrades of the chart. I don't see an issue to add options to statically reference PVs by name or label in the helm chart however.

3. Secret and values.yaml. I can't create an external secret and mount it to the pod(s). I have to put everything into values.yaml, which makes this data not really secret.

The values.yaml is a client-side file, which will never be uploaded to the cluster, same as any plain Kubernetes yaml containing a secret. We use tools like sops or the vault-plugin for ArgoCD to template secrets into the yaml-files. But I guess you are creating the secret directly by command line or some other tool. referencing external secrets is definitely something to add (The operator will do that from the start, since we can't put secrets into the CustomResources).

 

4. HA. To be fair, it's not really a Helm issue, but still. It's not supported by default. I can set `replicas: 2` but it will not help. We have to add some additional plugins instead of just some external endpoint of Redis or Memcached. Sharing some data between pods and in-memory databases does not sound robust to me. In our case, we use spot instances for k8s workloads, so there is a chance that all pods can die at the same time. I don't want to trust in-memory storage here. The configuration of the HA plugin does not look like something that can be used in a dynamic environment like Kubernetes, so I can't use HPA or something like this to scale pods horizontally.

 

Gerrit is not scalable out of the box, since the state of the index and git repositories has to be in sync on all instances, which can't be achieved with Gerrit core, and thus the chart does not support it as well. There are two plugins that add some support to scale Gerrit, the HA-plugin you mentioned and the multisite plugin. As you said the HA-plugin does not support dynamic scaling. We thus aim for using the multisite plugin, but that requires a quite complex setup of multiple plugins and external components. That's why we decided to go with developing an operator rather than using helm charts. This is still work in progress, but we hope to have this ready and deployed at SAP by the end of this year. As always any contributions helping us to reach this goal would be very much appreciated.

Best,

Thomas

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2cfdae90-7800-4965-adb8-611eabea2a97n%40googlegroups.com.

[Thomas Dräbing]

On Tue, 11 Apr 2023 at 13:12, Thomas Dräbing <thomas....@gmail.com> wrote:

Hi Dmytro,

thanks for the feedback. See my comments below

On Tue, 11 Apr 2023 at 12:16, 'Dmytro Sydorov' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hi Thomas,

> Could you tell me what flexibility you would like to have and which problems you see? This would also help a lot in developing the operator.

I have tried to install Gerrit to my k8s playground on AWS (EKS) and make it HA. Here I faced multiple issues. 

1. Expose endpoints. We must expose Gerrit web and SSH endpoints that listen to different ports and protocols but use the same domain. Ingress configuration is a bit tricky in this case, just a plain Service resource with annotations fits better, but the Helm chart does not expect any annotations: https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/helm-charts/gerrit/templates/gerrit.service.yaml. So I have to maintain the service outside of the Helm chart.

Using an ingress with SSH is indeed tricky. We decided to use istio for that and other reasons (only implemented for the gerrit-replica chart for now). If annotations in the service would already help you, this would only be a small addition to the helm chart. Let me add that for you.

Pushed for review here: https://gerrit-review.googlesource.com/c/k8s-gerrit/+/368634 

[Dmytro Sydorov]

Thank you, Thomas! 

Btw, are there any plans to publish the helm chart to some public registry?