Difference between Gerrit and gerrit operator

[Aankhi Talukdar]

Could someone please let me know what is the difference between Gerrit operator and Gerrit in the helm chart repo and which one is beneficial to use for Gerrit deployment?

Thanks

[Matthias Sohn]

On Wed, Jan 17, 2024 at 3:53 PM Aankhi Talukdar
<tallukd...@gmail.com> wrote:
>
> Could someone please let me know what is the difference between Gerrit operator and Gerrit in the helm chart repo and which one is beneficial to use for Gerrit deployment?

The Gerrit helm chart is a rather static way to deploy Gerrit in k8s
and you need to configure everything manually.

The Gerrit operator defines some custom resource for configuring a
Gerrit cluster on k8s on a higher level
reducing the amount of configuration you have to do manually and it
helps to configure it correctly.
In addition it manages the application lifecycle and automates tasks
necessary to run a Gerrit cluster.

Since we think the operator is the superior approach we focus on
evolving the operator and have deprecated the helm charts.

-Matthias

>
> Thanks
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3eb3c117-2404-44d9-b589-04dda1942288n%40googlegroups.com.

[Aankhi Talukdar]

Hi Matthias,

Thanks for your reply. So, do you mean the Gerrit operator helm chart got deprecated or the individual gerrit and gerrit replica helm charts? We are referring to the repo:
helm-charts/gerrit-operator - k8s-gerrit - Git at Google (googlesource.com)--> Is this chart deprecated?

Thanks

Aankhi

[Matthias Sohn]

On Mon, Feb 5, 2024 at 2:08 PM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Matthias,

Thanks for your reply. So, do you mean the Gerrit operator helm chart got deprecated or the individual gerrit and gerrit replica helm charts? We are referring to the repo:
helm-charts/gerrit-operator - k8s-gerrit - Git at Google (googlesource.com)--> Is this chart deprecated?

The Gerrit Operator helm chart is there to stay. I meant the individual gerrit and gerrit-replica helm charts are deprecated

and we don't maintain them any longer.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/27647451-bed2-4a38-9f2c-3609191b797en%40googlegroups.com.

[Aankhi Talukdar]

Hi Mathias,

Thanks for the confirmation.
Also, we have another situation here. I would be grateful if you could help:

We are in the process of setting up Gerrit with a Gerrit operator on a local Ubuntu cluster using Minikube for testing. We've successfully deployed the Gerrit operator, along with the Gerrit cluster custom resource (CR) containing one primary and one replica, and ensured that prerequisites like NFS and Ingress (ISTIO) are installed.

However, even though all the pods and deployments are running, the Gerrit UI is not accessible. Upon checking the logs of the Gerrit operator, we noticed some startup errors related to the informer, and there are also warnings about the version. (Screenshot attached below: Operator_logs.png). Can someone provide insights into why these errors might be occurring?

Thanks

Aankhi

[Matthias Sohn]

On Mon, Feb 5, 2024 at 2:34 PM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Mathias,

Thanks for the confirmation.

Please don't use top posting on this list [1] we prefer interleaved posting style [2] which simplifies following the conversation.

 

[1] https://en.wikipedia.org/wiki/Posting_style#Top-posting

[2] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

Also, we have another situation here. I would be grateful if you could help:

We are in the process of setting up Gerrit with a Gerrit operator on a local Ubuntu cluster using Minikube for testing. We've successfully deployed the Gerrit operator, along with the Gerrit cluster custom resource (CR) containing one primary and one replica, and ensured that prerequisites like NFS and Ingress (ISTIO) are installed.

However, even though all the pods and deployments are running, the Gerrit UI is not accessible. Upon checking the logs of the Gerrit operator, we noticed some startup errors related to the informer, and there are also warnings about the version. (Screenshot attached below: Operator_logs.png). Can someone provide insights into why these errors might be occurring?

Can you post the logs verbatim in your email instead of attaching very small screenshots ?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7ac49c7d-3864-437e-b125-cc3e56498052n%40googlegroups.com.

[Aankhi Talukdar]

Hi Matthias,

Please find the logs below:

Thanks

Aankhi

[Matthias Sohn]

On Tue, Feb 6, 2024 at 7:36 AM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Matthias,

Please find the logs below:

I don't know what's causing this rather generic error in your setup.

Did you try with the latest k8s-gerrit version ?

Thomas pushed some example setups using minikube here

https://gerrit-review.googlesource.com/c/k8s-gerrit/+/405561

Maybe this helps.

-Matthias

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/c27c516c-b2da-4898-b054-5c948f7eaf58n%40googlegroups.com.

[Thomas Dräbing]

On Tue, 6 Feb 2024 at 09:29, Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Feb 6, 2024 at 7:36 AM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Matthias,

Please find the logs below:

The error is caused by missing CRDs. The operator requires the CRDs of the resources it manages to be present in the cluster. The easiest way to install those, is to install the gerrit-operator-crds helm chart, which is also a dependency of the gerrit-operator helm chart.

HTH,

Thomas

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3T748jmbTUqj03w44YB4kyi8EO8A49P5XzfXzjPvUynQw%40mail.gmail.com.

[Thomas Dräbing]

On Wed, 7 Feb 2024 at 08:16, Thomas Dräbing <thomas....@gmail.com> wrote:

On Tue, 6 Feb 2024 at 09:29, Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Feb 6, 2024 at 7:36 AM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Matthias,

Please find the logs below:

The error is caused by missing CRDs. The operator requires the CRDs of the resources it manages to be present in the cluster. The easiest way to install those, is to install the gerrit-operator-crds helm chart, which is also a dependency of the gerrit-operator helm chart.

The other possibility is that the CRD version doesn't match the version expected by the operator (v1beta3). The newest version is v1beta5, so that is not unlikely. I will check whether the dockerhub images are up-to-date. It might be that the last update of the images failed.

[Aankhi Talukdar]

Hi Thomas,

The CRDs are already implemented as a dependency in the Gerrit operator.

Used the following commands to implement CRDs as per the documentation:

# Build the gerrit-operator-crds chart and store it in the charts/ subdirectory
helm dependency build helm-charts/gerrit-operator/
# Install the gerrit-operator-crds chart and the gerrit-operator chart
helm -n gerrit-operator install gerrit-operator helm-charts/gerrit-operator/

Also, the CRDs are installed as shown below:
kubectl get crds
NAME                                       CREATED AT
addons.k3s.cattle.io                       2024-02-07T06:26:44Z
etcdsnapshotfiles.k3s.cattle.io            2024-02-07T06:26:45Z
helmcharts.helm.cattle.io                  2024-02-07T06:26:45Z
helmchartconfigs.helm.cattle.io            2024-02-07T06:26:45Z
gerritnetworks.gerritoperator.google.com   2024-02-07T07:07:43Z
gitgcs.gerritoperator.google.com           2024-02-07T07:07:43Z
receivers.gerritoperator.google.com        2024-02-07T07:07:43Z
gerrits.gerritoperator.google.com          2024-02-07T07:07:43Z
gerritclusters.gerritoperator.google.com   2024-02-07T07:07:43Z

I too suspect there might be some issue with the version mismatch as shown in the logs. Could you please update on the same?

Thanks

Aankhi

[Thomas Dräbing]

On Wed, 7 Feb 2024 at 10:00, Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

The CRDs are already implemented as a dependency in the Gerrit operator.

Used the following commands to implement CRDs as per the documentation:

# Build the gerrit-operator-crds chart and store it in the charts/ subdirectory
helm dependency build helm-charts/gerrit-operator/
# Install the gerrit-operator-crds chart and the gerrit-operator chart
helm -n gerrit-operator install gerrit-operator helm-charts/gerrit-operator/

Also, the CRDs are installed as shown below:
kubectl get crds
NAME                                       CREATED AT
addons.k3s.cattle.io                       2024-02-07T06:26:44Z
etcdsnapshotfiles.k3s.cattle.io            2024-02-07T06:26:45Z
helmcharts.helm.cattle.io                  2024-02-07T06:26:45Z
helmchartconfigs.helm.cattle.io            2024-02-07T06:26:45Z
gerritnetworks.gerritoperator.google.com   2024-02-07T07:07:43Z
gitgcs.gerritoperator.google.com           2024-02-07T07:07:43Z
receivers.gerritoperator.google.com        2024-02-07T07:07:43Z
gerrits.gerritoperator.google.com          2024-02-07T07:07:43Z
gerritclusters.gerritoperator.google.com   2024-02-07T07:07:43Z

I too suspect there might be some issue with the version mismatch as shown in the logs. Could you please update on the same?

The container images were indeed out of date in dockerhub. So if you used those images, and the helm-charts at the current tip of the master branch, that would explain the error. I have pushed the updated container images.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/70e7c4f7-c285-462f-aa7d-b7ba71363d76n%40googlegroups.com.

[Aankhi Talukdar]

Hi Thomas,

Thanks for the update. The issue is resolved now. 

I could make the gerrit and gerrit replica up without ingress(with port forwarding).

Trying to implement ingress following the documentation:  https://gerrit-review.googlesource.com/c/k8s-gerrit/+/405561/4/Documentation/operator.md#401

The istio-ingressgateway service is assigned with an external service and the Gateway URL as: 127.0.0.1:80. (Screenshot attached)
When trying to access the application with the URL, it throws a "404 page not found error".
Could you let me know what else am I missing?

Thanks

Aankhi

[Thomas Dräbing]

On Wed, 7 Feb 2024 at 21:10, Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Thanks for the update. The issue is resolved now. 

I could make the gerrit and gerrit replica up without ingress(with port forwarding).

Trying to implement ingress following the documentation:  https://gerrit-review.googlesource.com/c/k8s-gerrit/+/405561/4/Documentation/operator.md#401

The istio-ingressgateway service is assigned with an external service and the Gateway URL as: 127.0.0.1:80. (Screenshot attached)
When trying to access the application with the URL, it throws a "404 page not found error".
Could you let me know what else am I missing?

 Please check the following:

- Is a Gateway and a VirtualService resource present in the namespace which contains Gerrit? If not, the operator did not configure the service mesh.

        - Did you enable ISTIO support in the gerrit-operator?

        - Was the Gerrit Operator restarted after the config change (helm sometimes does not do that). You should see some logs regarding istio resources in the Gerrit Operator logs.

- Were istio proxy sidecar containers installed in the Gerrit pods? Might require a restart of the pods to happen (I think I missed that in the documentation)

- Do you see any logs of your request in the ingress-gateway pod? Might require to enable access logs in istio.

- Do you see any access logs in the istio sidecar containers in the Gerrit pods?

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/41eadea4-86fb-4d9c-9d40-b07e5be72648n%40googlegroups.com.

[Aankhi Talukdar]

Hi Thomas,

Thanks for replying. I could bring up the Gerrit UI now. Thanks for helping.

Do we have Gerrit operator image supporting linux/arm64 platform as we could see that the image is supporting only Linux/amd64 architecture.

If so, please share the image. If not, could you please share with us the Dockerfile for the gerrit-operator image?

Thanks

Aankhi

[Matthias Sohn]

On Thu, Feb 8, 2024 at 11:40 AM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Thanks for replying. I could bring up the Gerrit UI now. Thanks for helping.

Do we have Gerrit operator image supporting linux/arm64 platform as we could see that the image is supporting only Linux/amd64 architecture.

If so, please share the image. If not, could you please share with us the Dockerfile for the gerrit-operator image?

The Gerrit operator is built by its maven build.

See https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/Documentation/operator-dev.md#publish

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CALaBVApq%3Drn_VjAsNkWst%2Bgv8TAu%2B_SApDUHBxJnn4-hJP7nMA%40mail.gmail.com.

[Aankhi Talukdar]

Hi Matthias,

When we tried to run the maven build using the cmd [mvn clean install], we got errors related to the plugins used. I don't get the usage of the plugins and even though the image is created locally, the image is not runnable in arm64 architecture.

As per the documentation mentioned over here:https://gerrit.googlesource.com/k8s-gerrit/+/54ee885f27451cf12528ff9348316a670802c19e/Documentation/operator-dev.md

[This will build the operator source code, and create an image out of the built artifacts. The built image is multi-platform - it will run on both amd64 and arm64 architectures. It is okay to run this build command from an ARM Mac.]

When run the build command from ARM Mac, getting build errors as provided in the log file attached.

Also, the created image is specific to amd64 platform alone.

Thanks

Aankhi

[Matthias Sohn]

On Mon, Feb 12, 2024 at 7:10 AM Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Matthias,

When we tried to run the maven build using the cmd [mvn clean install], we got errors related to the plugins used. I don't get the usage of the plugins and even though the image is created locally, the image is not runnable in arm64 architecture.

As per the documentation mentioned over here:https://gerrit.googlesource.com/k8s-gerrit/+/54ee885f27451cf12528ff9348316a670802c19e/Documentation/operator-dev.md

[This will build the operator source code, and create an image out of the built artifacts. The built image is multi-platform - it will run on both amd64 and arm64 architectures. It is okay to run this build command from an ARM Mac.]

When run the build command from ARM Mac, getting build errors as provided in the log file attached.

Also, the created image is specific to amd64 platform alone.

We use the jib-maven-plugin to build the container, maybe configuring the platform help ?

See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#how-do-i-specify-a-platform-in-the-manifest-list-or-oci-index-of-a-base-image

[Aankhi Talukdar]

Hi Matthias/ Thomas,

When we tried to install Gerrit with Minikube, the Gerrit UI was up and running. Now, we are trying to implement the same in the rancher environment. While doing so, we are getting a mount error while installing nfs provisioner as attached in the screenshot below. 
We have followed the below document to install nfs: 

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

We have tried to google various w/a and installed nfs-common as instructed. Also, checked that mount.cifs and mount.nfs are listed to /sbin.

aantal01@PW054HD3:/var/lib$ ls -l /sbin/mount.cifs

-rwsr-xr-x 1 root root 48200 Jun  1  2022 /sbin/mount.cifs

aantal01@PW054HD3:/var/lib$ ls -l /sbin/mount.nfs

-rwsr-xr-x 1 root root 121688 Jun 29  2023 /sbin/mount.nfs

We do not understand why we are getting this issue. Could you please help?

Thanks

Aankhi

[Aankhi Talukdar]

Screenshot attached

[Aankhi Talukdar]

Hi Matthias/ Thomas,

Can someone help with the above issue?

Thanks

Aankhi

[Thomas Dräbing]

Hi Aankhi,

sorry for the late reply. Unfortunately, I don't think I can help with this. I am unfamiliar with Rancher and the infrastructure that you use. For us the nfs-subdir-provisioner works out of the box more or less. Note, that you could try to mount the volume directly without a provisioner, since you likely do not plan to dynamically provision them [1].

Regarding the multiplatform builds, I am planing to add support for it soon.

Best Regards,

Thomas

[1] https://kubernetes.io/docs/concepts/storage/volumes/#nfs

[Aankhi Talukdar]

Hi Thomas,

Thanks for your reply. 

I've attempted to implement the volume mount directly without using a provisioner by creating PersistentVolumes (PV) and PersistentVolumeClaims (PVC) in a bound state. However, when I try to include the sharedStorage section in the Gerrit cluster YAML file along with the node affinity section, I encounter the following error:

"Error from server (BadRequest): error when creating "gerrit-cluster.yaml": GerritCluster in version "v1beta4" cannot be handled as a GerritCluster: strict decoding error: unknown field "spec.gerrits[0].spec.affinity.requiredDuringSchedulingIgnoredDuringExecution"

I also attempted changing the version to v1beta5, but the issue persisted.

Could you please advise on how to properly integrate the volume mount section into the Gerrit cluster YAML file without using a provisioner? Alternatively, if possible, could you share the repository you used to deploy Gerrit on AWS?

Thanks

Aankhi

[Aankhi Talukdar]

Hi Thomas,

Can you please help with the above issue? 

Thanks

Aankhi

[Thomas Dräbing]

Hi Aankhi,

the key `spec.gerrits[0].spec.affinity.requiredDuringSchedulingIgnoredDuringExecution` doesn't exist. I guess you meant `spec.gerrits[0].spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` [1].

I just saw that there is a formatting error in the example provided in the documentation. I will push a fix for that.

Best Regards,

Thomas

[1] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

[Aankhi Talukdar]

Hi Thomas,

Thanks for pointing out the formatting error. 

Could you please also let us know what should be the URL for globalref-db under the libs section of gerrit-cluster.yaml file? How did we set the URL and sha1 for the same?

I am trying to pull the module: https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.9/job/module-global-refdb-bazel-stable-3.9/lastSuccessfulBuild/artifact/bazel-bin/plugins/global-refdb/global-refdb.jar
but this is giving us sha1 mismatch issue:
initializer.tasks.download_plugins.InvalidPluginException: SHA1 of downloaded file (e4344b4e9ec70b52085fc8a8e313e93a328f4675) did not match expected SHA1 (0797df9f4d3937f2f78f1b8a0d9e5092bfaeee58). Removed downloaded file (/var/gerrit/lib/global-refdb.jar)
NOTE: I am following gerrit cluster API https://gerrit.googlesource.com/k8s-gerrit/+/HEAD/Documentation/operator-api-reference.md#gerritcluster to create gerrit cluster.

Logs attached

Thanks

Aankhi

[Aankhi Talukdar]

Hi Thomas,

Any update on the issue?
Note: My rancher architecture has three master nodes and three worker nodes. 

Thanks

Aankhi

[Thomas Dräbing]

On Wed, 21 Feb 2024 at 21:17, Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Thanks for pointing out the formatting error. 

Could you please also let us know what should be the URL for globalref-db under the libs section of gerrit-cluster.yaml file? How did we set the URL and sha1 for the same?

I am trying to pull the module: https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.9/job/module-global-refdb-bazel-stable-3.9/lastSuccessfulBuild/artifact/bazel-bin/plugins/global-refdb/global-refdb.jar
but this is giving us sha1 mismatch issue:
initializer.tasks.download_plugins.InvalidPluginException: SHA1 of downloaded file (e4344b4e9ec70b52085fc8a8e313e93a328f4675) did not match expected SHA1 (0797df9f4d3937f2f78f1b8a0d9e5092bfaeee58). Removed downloaded file (/var/gerrit/lib/global-refdb.jar)

The scripts initializing the Gerrit site will verify that the downloaded file has the same sha1 sum as configured in the CustomResource for that lib. In this case, these SHA1 sums do not match. Note, that since you use the last successful build, the SHA sum will continuously change with each new build. You should rather use a stable URL (gerrit-ci.gerritforge.com does not really provide those and for plugins no maven repository exists, so currently the only solution would be to deploy it in some object store you own or to have a custom build of the gerrit.war file where the lib is built in).

Note, that the global-refdb lib is already built into the container image and will automatically be installed, if you scale your primary Gerrit to more than 1 pod.

[Aankhi Talukdar]

Hi Thomas

Thanks for the information.

when we try to install the Gerrit-cluster via the below yaml file, we could see init containers as well and the pod is going to a pending state as the init container is failing with the error msg: crashloopbackoffHow are these init containers configured with gerritcluster setup?The container logs mentioned below:swaveg01@K6X6QLT91G gerrit-cluster % k logs gerrit-0 -c gerrit-init -n gerrit
[2024-02-22 14:51:03,717] INFO Requiring plugins: ['healthcheck']
[2024-02-22 14:51:03,717] INFO Requiring libs: []
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/var/tools/gerrit-initializer/__main__.py", line 18, in <module>
    main()
  File "/var/tools/gerrit-initializer/main.py", line 89, in main
    args.func(args)
  File "/var/tools/gerrit-initializer/main.py", line 31, in _run_init
    init.GerritInit(args.site, config).execute()
  File "/var/tools/gerrit-initializer/initializer/tasks/init.py", line 180, in execute
    self._symlink_mounted_site_components()
  File "/var/tools/gerrit-initializer/initializer/tasks/init.py", line 122, in _symlink_mounted_site_components
    self._ensure_symlink(f"{MNT_PATH}/git", f"{self.site}/git")
  File "/var/tools/gerrit-initializer/initializer/tasks/init.py", line 115, in _ensure_symlink
    shutil.rmtree(target)
  File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
PermissionError: [Errno 13] Permission denied: '/var/gerrit/git'

Thanks

Aankhi

[Matthias Sohn]

The init container installs gerrit including configured plugins and lib modules.

See https://gerrit.googlesource.com/k8s-gerrit/+/refs/heads/master/container-images/gerrit-init/

The container contains some python scripts automating some tasks which aren't handled by the gerrit init command.

Looks like you have an issue with file system permissions so that these python scripts cannot

install gerrit in the mounted filesystem.

[Thomas Dräbing]

Are you using NFS? Then please try to enable the NFS workaround, especially the option to chown the NFS volumes on startup. Note, that this is only done, if the root dir of the volume does not match the expected permissions. Thus, the performance impact should be minimal.

Kubernetes support for NFS is kind of limited. Thus, this is required.

[Aankhi Talukdar]

Hi Thomas,

I have enabled the NFS workaround and the Gerrit pod is up and running now. Thanks for the help.

I have another question: I am deploying the Gerrit cluster in AWS. I have enabled Istio using the documentation: https://gerrit-review.googlesource.com/c/k8s-gerrit/+/405561/4/Documentation/operator.md#523

### Istio

The Gerrit Operator can also manage network routing configuration in the

GerritCluster. To do that we need to install one of the supported Ingress

providers like Istio.

To install Istio with default configuration download istioctl:

```sh

curl -sL https://istio.io/downloadIstioctl | sh -

```

Then install istio:

```sh

istioctl install

The istio resources are all up and running.

NAME                                        READY   STATUS    RESTARTS   AGE

pod/istio-ingressgateway-7ddcd66b54-4d8rd   1/1     Running   0          17h

pod/istiod-77fc789879-n9z6g                 1/1     Running   0          17h

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                                        AGE

service/istio-ingressgateway   LoadBalancer   10.100.254.212   ad221d25fac674fdc834ea74d61f0dcf-226077390.eu-west-1.elb.amazonaws.com   15021:31806/TCP,8080:30110/TCP,443:31613/TCP   17h

service/istiod                 ClusterIP      10.100.24.156    <none>                                                                   15010/TCP,15012/TCP,443/TCP,15014/TCP          17h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/istio-ingressgateway   1/1     1            1           17h

deployment.apps/istiod                 1/1     1            1           17h

NAME                                              DESIRED   CURRENT   READY   AGE

replicaset.apps/istio-ingressgateway-7ddcd66b54   1         1         1       17h

replicaset.apps/istiod-77fc789879                 1         1         1       17h

NAME                                                       REFERENCE                         TARGETS         MINPODS   MAXPODS   REPLICAS   AGE

horizontalpodautoscaler.autoscaling/istio-ingressgateway   Deployment/istio-ingressgateway   <unknown>/80%   1         5         1          17h

horizontalpodautoscaler.autoscaling/istiod                 Deployment/istiod                 <unknown>/80%   1         5         1          17h

However, we cannot see the ingress classes in AWS and I guess that's the reason, we are not able to access the Gerrit UI with the external IP. Do you suggest we follow some other documentation for ISTIO for AWS or are there some issues that we need to focus on?

Thanks

Aankhi

[Thomas Dräbing]

On Fri, Feb 23, 2024, 09:06 Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Istio does not use Ingress resources. Check whether the Ingress gateway service has an external IP. That should be the IP to use. You will need to configure your DNS to have an A entry for that IP.

If there is no external IP check the status of the service for errors.

[Aankhi Talukdar]

Hi Thomas,

Thanks for the confirmation. The issue is solved now.

Next, we want to add one more pod in the Replica pod. We have two nodes in the EKS cluster. While deploying the Gerrit cluster with the configuration of two replicas into two nodes, we are getting the following issue:

Failed to read NoteDb schema version1 error
        at com.google.gerrit.server.schema.NoteDbSchemaVersionCheck.start(NoteDbSchemaVersionCheck.java:90)
        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
        at com.google.gerrit.pgm.Daemon.start(Daemon.java:405)
        at com.google.gerrit.pgm.Daemon.run(Daemon.java:298)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:252)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:148)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:93)
        at Main.main(Main.java:30)

Could you let us know how to configure NoteDb inside the Gerrit cluster? Our version of Gerrit is 3.9.1.

Thanks

Aankhi

[Thomas Dräbing]

On Tue, 27 Feb 2024 at 07:16, Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Thanks for the confirmation. The issue is solved now.

Next, we want to add one more pod in the Replica pod. We have two nodes in the EKS cluster. While deploying the Gerrit cluster with the configuration of two replicas into two nodes, we are getting the following issue:

Failed to read NoteDb schema version1 error
        at com.google.gerrit.server.schema.NoteDbSchemaVersionCheck.start(NoteDbSchemaVersionCheck.java:90)
        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
        at com.google.gerrit.pgm.Daemon.start(Daemon.java:405)
        at com.google.gerrit.pgm.Daemon.run(Daemon.java:298)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:252)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:148)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:93)
        at Main.main(Main.java:30)

Could you let us know how to configure NoteDb inside the Gerrit cluster? Our version of Gerrit is 3.9.1.

NoteDB is the git-based database used by Gerrit, i.e. it lives within the `$SITE/git` directory together with your project repositories. If Gerrit can't get the NoteDB schema version, this hints on issues with reading the repositories from your NFS.

One thing I would suggest, since you now likely did multiple experiments that might have led to an inconsistent state of the Gerrit site (incomplete initialization for example) is to completely delete the GerritCluster resource and all PVs that were used by Gerrit. Also clean the NFS volume, os that you can start from a clean state. This might already help.

If not, please share the complete logs and the output of `ls -la` on the `var/gerrit/git/All-Projects` directory in one of the Gerrit pods.

Thanks,

Thomas

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/98752529-4c42-4588-b377-f6aa5b09a18dn%40googlegroups.com.

[Aankhi Talukdar]

Hi Thomas,

Thanks for your support. We are done with all the high-availability plugin tests for Gerrit-operator in the AWS EKS cluster, and everything is working well.

We want to work on a multi-site plugin for the Gerrit operator on the EKS cluster. We found many documentation on multi-site, but we are not getting any concrete idea on how to start with it. Could you please guide us on how to configure the multi-site plugin for Gerrit in the EKS cluster?

Thanks

Aankhi

[Thomas Dräbing]

On Fri, 1 Mar 2024 at 08:10, Aankhi Talukdar <tallukd...@gmail.com> wrote:

Hi Thomas,

Thanks for your support. We are done with all the high-availability plugin tests for Gerrit-operator in the AWS EKS cluster, and everything is working well.

We want to work on a multi-site plugin for the Gerrit operator on the EKS cluster. We found many documentation on multi-site, but we are not getting any concrete idea on how to start with it. Could you please guide us on how to configure the multi-site plugin for Gerrit in the EKS cluster?

Hi Aankhi,

awesome to hear that it is working for you now! Work on adding multisite support has already been started by GerritForge. They plan to use it as an alternative for the HA- setup to avoid the requirement of shared filesystems. Please check out the recording of a meeting where we discuss this plan [1]. There is also already a first series of changes that start with the implementation [2]. For questions you can either use the mailing list or the #gerrit-multi-site and #k8s-gerrit channels on Discord [3].

Are the plans of GerritForge matching with yours or are you rather interested in multi-region/cluster support? It would be great to hear about your plans! 

Best Regards,

Thomas

[1] https://www.youtube.com/watch?v=Rf-LEbXDR3I

[2] https://gerrit-review.googlesource.com/c/k8s-gerrit/+/409743 and followup changes

[3] https://discord.gg/HkGbBJHYbY

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/63156e93-12d8-4093-af8a-e8df2f05af0en%40googlegroups.com.