Gerrit and AOSP Mirror

[Sriram G]

Hello everybody, 

I am following the steps mentioned in the following thread

https://groups.google.com/g/repo-discuss/c/FS0kXLALrCU/m/bHB8chiSAQAJ

I am facing an issue. The AOSP mirror could be synced using the following command

repo init -u https://android.googlesource.com/platform/manifest -b android-13.0.0_r30 --mirror

repo sync

Since we only want to have the Android 13 as the base, I wanted to push refs corresponding to those only. As a simple test I tried pushing selected refs for frameworks/base (about 4.7GB db size)  using the following command. The idea was to get this working before I do a "repo forall"

git push ssh://gerritadmin@localhost:29418/framework/base +refs/heads/android13-qpr* +refs/tags/android-13*

I am encountering multiple issues and have been iterating. 

1. android-13.0.0_r20 -> android-13.0.0_r20 (prohibited by Gerrit: not permitted: forge committer) - I worked around this by not pushing the tags and only the heads

2. android13-qpr1-s1-release (more than 10000 commits, and skip-validation not set - This I could work around by passing an option

3. I once again got an error stating the skip-validation is not enabled on Gerrit. I had to enable Forge Server Identity

4. And now I am getting a timeout limit

I have the following question:

1. Is there a standard Gerrit Configuration available that is recommended for large projects such as AOSP Mirroring

2. Does anybody have any experiences / learnings / best practices to share with setting up AOSP mirroring?  

3. Any suggestions on how the first problem can be addressed. I can confirm that "Forge Committer Identity" is allowed for Administrators

I know the approach works as I could do it successfully for simpler repos that I have experimented with. 

Thanks & Regards, 

Sriram

[Christian Gagneraud]

Hi Sriram,

On Thu, 2 Mar 2023 at 08:44, Sriram G <mgs...@gmail.com> wrote:
>
> Hello everybody,
> I am following the steps mentioned in the following thread
>
> https://groups.google.com/g/repo-discuss/c/FS0kXLALrCU/m/bHB8chiSAQAJ
>
> I am facing an issue. The AOSP mirror could be synced using the following command

We use our own script for mirroring but we had to tweak access
settings on gerrit, see below

> repo init -u https://android.googlesource.com/platform/manifest -b android-13.0.0_r30 --mirror
>
> repo sync
>
>
> Since we only want to have the Android 13 as the base, I wanted to push refs corresponding to those only. As a simple test I tried pushing selected refs for frameworks/base (about 4.7GB db size) using the following command. The idea was to get this working before I do a "repo forall"
>
> git push ssh://gerritadmin@localhost:29418/framework/base +refs/heads/android13-qpr* +refs/tags/android-13*
>
>
> I am encountering multiple issues and have been iterating.
>
> 1. android-13.0.0_r20 -> android-13.0.0_r20 (prohibited by Gerrit: not permitted: forge committer) - I worked around this by not pushing the tags and only the heads

we have the following settings for refs/heads/* and refs/tags/*
Allowed for service users:
Create Annotated Tag
Forge Author Identity
Forge Committer Identity
Forge Server Identity
Push Merge Commit
Create Reference
Create Signed Tag
Create Annotated Tag

> 2. android13-qpr1-s1-release (more than 10000 commits, and skip-validation not set - This I could work around by passing an option
> 3. I once again got an error stating the skip-validation is not enabled on Gerrit. I had to enable Forge Server Identity
> 4. And now I am getting a timeout limit

use these git options when pushing:
-o deadline=10m -o skip-validation

We have this setting on gerrit side:
git config -f /path/to/gerrit/etc/gerrit.config receive.timeout 10min

>
> I have the following question:
> 1. Is there a standard Gerrit Configuration available that is recommended for large projects such as AOSP Mirroring
> 2. Does anybody have any experiences / learnings / best practices to share with setting up AOSP mirroring?
> 3. Any suggestions on how the first problem can be addressed. I can confirm that "Forge Committer Identity" is allowed for Administrators

Use a dedicated user that belongs to the "service-users" group.
Grant the permissions above to the service-users group

> I know the approach works as I could do it successfully for simpler repos that I have experimented with.

Our first attempt was to have a full mirror of
https://android.googlesource.com/platform/manifest, 800GB.
Now we do partial mirror, we only mirror a few AOSP branches and their
associated tags.

Basic instructions for full mirror
---------------------------------------
repo init -u https://android.googlesource.com/platform/manifest --mirror
repo sync
repo forall -c 'echo $REPO_PATH; ssh gerrit create-project
aosp/$REPO_PATH --owner AOSP --parent aosp || echo failed;'
repo forall -c 'echo $REPO_PATH; git push -o deadline=10m -o
skip-validation ssh://gerrit/aosp/$REPO_PATH +refs/heads/*
+refs/tags/* || echo failed'
----------------------------------

We're now using a python script for our partial mirror.

Hope this helps.

Chris

[Sriram G]

Hi Christian, 

Thanks a lot. This worked and I have been able to successfully push everything into my Gerrit instance.

On Thursday, March 2, 2023 at 5:45:13 AM UTC+5:30 Christian Gagneraud wrote:

Hi Sriram,

On Thu, 2 Mar 2023 at 08:44, Sriram G <mgs...@gmail.com> wrote:
>
> Hello everybody,
> I am following the steps mentioned in the following thread
>
> https://groups.google.com/g/repo-discuss/c/FS0kXLALrCU/m/bHB8chiSAQAJ
>
> I am facing an issue. The AOSP mirror could be synced using the following command

We use our own script for mirroring but we had to tweak access
settings on gerrit, see below

 I am also contemplating a script as there are some additional steps and checks that I want to do. I applied the tweaks you mentioned and it worked perfectly

 

> repo init -u https://android.googlesource.com/platform/manifest -b android-13.0.0_r30 --mirror
>
> repo sync
>
>
> Since we only want to have the Android 13 as the base, I wanted to push refs corresponding to those only. As a simple test I tried pushing selected refs for frameworks/base (about 4.7GB db size) using the following command. The idea was to get this working before I do a "repo forall"
>
> git push ssh://gerritadmin@localhost:29418/framework/base +refs/heads/android13-qpr* +refs/tags/android-13*
>
>
> I am encountering multiple issues and have been iterating.
>
> 1. android-13.0.0_r20 -> android-13.0.0_r20 (prohibited by Gerrit: not permitted: forge committer) - I worked around this by not pushing the tags and only the heads

we have the following settings for refs/heads/* and refs/tags/*
Allowed for service users:
Create Annotated Tag
Forge Author Identity
Forge Committer Identity
Forge Server Identity
Push Merge Commit
Create Reference
Create Signed Tag
Create Annotated Tag

> 2. android13-qpr1-s1-release (more than 10000 commits, and skip-validation not set - This I could work around by passing an option
> 3. I once again got an error stating the skip-validation is not enabled on Gerrit. I had to enable Forge Server Identity
> 4. And now I am getting a timeout limit

use these git options when pushing:
-o deadline=10m -o skip-validation

Agree. I had figured this out from Gerrit Documentation. Your tweaks helped me with pushing the tags as well without any trouble 

We have this setting on gerrit side:
git config -f /path/to/gerrit/etc/gerrit.config receive.timeout 10min

>
> I have the following question:
> 1. Is there a standard Gerrit Configuration available that is recommended for large projects such as AOSP Mirroring
> 2. Does anybody have any experiences / learnings / best practices to share with setting up AOSP mirroring?
> 3. Any suggestions on how the first problem can be addressed. I can confirm that "Forge Committer Identity" is allowed for Administrators

Use a dedicated user that belongs to the "service-users" group.
Grant the permissions above to the service-users group

> I know the approach works as I could do it successfully for simpler repos that I have experimented with.

Our first attempt was to have a full mirror of
https://android.googlesource.com/platform/manifest, 800GB.
Now we do partial mirror, we only mirror a few AOSP branches and their
associated tags.

 

I will eventually go down this path for optimization, I think 

[Mk]

Hello Team,

 I have mirrored the AOSP source code into our local system using Christian suggested bash method, But push step fails with below errors.

Pushing device/linaro/dragonboard-kernel to Gerrit...
Failed to push device/linaro/dragonboard-kernel

Push command used in our script as follows - repo forall -c 'echo $REPO_PATH; git push -o deadline=10m -o skip-validation ssh://gerrit/aosp/$REPO_PATH +refs/heads/* +refs/tags/* || echo failed'

Please let me know what i'm missing here?

[Mohan]

I've updated the receive.timeout to 30 minutes in the gerrit.config file, but I'm still encountering the same error when trying to push. Can you please assist me in successfully pushing to our Gerrit instance?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/zC4PmfSLoUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0964cab6-9a7f-4f62-a9f5-3b2f68a6675dn%40googlegroups.com.

[Mk]

Even with the single repository, i'm facing the same issue while trying to push our local gerrit server.

repo init -u https://android.googlesource.com/platform/manifest -b android-14.0.0_r9 --mirror
repo sync device/generic/car
ssh -T -p 29418 gerri...@local-gerrit.com gerrit create-project aosp/device/generic/car
repo forall -c '
  echo "Pushing $REPO_PATH to Gerrit..."
  git push -o deadline=20m -o skip-validation ssh://gerri...@local-gerrit.com:29418/aosp/device/generic/car +refs/heads/* +refs/tags/* || echo "Failed to push $REPO_PATH"

Pushing device/generic/car to Gerrit...
Failed to push device/generic/car

Verified the network connectivity to ensure there are no connectivity issues. However i have not been able to resolve the issue. i would appreciate any assistance or guidance you can provide to help successfully push the changes to our local gerrit will be helpful.

[Mk]

While trying to push to our local gerrit, i could see only the below messages.

[2024-03-08T11:45:54.657Z] [ReceiveCommits-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2ebc267a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@24444d0a[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@2a83a7f2]]]-for-SSH git-receive-pack /aosp/device/generic/car (gerrit-user)] INFO  com.googlesource.gerrit.plugins.hooks.HookFactory : hooks.refUpdateHook resolved to /var/gerrit/hooks/ref-update [CONTEXT RECEIVE_ID="aosp/device/generic/car-1709898354640-135b127d" project="aosp/device/generic/car" request="GIT_RECEIVE" ]

[2024-03-08T11:45:55.134Z] [SSH git-receive-pack /aosp/device/generic/car (gerrit-user)] INFO  com.google.gerrit.server.git.MultiProgressMonitor : Processing changes: refs: 595 (\) [CONTEXT ratelimit_period="1 MINUTES" ]

[Mk]

Finally i could able to solve the problem with verbose mode enabled and i had to add push permission under refs/tags/* reference.