Pushing big AOSP repositories

562 views
Skip to first unread message

Xut

unread,
Nov 1, 2023, 2:00:43 AM11/1/23
to Repo and Gerrit Discussion

Dariusz Luksza

unread,
Nov 1, 2023, 8:02:33 AM11/1/23
to Xut, Repo and Gerrit Discussion
On Wed, Nov 1, 2023 at 6:00 AM Xut <awoi...@gmail.com> wrote:


There's another way of importing repositories, especially huge ones, you can just simply rsync or scp into Gerrit's git root.

First of all, be sure that you create the destination repository upfront, as it will require "meta" branches in order to be visible in Gerrit.

Secondly, be sure that the source repository is not used by anyone, having a local bare clone will do. You only want to copy git data, without the worktree.

Finally do the rsync or scp of the repository data into `$gerrit_site/git/$repo_name`. When the copying process is over, you should be able to use the imported repo.

This method bypasses the Gerrit SSH and HTTP servers, as well as internal processing, ACLs etc. You may consider running a Gerrit GC for that repo afterwards to create bitmaps.

Hope this helps.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/be0b22a7-aaac-49c3-81d0-8ec4198e6ccbn%40googlegroups.com.


--
Blog: https://luksza.org http://habitchallenge.co
LinkedIn: https://linkedin.com/in/dluksza

P.S. Have you checked the Habit Challenge app? You should ;)

Xut

unread,
Nov 1, 2023, 8:56:38 PM11/1/23
to Repo and Gerrit Discussion
On Wednesday, November 1, 2023 at 9:02:33 PM UTC+9 Dariusz Luksza wrote:

There's another way of importing repositories, especially huge ones, you can just simply rsync or scp into Gerrit's git root.

First of all, be sure that you create the destination repository upfront, as it will require "meta" branches in order to be visible in Gerrit.

Secondly, be sure that the source repository is not used by anyone, having a local bare clone will do. You only want to copy git data, without the worktree.

Finally do the rsync or scp of the repository data into `$gerrit_site/git/$repo_name`. When the copying process is over, you should be able to use the imported repo.

This method bypasses the Gerrit SSH and HTTP servers, as well as internal processing, ACLs etc. You may consider running a Gerrit GC for that repo afterwards to create bitmaps.

Hope this helps.


I had actually gave it a try but I couldn't make it work so I assumed such approach was not supported. This is how I'm trying, do I miss something?
$scp -P 29418 -r local-mirror-project.git my-...@gerrit.myserver.com:/var/gerrit/git
subsystem request failed on channel 0
scp: Connection closed

This are the logs in Gerrit:
[2023-11-02T07:57:17.299+09:00] [sshd-SshDaemon[2da16d31](port=22)-nio2-thread-3] WARN  org.apache.sshd.server.channel.ChannelSession : handleSubsystem(ChannelSession[id=0, recipient=0]-ServerSessionImpl[my-user@/X.X.X.X:50332]) No factories for subsystem: sftp

I tried as well with -O option as mentioned here without success
https://bugreports.qt.io/browse/QTQAINFRA-4974?focusedCommentId=660149&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

menghu...@gmail.com

unread,
Nov 2, 2023, 1:42:13 AM11/2/23
to Repo and Gerrit Discussion
I would suggest that you bypass the Gerrit service and use native git directly to operate the bare repo in the Gerrit backend.

Matthias Sohn

unread,
Nov 2, 2023, 4:27:29 AM11/2/23
to Xut, Repo and Gerrit Discussion
On Thu, Nov 2, 2023 at 1:56 AM Xut <awoi...@gmail.com> wrote:


On Wednesday, November 1, 2023 at 9:02:33 PM UTC+9 Dariusz Luksza wrote:

There's another way of importing repositories, especially huge ones, you can just simply rsync or scp into Gerrit's git root.

First of all, be sure that you create the destination repository upfront, as it will require "meta" branches in order to be visible in Gerrit.

Secondly, be sure that the source repository is not used by anyone, having a local bare clone will do. You only want to copy git data, without the worktree.

Finally do the rsync or scp of the repository data into `$gerrit_site/git/$repo_name`. When the copying process is over, you should be able to use the imported repo.

This method bypasses the Gerrit SSH and HTTP servers, as well as internal processing, ACLs etc. You may consider running a Gerrit GC for that repo afterwards to create bitmaps.

Hope this helps.


I had actually gave it a try but I couldn't make it work so I assumed such approach was not supported. This is how I'm trying, do I miss something?
$scp -P 29418 -r local-mirror-project.git my-...@gerrit.myserver.com:/var/gerrit/git

On port 29418 Gerrit's ssh daemon is listening, instead you should use the host's ssh port 22 to scp the repo behind Gerrit's back.
 
subsystem request failed on channel 0
scp: Connection closed

This are the logs in Gerrit:
[2023-11-02T07:57:17.299+09:00] [sshd-SshDaemon[2da16d31](port=22)-nio2-thread-3] WARN  org.apache.sshd.server.channel.ChannelSession : handleSubsystem(ChannelSession[id=0, recipient=0]-ServerSessionImpl[my-user@/X.X.X.X:50332]) No factories for subsystem: sftp

I tried as well with -O option as mentioned here without success
https://bugreports.qt.io/browse/QTQAINFRA-4974?focusedCommentId=660149&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Estelle Brunk

unread,
Nov 2, 2023, 4:50:05 AM11/2/23
to Xut, Repo and Gerrit Discussion
Tell us more?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/be0b22a7-aaac-49c3-81d0-8ec4198e6ccbn%40googlegroups.com.

menghu...@gmail.com

unread,
Nov 2, 2023, 9:02:52 AM11/2/23
to Repo and Gerrit Discussion
The concept is to run Git via the "Linux ssh daemon", not Gerrit's ssh.
cd local-mirror-project
git remote add gerritbackend gerritlocaluser@gerrit.myserver.com:/var/gerrit/repositories/your_git_project.git
git push gerritbackend local-branch:refs/heads/your-reference-branch
but please do it very carefully and backup your-git-project.git before actually push it, since you're bypassing Gerrit ACL's control.

Xut

unread,
Nov 3, 2023, 8:51:44 PM11/3/23
to Repo and Gerrit Discussion
At the end I was able to find the root cause so let me write it down here in case someone ends in this thread which a similar issue.
The Gerrit server is hosted in Google Cloud behind a Load Balancer which its corresponding "Backend service"

I had missed the fact that these backend services have a 30s timeout [1] which was causing the "502 Bad Gateway error".

After increasing it (via Terraform [2]) the push took longer until it failed with a "Server Deadline Exceeded (receive.timeout=15m)"

This error was straight forward though, I increased 'receive.timeout' in 'gerrit.config' and I was finally able to push all AOSP repositories.

Thank you all for your help!

[1]: https://cloud.google.com/load-balancing/docs/backend-service#timeout-setting
[2]: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_backend_service#timeout_sec
Reply all
Reply to author
Forward
0 new messages