Mirroring from github

[Joachim Holst]

Hi all,

Due to some interesting reasons, I would need to mirror several git repositories from Github into my local Gerrit instance.

I do assume that this should be possible with the pull-replication plugin (i've seen references that it should be possible in other places), but since I'm most likely stupid, I haven't managed to figure out how to do it. :-(

In Gerrit I've set up a "subdirectory" called "Mirrors" which is where I'd like to keep mirrored stuff.

As an example, I'd like to mirror curl (https://github.com/curl/curl.git) from GitHub and keep it in "Mirrors/curl" on my local Gerrit system.

Could someone who is more knowledgeable than me help me with a "pull replication from Github for real idiots" instruction?

Have been trying to figure this out on my own for several weeks now without success :-(

If the pull-replication plugin can't do it, then I'd appreciate any other way to implement simple mirroring to gerrit from external non-gerrit repository.

BRs,
/Jocke!

[Luca Milanesio]

Have you tried the sample documented at [1]?

Luca.

[1] https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/src/main/resources/Documentation/config.md#enabling-replication

BRs,
/Jocke!

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f8e9dbb1-51fc-4166-853e-1301b476c7f1n%40googlegroups.com.

[Joachim Holst]

Hi Luca,

Thanks for your reply.

Before anything else, if my answer seems offensive, I do apologise. I've had a real shitty day which affects my writing skills quite negatively :-(

I've been RTFM:ing for weeks and experimenting for days. I'm no Gerrit master, nor a full time admin who spends 8h/day taking care of the system. I'm a happy amateur who's managed to get it to work. 

I'm currently running on Gerrit 3.5.0-RC0 if that helps..

The referenced example seems to relate to ssh mirroring, which I don't intend to do if it can be avoided.

If required, how do I create an authgroup and assign an SSH key to one of those? Creating groups in Gerrit is simple but I've never heard that you can assign an SSH key to a group. I have no control over LDAP/AD groups, so I can't create extra users for this, and IT doesn't allow it either.

From my point of view, the example seems to be missing one or more key elements which have gotten me stuck. :-( 

• How do I get the ${name} replaced with something useful? I know that it's pulled from Gerrit somehow, but how does the plugin know from where to pull it?
  
• How do I tell the plugin which git repos to mirror and where to put the mirrors? I don't want to mirror stuff into the root of Gerrit.
• Do I first have to clone the repository manually, push it up to a new repo in Gerrit and then expect that the plugin "automagically" decides to update the mirror?
  
• As stated, I created a new repo on Gerrit called "Mirrors" that serves as a parent for other repositories. I want to place all my mirrors under that repository so that I know where to find them.
  

All I ask for is a very basic "step-by-step" instruction on how to get this working using the curl example above. Once I've got that, I most likely can take care of the rest by myself.

BRs,

/Jocke!

[Luca Milanesio]

Hi Joachim,
Answers inline below.

> On 25 Aug 2022, at 17:13, Joachim Holst <sm7...@gmail.com> wrote:
>
> Hi Luca,
>
> Thanks for your reply.
>
> Before anything else, if my answer seems offensive, I do apologise. I've had a real shitty day which affects my writing skills quite negatively :-(
>
> I've been RTFM:ing for weeks and experimenting for days.

I guess RTFM stands for “Read The Fantastic Manual” :-)
I know the documentation can be overwhelming at times, I’ll give you the direct pointers that what parts are relevant for you in the references below.

> I'm no Gerrit master, nor a full time admin who spends 8h/day taking care of the system. I'm a happy amateur who's managed to get it to work.

Replication can be tricky sometimes, but I’m sure you’ll be able to get it to work.

>
> I'm currently running on Gerrit 3.5.0-RC0 if that helps..

I’d not recommend to run on RCs, and definitely not an RC0. Any reason for not using the latest Gerrit? (v3.6.1)

> The referenced example seems to relate to ssh mirroring, which I don't intend to do if it can be avoided.

Sure, you can use any Git URL there, it doesn’t have to be Git/SSH.

> If required, how do I create an authgroup and assign an SSH key to one of those? Creating groups in Gerrit is simple but I've never heard that you can assign an SSH key to a group. I have no control over LDAP/AD groups, so I can't create extra users for this, and IT doesn't allow it either.

You can use Git/HTTPS and set the credentials in secure.config, exactly as you do with the replication plugin (see [2]).

>
> From my point of view, the example seems to be missing one or more key elements which have gotten me stuck. :-(
> • How do I get the ${name} replaced with something useful? I know that it's pulled from Gerrit somehow, but how does the plugin know from where to pull it?

The ${name} is a placeholder that is replaced automatically by Gerrit with the actual project name.

Example:
[remote "myremote”]
url = ger...@host-two.example.com:/some/path/${name}.git

Would be translated automatically at runtime to:
url = ger...@host-two.example.com:/some/path/repo1.git
url = ger...@host-two.example.com:/some/path/repo2.git
url = ger...@host-two.example.com:/some/path/repo3.git

For the projects ‘repo1’, ‘repo2’ and ‘repo3’

> • How do I tell the plugin which git repos to mirror and where to put the mirrors? I don't want to mirror stuff into the root of Gerrit.

The list of projects (or regex) can be defined using the ‘remote.NAME.projects’ config.

For the above example, if you want to replicate the repos ‘repo1’, ‘repo2’ and ‘repo3’ you can add:

[remote "myremote”]
projects = ^repo[1-3]

> • Do I first have to clone the repository manually, push it up to a new repo in Gerrit and then expect that the plugin "automagically" decides to update the mirror?

Yes, the local repository needs to exist, even if that is empty, otherwise Gerrit will not find any repository to mirror.

> • As stated, I created a new repo on Gerrit called "Mirrors" that serves as a parent for other repositories. I want to place all my mirrors under that repository so that I know where to find them.

> All I ask for is a very basic "step-by-step" instruction on how to get this working using the curl example above. Once I've got that, I most likely can take care of the rest by myself.

In your example, to mirror https://github.com/curl/curl.git locally you would need to have this section:

[remote "curl-on-github”]
url = https://github.com/curl/curl.git
fetch = +refs/heads/*:refs/heads/*
projects = Mirror/curl

P.S. Bear in mind that when using the pull-replication from a remote end that is not Gerrit, the triggering of the replication needs to happen manually using the command ‘pull-replication start’, see [3] for more details.

HTH

Luca.

>
> BRs,
> /Jocke!
>
>
> On Thu, Aug 25, 2022 at 2:46 PM Luca Milanesio <luca.mi...@gmail.com> wrote:
>
>
>> On 25 Aug 2022, at 06:46, Joachim Holst <sm7...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Due to some interesting reasons, I would need to mirror several git repositories from Github into my local Gerrit instance.
>>
>> I do assume that this should be possible with the pull-replication plugin (i've seen references that it should be possible in other places), but since I'm most likely stupid, I haven't managed to figure out how to do it. :-(
>>
>> In Gerrit I've set up a "subdirectory" called "Mirrors" which is where I'd like to keep mirrored stuff.
>>
>> As an example, I'd like to mirror curl (https://github.com/curl/curl.git) from GitHub and keep it in "Mirrors/curl" on my local Gerrit system.
>>
>> Could someone who is more knowledgeable than me help me with a "pull replication from Github for real idiots" instruction?
>>
>> Have been trying to figure this out on my own for several weeks now without success :-(
>>
>> If the pull-replication plugin can't do it, then I'd appreciate any other way to implement simple mirroring to gerrit from external non-gerrit repository.
>
> Have you tried the sample documented at [1]?
>
> Luca.
>
> [1] https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/src/main/resources/Documentation/config.md#enabling-replication

[2] https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/resources/Documentation/config.md#file-2
[3] https://gerrit.googlesource.com/plugins/pull-replication/+/refs/heads/master/src/main/resources/Documentation/cmd-start.md

[Joachim Holst]

Hi Luca,

Thanks for your clarifications. I think that those finally connected all the dots for me.

Right now I'm off-site for a couple of days, but I'll give it a try as soon as I get back.

Reason for running the Gerrit version that I am, is that it was the latest version that supported all the plugins I required at the time I could do the upgrade. Stupid reason, I know :-)

Regarding the starting of the sync manually, is that a one-time action or does it have to be done repeatedly via a cron job or similar?

BRs,

/Jocke!

[Luca Milanesio]

On 26 Aug 2022, at 07:50, Joachim Holst <sm7...@gmail.com> wrote:

Hi Luca,

Thanks for your clarifications. I think that those finally connected all the dots for me.

Right now I'm off-site for a couple of days, but I'll give it a try as soon as I get back.

Reason for running the Gerrit version that I am, is that it was the latest version that supported all the plugins I required at the time I could do the upgrade. Stupid reason, I know :-)

Regarding the starting of the sync manually, is that a one-time action or does it have to be done repeatedly via a cron job or similar?

A crontab would be required, because Gerrit won’t have the event to understand when something has changed on GitHub.

P.S. Don’t put your crontab interval too tight or GitHub would start throttling you.

[Joachim Holst]

Hi again Luca,

Tried this on my home system (running Gerrit 3.5.2) and it did sync curl on startup (have enabled "replicateOnStartup") and "pull replication start" also worked as expected.

Thank you _VERY_ much for your kind help!

BRs,

/Jocke!