How to check if Master and Mirror repository are in Sync

[anish....@gmail.com]

Hello Team,

we have one master and 6 mirrors.

The replication is enabled using the plugin across all the mirrors.

Although we see that the replication is successful, there are times when the master and mirror data are not in sync or update.

Questions:

How do we ensure that the Master and Mirror data are insync. Any option apart from using the replication plugin??

How do we check if the master and mirror data are in sync and if they are not how do we list which repos are out of Sync

Thank you,

Anish

[Luca Milanesio]

On 22 Mar 2021, at 12:11, anish....@gmail.com <anish....@gmail.com> wrote:

Hello Team,

we have one master and 6 mirrors.

The replication is enabled using the plugin across all the mirrors.

Although we see that the replication is successful, there are times when the master and mirror data are not in sync or update.

Questions:

How do we ensure that the Master and Mirror data are insync. Any option apart from using the replication plugin??

The replication_log gives you that information.

How do we check if the master and mirror data are in sync and if they are not how do we list which repos are out of Sync

When the replication fails, the line with the error contains also:

- repository name

- refs that failed to sync

HTH

Luca.

Thank you,

Anish

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2487e6ac-9e3f-45db-8b1f-26e26af55bf2n%40googlegroups.com.

[anish....@gmail.com]

Hello Luca,

We are currently monitoring it but we have not seeing any "Sync Failures"

Most times we only come to know when we do a build and we find that the mirrors has stale data and the latest is on the production server.

Although, the Logs don't point out to any error.

Question:

Q1> How do we manually check if the Master and Mirrors are in Sync???

Q2> Apart from the replication plugin, can we use a manual rsync to have this done???

Thank you,

Anish

[Luca Milanesio]

On 22 Mar 2021, at 15:17, anish....@gmail.com <anish....@gmail.com> wrote:

Hello Luca,

We are currently monitoring it but we have not seeing any "Sync Failures"

Most times we only come to know when we do a build and we find that the mirrors has stale data and the latest is on the production server.

That is very strange: you do not see any failed replication events on the replication_log but they are out of sync.

Do you have downtime between the primary and replica nodes?

If yes, do you replicate projects at Gerrit startup?

Although, the Logs don't point out to any error.

Question:

Q1> How do we manually check if the Master and Mirrors are in Sync???

Just check the refdb on the bare repository of the primary and replicas, see the ‘git show-ref’ at [1].

Q2> Apart from the replication plugin, can we use a manual rsync to have this done???

Yes, but you should first investigate why that happens.

HTH

Luca.

[1] https://git-scm.com/docs/git-show-ref

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/64df1d83-6565-43db-aad0-b03477cf715fn%40googlegroups.com.

[anish....@gmail.com]

Hello Luca,

Do you have downtime between the primary and replica nodes?

If yes, do you replicate projects at Gerrit startup?

No Downtime between replication. Also, after startup we have disabled the replication.

It takes a huge toll on the server which mostly brings down gerrit so we have stopped it on restart.

Q2> Apart from the replication plugin, can we use a manual rsync to have this done???

Yes, but you should first investigate why that happens.

Thank you, We will be investigating further.................But do you suggest any instructions on running both the rsync and replication at the same time???

Thank you,

Anish

[Luca Milanesio]

On 22 Mar 2021, at 18:07, anish....@gmail.com <anish....@gmail.com> wrote:

Hello Luca,

Do you have downtime between the primary and replica nodes?

If yes, do you replicate projects at Gerrit startup?

No Downtime between replication. Also, after startup we have disabled the replication.

It takes a huge toll on the server which mostly brings down gerrit so we have stopped it on restart.

You should do a root cause analysis to understand how come that the primary and secondary replicas get out of sync when there is no downtime and no errors reported on the replication_log.

Honestly I never had that situation: whenever they were out of sync there was always a reason for it.

Does anyone push to the bare repositories (primary or replica) behind Gerrit’s back?

Q2> Apart from the replication plugin, can we use a manual rsync to have this done???

Yes, but you should first investigate why that happens.

Thank you, We will be investigating further.................But do you suggest any instructions on running both the rsync and replication at the same time???

Rsync? Can you clarify how do you replicate the primary and replicas?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bd6d105a-c588-46fd-9eb9-78ff0ce3c38bn%40googlegroups.com.

[anish anil]

Does anyone push to the bare repositories (primary or replica) behind Gerrit’s back?

Not that we are aware off.............We had issues in the past when this happened when a user who did a push and delete from the gerrit repo directly  and has got the server down.

After that we have restricted all Server access. So nobody can do any direct push to the master gerrit Repo.

Rsync? Can you clarify how do you replicate the primary and replicas?

We are currently using the replication plugin --> The entries are mainted in the replication.config file

[gerrit]
        replicateOnStartup = false
        defaultForceUpdate = false
        autoReload = true

[remote "Bangalore-grtmir02"]
        url = ger...@grtmir02.ad.hm.com:/data/gerrit/git/${name}.git
        push = +refs/heads/*:refs/heads/*
        push = +refs/tags/*:refs/tags/*
        push = +refs/changes/*:refs/changes/*
        projects = ^(?:(?!google/aosp/).)*$
        timeout = 70
#        replicationDelay = 5
        replicationDelay = 15
        authGroup = Administrators
        threads = 4
        mirror = true
        rescheduleDelay = 5
[remote "MB-grtmir01"]
        url = ger...@grtmir01.ad.hm.com:/data/gerrit/git/${name}.git
        push = +refs/heads/*:refs/heads/*
        push = +refs/tags/*:refs/tags/*
        push = +refs/changes/*:refs/changes/*
        projects = ^(?:(?!google/aosp/).)*$
        timeout = 45
        threads = 4
#        replicationDelay = 5
        replicationDelay = 15
        authGroup = Administrators
        mirror = true
        rescheduleDelay = 5

*********************

The whole intention is to foolproof this Master-mirror replication.

We want to frame a mechanism which will force move the data from Master-mirro even if the replication plugin fails to do so.... 

Thank you,

Anish

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/96B19187-04A3-4625-8FB8-4D7EA97BD370%40gmail.com.

[Sven Selberg]

I don't have a direct answer but since I have been given this some thought previously I thought I'd give my perspective.

It depends on how you define "in sync", generally it's not interesting, if the mirror will catch up eventually

you are only faced with a soft timing problem that there are plenty of workarounds for.

As long as the primary and mirror doesn't share file-system, and the primary is active, primary and mirror will

very seldom be in total sync.

This makes it hard to notice if a difference between primary and mirror is caused by:

* an error

* replication not having caught up yet

* the ref being updated between you read the state from primary and you read the state from mirror.

The only way to really check consistency is to:

1. Set primary in read-only

2. Wait for all ongoing replication to finish

And then:

    for all project,ref in primary.intersection(mirror):

        assert(sha1_of(primary, project, ref) is equal to sha1_of(mirror, project, ref)

but that's fairly intrusive and as Luca pointed out:

If you don't see any problems chances are you don't have any.

If you see problems you probably get a better ROI from trying to find and fix the root cause then determine momentary

consistency.

I know from experience that customers/stakeholders may come with the "low hanging fruit" argument of "why don't

you just check for consistency!?" when they experience issues with primary and mirror not being in sync but it's just

not possible to check that on a running system.

If constant consistency is a hard requirement you probably need to switch to a multi-primary setup so that you can wait for

consistency within the scope of the individual git-transaction.

If you choose to go this route Luca is the expert.

Thank you,

Anish

[Luca Milanesio]

Thanks Sven for raising this to attention, because that’s exactly what I was thinking about.

In multi-site we heave a metric that measures “how far behind” is one node towards the global-refdb (points to the latest version of the refs).

When we graph that information we actually see in real terms what you say: only over the weekend when the traffic is low they are mostly in sync, otherwise there are always “out of sync” in one form or another.

I typically describe replication like a big river and nodes like bridges crossing the river.

If you put some different colour paper boats at the source you will see that all will flow underneath all bridges, but if you take a snapshot in time, every bridge sees a different colours of boats.

If you check all the refs of all the repos (that is basically what the multi-site plugin does) and you graph them, you’ll see that happening.

Hope that explains why that happens.

Luca.

Thank you,

Anish

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ae98077b-6987-4ae2-9f80-73abf343646an%40googlegroups.com.