Replication issue on gerrit master

[tech....@gmail.com]

Hello Team,

We have a situation. 

We have gerrit master mirror setup. 1 master 5 mirrors and arround 5000 projects.

Gerrit Version 3.2.11

All of the sudden, Gerrit starts replicating all of the repositories in one shot once or twice a day. and Task are getting queued (28000 queue)

We have checked no automation or rest call or ssh call has been placed to trigger that. Even some sleeping repositories(unused) are also getting triggered for replication.

Is there any way to monitor or find out the root cause or there could be a bug sleeping it the application itself? 

[Martin Fick]

On 6/24/22 1:35 AM, tech....@gmail.com wrote:
> Hello Team,All of the sudden, Gerrit starts replicating all of the

> repositories in one shot once or twice a day. and Task are getting
> queued (28000 queue)
> We have checked no automation or rest call or ssh call has been placed
> to trigger that. Even some sleeping repositories(unused) are also
> getting triggered for replication.

This very much sounds like someone/thing is calling the ssh 'replication
all' command.

Could the application itself be restarting somehow?

> Is there any way to monitor or find out the root cause or there could
> be a bug sleeping it the application itself?

Have you scoured the logs looking for this?Also, check use show-queue
around that time to see if something has triggered this,

-Martin

[Matthias Sohn]

On Fri, Jun 24, 2022 at 5:50 PM Martin Fick <quic_...@quicinc.com> wrote:

On 6/24/22 1:35 AM, tech....@gmail.com wrote:
> Hello Team,All of the sudden, Gerrit starts replicating all of the
> repositories in one shot once or twice a day. and Task are getting
> queued (28000 queue)
> We have checked no automation or rest call or ssh call has been placed
> to trigger that. Even some sleeping repositories(unused) are also
> getting triggered for replication.

This very much sounds like someone/thing is calling the ssh 'replication
all' command.

Could the application itself be restarting somehow?

Did you configure gerrit.replicateOnStartup = true ?

 

> Is there any way to monitor or find out the root cause or there could
> be a bug sleeping it the application itself?
Have you scoured the logs looking for this?Also, check use show-queue
around that time to see if something has triggered this,

-Martin

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/26912c32-9ae3-b2c9-f9fb-add043ac8ee7%40quicinc.com.

[Shad Perwez]

Hi all,

Answering your question.

1. We have analysed the logs, no user has done replication start all.

2. It was 3 times since last week. And we didn't git anything from the logs why mysteriously replication strted.

3.  Now it looks stable enough after , we let the replication complete.

4. Replication on strtup is on false. And application is never been strted 

[Martin Fick]

On 6/26/22 6:32 AM, Shad Perwez wrote:

Please answer inline, or it is very hard to figure out what you are specifically replying to.

On Fri, Jun 24, 2022 at 5:50 PM Martin Fick <quic_...@quicinc.com> wrote:

On 6/24/22 1:35 AM, tech....@gmail.com wrote:
> Hello Team,All of the sudden, Gerrit starts replicating all of the
> repositories in one shot once or twice a day. and Task are getting
> queued (28000 queue)
> We have checked no automation or rest call or ssh call has been placed
> to trigger that. Even some sleeping repositories(unused) are also
> getting triggered for replication.
 

> Is there any way to monitor or find out the root cause or there could

> be a bug sleeping it the application itself?
Have you scoured the logs looking for this?Also, check use show-queue
around that time to see if something has triggered this,

It's not clear whether you responded to that last suggestion? What do these replication push tasks look like, do they include a specific ref, or are they for "..al.." refs?

The replication plugin is extremely well instrumented with log messages, so if you are willing to bump up the log level before this happens, you are likely to get a good hint as to what is triggering this,

-Martin

[tech....@gmail.com]

Hello,

Can you suggest a more deep logging configuration for replication in gerrit?

As we have no clue what triggering this.

For now, we have not seen this bulk replication from last one week. 

[Matthias Sohn]

On Fri, Jul 1, 2022 at 9:01 AM tech....@gmail.com <tech....@gmail.com> wrote:

Hello,

Can you suggest a more deep logging configuration for replication in gerrit?

As we have no clue what triggering this.

For now, we have not seen this bulk replication from last one week. 

enabling debug log on

https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/java/com/googlesource/gerrit/plugins/replication/PushOne.java

should provide details

 

On Monday, 27 June 2022 at 02:32:04 UTC+5:30 quic_...@quicinc.com wrote:

On 6/26/22 6:32 AM, Shad Perwez wrote:

Please answer inline, or it is very hard to figure out what you are specifically replying to.

On Fri, Jun 24, 2022 at 5:50 PM Martin Fick <quic_...@quicinc.com> wrote:

On 6/24/22 1:35 AM, tech....@gmail.com wrote:
> Hello Team,All of the sudden, Gerrit starts replicating all of the
> repositories in one shot once or twice a day. and Task are getting
> queued (28000 queue)
> We have checked no automation or rest call or ssh call has been placed
> to trigger that. Even some sleeping repositories(unused) are also
> getting triggered for replication.
 

> Is there any way to monitor or find out the root cause or there could

> be a bug sleeping it the application itself?
Have you scoured the logs looking for this?Also, check use show-queue
around that time to see if something has triggered this,

It's not clear whether you responded to that last suggestion? What do these replication push tasks look like, do they include a specific ref, or are they for "..al.." refs?

The replication plugin is extremely well instrumented with log messages, so if you are willing to bump up the log level before this happens, you are likely to get a good hint as to what is triggering this,

-Martin

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bfe033f4-e010-49be-812b-87b5b1896c2cn%40googlegroups.com.