how to monitor replication.log

[ying...@gmail.com]

Hi experts,

We're trying to monitor if replication works well real-time; and will pop message to user when error happens with mail or some other tools instead of check logs under ~/review_sites/logs in gerrit host server.
1. Not sure if there is some parameters to complete it. 
2. if there is no parameter, we plan to write script to check it because it could be great for user to know how about replication work. However, not easy for us to tell when replication starts/replication stop; not sure if replication error only show up in replication.log.
Could you please help to advice?

Thanks a lot.

BR,
Ying

[Luca Milanesio]

Hi Ying,

I am pushing changes to enable real-time replication monitoring :-)

They are currently on the GitHub plugin (because I needed them) but they can be easily extracted into a separate plugin.

Main features are:

- real-time storing of last replication status on a per-project / branch / ref basis

- REST API to get the current replication status and list of errors per-project basis

- e-mail triggering when replication fails

In the meantime you can still get something out of Gerrit stream events as the replication events are sent over there as well.

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

Hi Ying,

one question: why are you interested in when the replication started / stopped?

Would you like to collect replication statistics to highlight latency problems?

Luca.

[开心果]

Hi Luca,

I think I didn't describe my question clearly.

My topic is related with replication plugin.

I installed gerrit v2.11 into ubuntu with replication plugin v2.11. Our team think replication plugin is really good, so we plan to take it into our daily work. Now I already knew about how to configure replication plugin. And I'm studying about how to handle error when it show up. Because we want to use some tricks to monitor log if there is error in real time instead of mannally check log in gerrit master server.As far as I know, seems that replication.log under ~/review_site/logs are recorded all logs for replication history. Not sure if there is other log to track replication work (for example, record when replication start/stop). I just want to know more about this. 

BR,
Ying

------------------ Original ------------------

From:  "Luca Milanesio";<luca.mi...@gmail.com>;

Date:  Wed, May 27, 2015 06:37 PM

To:  "yingsunte"<ying...@gmail.com>;

Cc:  "repo-discuss"<repo-d...@googlegroups.com>; "sy990361"<sy99...@hotmail.com>;

Subject:  Re: how to monitor replication.log

[Luca Milanesio]

Hi Ying,

you can either watch the replication log which contains all the activity (warning: you need to parse it as-is and it may change in the future) or listen to Gerrit stream events (stable JSON) which include the replication events.

The stuff that you’ll need to write to monitor replication … is what I am writing at the moment as well for the GitHub plugin :-)

Luca.

[Luca Milanesio]

Hi Ying,

see below a sample session:

lucamilanesio@skywalker-mac:github  (master)$ ssh -p 29418 lucamilanesio@localhost gerrit stream-events
{"submitter":{"name":"Luca Milanesio","email":"lu...@milanesio.org","username":"lucamilanesio"},"refUpdate":{"oldRev":"d5fa13216c9882a3d3ecfba70e1fc4f91bd79657","newRev":"c563626a3166f029c60c92c12e678ba5738b5364","refName":"refs/meta/config","project":"lucamilanesio/rome"},"type":"ref-updated","eventCreatedOn":1432807398}
{"project":"lucamilanesio/rome","ref":"refs/meta/config","targetNode":"github.com","status":"succeeded","refStatus":"OK","type":"ref-replicated","eventCreatedOn":1432807414}
{"project":"lucamilanesio/rome","ref":"refs/meta/config","nodesCount":1,"type":"ref-replication-done","eventCreatedOn":1432807414}

As you can see the replication start, status and end are provided as single-line JSON element in the stream events output.

Luca.

On 28 May 2015, at 09:18, 开心果 <7852...@qq.com> wrote:

Hi Luca,

That's what I want to know. Thanks a lot.

One more question, I know I can call "ssh -p <port> <host> gerrit stream-events" to get events, but not sure where to get JSON in gerrit server (ubuntu OS). Could you please help to advice?

BR,
Ying

------------------ 原始邮件 ------------------

发件人: "Luca Milanesio";<luca.mi...@gmail.com>;

发送时间: 2015年5月28日(星期四) 下午2:46

收件人: "开心果"<ying...@gmail.com>;

抄送: "repo-discuss"<repo-d...@googlegroups.com>; "sy990361"<sy99...@hotmail.com>;

主题: Re: how to monitor replication.log

[Luca Milanesio]

I see … you’d like something like the stream events but saved to a log file. That doesn’t exist yet but you can easily create it yourself ;-)

Luca.

On 1 Jun 2015, at 11:01, 开心果 <7852...@qq.com> wrote:

Hi Luca,

I know this is really good way to monitor replication plugin log. About how to monitor the log, we still have questions with command "ssh -p 29418 lucamilanesio@localhost gerrit stream-events". If try to write a script to call this command and we want to stop it after no new stream events added, then parse the logs. Could you please help to advice how to stop this command running inside script?

BR,
Ying

------------------ 原始邮件 ------------------

发件人: "Luca Milanesio";<luca.mi...@gmail.com>;

发送时间: 2015年5月28日(星期四) 晚上6:05

收件人: "开心果"<7852...@qq.com>;

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/0ZFK5T9DgkE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

[Ryoichi Taniguchi]

Hi Luca,

I am also in the similar situation.. Ensuring git consistency both master and slave is being required. 

It seems that similar log "ref-replicated" or "ref-replicated-done" can be printouted with gerrit stream-events, with replication plugin, 

I am just wondering, are you aware your changeset(which sha1?) on github plugin is more effective now? hope to hear soon.

warm regards

Ryoichi

2015年6月1日月曜日 19時05分19秒 UTC+9 lucamilanesio:

I see … you’d like something like the stream events but saved to a log file. That doesn’t exist yet but you can easily create it yourself ;-)

Luca.

On 1 Jun 2015, at 11:01, 开心果 <7852...@qq.com> wrote:

Hi Luca,

I know this is really good way to monitor replication plugin log. About how to monitor the log, we still have questions with command "ssh -p 29418 lucamilanesio@localhost gerrit stream-events". If try to write a script to call this command and we want to stop it after no new stream events added, then parse the logs. Could you please help to advice how to stop this command running inside script?

BR,
Ying

------------------ 原始邮件 ------------------

发件人: "Luca Milanesio";<luca.mi...@gmail.com>;

发送时间: 2015年5月28日(星期四) 晚上6:05

收件人: "开心果"<7852...@qq.com>;

抄送: "repo-discuss"<repo-dis...@googlegroups.com>; "sy990361"<sy99...@hotmail.com>;

主题: Re: how to monitor replication.log

Hi Ying,

see below a sample session:

lucamilanesio@skywalker-mac:github  (master)$ ssh -p 29418 lucamilanesio@localhost gerrit stream-events
{"submitter":{"name":"Luca Milanesio","email":"lu...@milanesio.org","username":"lucamilanesio"},"refUpdate":{"oldRev":"d5fa13216c9882a3d3ecfba70e1fc4f91bd79657","newRev":"c563626a3166f029c60c92c12e678ba5738b5364","refName":"refs/meta/config","project":"lucamilanesio/rome"},"type":"ref-updated","eventCreatedOn":1432807398}
{"project":"lucamilanesio/rome","ref":"refs/meta/config","targetNode":"github.com","status":"succeeded","refStatus":"OK","type":"ref-replicated","eventCreatedOn":1432807414}
{"project":"lucamilanesio/rome","ref":"refs/meta/config","nodesCount":1,"type":"ref-replication-done","eventCreatedOn":1432807414}

As you can see the replication start, status and end are provided as single-line JSON element in the stream events output.

Luca.

On 28 May 2015, at 09:18, 开心果 <7852...@qq.com> wrote:

Hi Luca,

That's what I want to know. Thanks a lot.

One more question, I know I can call "ssh -p <port> <host> gerrit stream-events" to get events, but not sure where to get JSON in gerrit server (ubuntu OS). Could you please help to advice?

BR,
Ying

------------------ 原始邮件 ------------------

发件人: "Luca Milanesio";<luca.mi...@gmail.com>;

发送时间: 2015年5月28日(星期四) 下午2:46

收件人: "开心果"<ying...@gmail.com>;

抄送: "repo-discuss"<repo-dis...@googlegroups.com>; "sy990361"<sy99...@hotmail.com>;

[luca.mi...@gmail.com]

In Gerrit Master the replication plugin will issue metrics as well, those would be useful for monitoring.

The GitHub plugin / master catches the replication events and store the latest replication status on the file system.

Luca

Sent from my iPhone

[Remy Bohmer]

Hi,

It actually can be done.
Setup a rabbitmq server with the rabbitmq plugin in Gerrit. Then use logstash to read from the rabbitmq server and write to file or stdout. This delivers a json stream of gerrit events.

Going one step further you can use logstash to store the event data in an elasticsearch index and use kibana or grafana to display the event data in a realtime updated graph.

In the end rabbitmq is way more reliable than gerrit streamevents as well, because the persistency guarantees you will never loose any event.

Kind regards,

Remy

Op 1 jun. 2015 12:05 schreef "Luca Milanesio" <luca.mi...@gmail.com>:

[David Pursehouse]

I'm not sure that "monitoring the replication log" is what we are really
looking for here.

The problem we've seen is that when the replication plugin is reloaded,
any events that occur during the reload are not received by the plugin,
and thus some refs don't get replicated to remotes.

On 11/27/2015 04:34 PM, Remy Bohmer wrote:
> Hi,
>
> It actually can be done.
> Setup a rabbitmq server with the rabbitmq plugin in Gerrit. Then use
> logstash to read from the rabbitmq server and write to file or stdout.
> This delivers a json stream of gerrit events.
>
> Going one step further you can use logstash to store the event data in
> an elasticsearch index and use kibana or grafana to display the event
> data in a realtime updated graph.
>
> In the end rabbitmq is way more reliable than gerrit streamevents as
> well, because the persistency guarantees you will never loose any event.
>
> Kind regards,
>
> Remy
>
> Op 1 jun. 2015 12:05 schreef "Luca Milanesio" <luca.mi...@gmail.com

> <mailto:luca.mi...@gmail.com>>:

>
> I see … you’d like something like the stream events but saved to a
> log file. That doesn’t exist yet but you can easily create it
> yourself ;-)
>
> Luca.
>
>> On 1 Jun 2015, at 11:01, 开心果 <7852...@qq.com

>> <mailto:7852...@qq.com>> wrote:
>>
>> Hi Luca,
>>
>> I know this is really good way to monitor replication plugin log.
>> About how to monitor the log, we still have questions with command
>> "ssh -p 29418 lucamilanesio@localhost gerrit stream-events". If
>> try to write a script to call this command and we want to stop it
>> after no new stream events added, then parse the logs. Could you
>> please help to advice how to stop this command running inside script?
>>
>> BR,
>> Ying
>>
>>
>> ------------------ 原始邮件 ------------------

>> *发件人:* "Luca Milanesio";<luca.mi...@gmail.com
>> <mailto:luca.mi...@gmail.com>>;
>> *发送时间:* 2015年5月28日(星期四) 晚上6:05
>> *收件人:* "开心果"<7852...@qq.com <mailto:7852...@qq.com>>;
>> *抄送:* "repo-discuss"<repo-d...@googlegroups.com
>> <mailto:repo-d...@googlegroups.com>>;
>> "sy990361"<sy99...@hotmail.com <mailto:sy99...@hotmail.com>>;
>> *主题:* Re: how to monitor replication.log

>>
>> Hi Ying,
>> see below a sample session:
>>
>> lucamilanesio@skywalker-mac:github (master)$ ssh -p 29418
>> lucamilanesio@localhost gerrit stream-events
>> {"submitter":{"name":"Luca Milanesio","email":"lu...@milanesio.org

>> <mailto:lu...@milanesio.org>","username":"lucamilanesio"},"refUpdate":{"oldRev":"d5fa13216c9882a3d3ecfba70e1fc4f91bd79657","newRev":"c563626a3166f029c60c92c12e678ba5738b5364","refName":"refs/meta/config","project":"lucamilanesio/rome"},"type":"ref-updated","eventCreatedOn":1432807398}

>> {"project":"lucamilanesio/rome","ref":"refs/meta/config","targetNode":"github.com

>> <http://github.com/>","status":"succeeded","refStatus":"OK","type":"ref-replicated","eventCreatedOn":1432807414}

>> {"project":"lucamilanesio/rome","ref":"refs/meta/config","nodesCount":1,"type":"ref-replication-done","eventCreatedOn":1432807414}
>>
>> As you can see the replication start, status and end are provided
>> as single-line JSON element in the stream events output.
>>
>> Luca.
>>
>>> On 28 May 2015, at 09:18, 开心果 <7852...@qq.com

>>> <mailto:7852...@qq.com>> wrote:
>>>
>>> Hi Luca,
>>>
>>> That's what I want to know. Thanks a lot.
>>>
>>> One more question, I know I can call "ssh -p <port> <host> gerrit
>>> stream-events" to get events, but not sure where to get JSON in
>>> gerrit server (ubuntu OS). Could you please help to advice?
>>>
>>> BR,
>>> Ying
>>>
>>> ------------------ 原始邮件 ------------------

>>> *发件人:* "Luca Milanesio";<luca.mi...@gmail.com
>>> <mailto:luca.mi...@gmail.com>>;
>>> *发送时间:* 2015年5月28日(星期四) 下午2:46
>>> *收件人:* "开心果"<ying...@gmail.com
>>> <mailto:ying...@gmail.com>>;
>>> *抄送:* "repo-discuss"<repo-d...@googlegroups.com
>>> <mailto:repo-d...@googlegroups.com>>;
>>> "sy990361"<sy99...@hotmail.com <mailto:sy99...@hotmail.com>>;
>>> *主题:* Re: how to monitor replication.log

>>>
>>> Hi Ying,
>>> you can either watch the replication log which contains all the
>>> activity (warning: you need to parse it as-is and it may change
>>> in the future) or listen to Gerrit stream events (stable JSON)
>>> which include the replication events.
>>>
>>> The stuff that you’ll need to write to monitor replication … is
>>> what I am writing at the moment as well for the GitHub plugin :-)
>>>
>>> Luca.
>>>
>>>> On 28 May 2015, at 04:07, 开心果 <ying...@gmail.com

>>>> <mailto:ying...@gmail.com>> wrote:
>>>>
>>>> Hi Luca,
>>>>
>>>> I think I didn't describe my question clearly.
>>>>
>>>> My topic is related with replication plugin.
>>>>
>>>> I installed gerrit v2.11 into ubuntu with replication plugin
>>>> v2.11. Our team think replication plugin is really good, so we
>>>> plan to take it into our daily work. Now I already knew about
>>>> how to configure replication plugin. And I'm studying about how
>>>> to handle error when it show up. Because we want to use some
>>>> tricks to monitor log if there is error in real time instead of
>>>> mannally check log in gerrit master server.As far as I know,
>>>> seems that replication.log under ~/review_site/logs are recorded
>>>> all logs for replication history. Not sure if there is other log
>>>> to track replication work (for example, record when replication
>>>> start/stop). I just want to know more about this.
>>>>
>>>> BR,
>>>> Ying
>>>>
>>>> ------------------ Original ------------------

>>>> *From: * "Luca Milanesio";<luca.mi...@gmail.com
>>>> <mailto:luca.mi...@gmail.com>>;
>>>> *Date: * Wed, May 27, 2015 06:37 PM
>>>> *To: * "yingsunte"<ying...@gmail.com
>>>> <mailto:ying...@gmail.com>>;
>>>> *Cc: * "repo-discuss"<repo-d...@googlegroups.com
>>>> <mailto:repo-d...@googlegroups.com>>;
>>>> "sy990361"<sy99...@hotmail.com <mailto:sy99...@hotmail.com>>;
>>>> *Subject: * Re: how to monitor replication.log

>>>>
>>>> Hi Ying,
>>>> one question: why are you interested in when the replication
>>>> started / stopped?
>>>> Would you like to collect replication statistics to highlight
>>>> latency problems?
>>>>
>>>> Luca.
>>>>
>>>>> On 27 May 2015, at 11:13, Luca Milanesio

>>>>> <luca.mi...@gmail.com <mailto:luca.mi...@gmail.com>> wrote:
>>>>>
>>>>> Hi Ying,
>>>>> I am pushing changes to enable real-time replication monitoring :-)
>>>>>
>>>>> They are currently on the GitHub plugin (because I needed them)
>>>>> but they can be easily extracted into a separate plugin.
>>>>>
>>>>> Main features are:
>>>>> - real-time storing of last replication status on a per-project
>>>>> / branch / ref basis
>>>>> - REST API to get the current replication status and list of
>>>>> errors per-project basis
>>>>> - e-mail triggering when replication fails
>>>>>
>>>>> In the meantime you can still get something out of Gerrit
>>>>> stream events as the replication events are sent over there as
>>>>> well.
>>>>>
>>>>> Luca.
>>>>>
>>>>>> On 27 May 2015, at 10:20, ying...@gmail.com

>>>>>> <mailto:ying...@gmail.com> wrote:
>>>>>>
>>>>>> Hi experts,
>>>>>>
>>>>>> We're trying to monitor if replication works well real-time;
>>>>>> and will pop message to user when error happens with mail or
>>>>>> some other tools instead of check logs under
>>>>>> ~/review_sites/logs in gerrit host server.
>>>>>> 1. Not sure if there is some parameters to complete it.
>>>>>> 2. if there is no parameter, we plan to write script to check
>>>>>> it because it could be great for user to know how about
>>>>>> replication work. However, not easy for us to tell when
>>>>>> replication starts/replication stop; not sure if replication
>>>>>> error only show up in replication.log.
>>>>>> Could you please help to advice?
>>>>>>
>>>>>> Thanks a lot.
>>>>>>
>>>>>> BR,
>>>>>> Ying
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> To unsubscribe, email
>>>>>> repo-discuss...@googlegroups.com

>>>>>> <mailto:repo-discuss...@googlegroups.com>

>>>>>> More info at http://groups.google.com/group/repo-discuss?hl=en
>>>>>>
>>>>>> ---
>>>>>> You received this message because you are subscribed to the
>>>>>> Google Groups "Repo and Gerrit Discussion" group.
>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>> it, send an email to repo-discuss...@googlegroups.com

>>>>>> <mailto:repo-discuss...@googlegroups.com>.

>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>
>>
>>
>> --
>> --
>> To unsubscribe, email repo-discuss...@googlegroups.com

>> <mailto:repo-discuss...@googlegroups.com>

>> More info at http://groups.google.com/group/repo-discuss?hl=en
>>
>> ---
>> You received this message because you are subscribed to a topic in
>> the Google Groups "Repo and Gerrit Discussion" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/repo-discuss/0ZFK5T9DgkE/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email
>> to repo-discuss...@googlegroups.com

>> <mailto:repo-discuss...@googlegroups.com>.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com

> <mailto:repo-discuss%2Bunsu...@googlegroups.com>

> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

[Ryoichi Taniguchi]

Thanks Luca, Remy and David,

I understand the point of the previous fix. as commented, 

task of git consistency check, caused by replication plugin's reload, is the one we face.

2015年11月27日金曜日 16時39分44秒 UTC+9 David Pursehouse:

>>>>>>     <mailto:repo-discuss+unsub...@googlegroups.com>

>>>>>>     More info at http://groups.google.com/group/repo-discuss?hl=en
>>>>>>
>>>>>>     ---
>>>>>>     You received this message because you are subscribed to the
>>>>>>     Google Groups "Repo and Gerrit Discussion" group.
>>>>>>     To unsubscribe from this group and stop receiving emails from
>>>>>>     it, send an email to repo-discuss...@googlegroups.com

>>>>>>     <mailto:repo-discuss+unsub...@googlegroups.com>.

>>>>>>     For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>
>>
>>
>>     --
>>     --
>>     To unsubscribe, email repo-discuss...@googlegroups.com

>>     <mailto:repo-discuss+unsub...@googlegroups.com>

>>     More info at http://groups.google.com/group/repo-discuss?hl=en
>>
>>     ---
>>     You received this message because you are subscribed to a topic in
>>     the Google Groups "Repo and Gerrit Discussion" group.
>>     To unsubscribe from this topic, visit
>>     https://groups.google.com/d/topic/repo-discuss/0ZFK5T9DgkE/unsubscribe.
>>     To unsubscribe from this group and all its topics, send an email
>>     to repo-discuss...@googlegroups.com

>>     <mailto:repo-discuss+unsub...@googlegroups.com>.

>>     For more options, visit https://groups.google.com/d/optout.
>
>     --
>     --
>     To unsubscribe, email repo-discuss...@googlegroups.com
>     <mailto:repo-discuss%2Bunsu...@googlegroups.com>
>     More info at http://groups.google.com/group/repo-discuss?hl=en
>
>     ---
>     You received this message because you are subscribed to the Google
>     Groups "Repo and Gerrit Discussion" group.
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to repo-discuss...@googlegroups.com

>     <mailto:repo-discuss+unsub...@googlegroups.com>.

>     For more options, visit https://groups.google.com/d/optout.
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss+unsub...@googlegroups.com>.