Gerrits "review" command is hanging
Douglas Alan
Gerrit. The "review" command actually does what it's told to, but
unfortunately it never returns, and this lack of returning gums up
Jenkins.
I.e., let's say I do the following from a Bash command line:
$ ssh -p 29418 jenkins@localhost gerrit review --project gerrit-
sandbox --message "'Hello'" \
--code-review 0 --verified 1 dc2232531701fa0b0f3b05a49585e
This will add a comment that reads "Hello" to the patch set in Gerrit.
It will also successfully add the "Verified +1" label to the patch
set. So far, so good. What's not so good is that I never get another
Bash prompt. (Unless I type ^C.)
The problem doesn't seem to be a general problem with ssh
communicating with Gerrit. E.g., I can issue different Gerrit commands
via ssh in this manner, and not all of them hang. For instance, if I
replace the patch set ID in the above command line with a non-existent
one, I *do* instantly get a new Bash prompt, along with the following
error message:
fatal: "dc2232531701fa0b0f3b05a49585E" no such patch set
That, of course, is exactly what we would expect. I can also do
"create-project" and "gsql", etc.
This problem occurs when Gerrit is running on one of our CentOS
servers. (I tried a different CentOS server of ours, but that didn't
fix the problem.) If I copy the Gerrit installation to my desktop Mac,
on the other hand, and run it there, then everything works fine. The
Gerrit log files show the difference. On the Mac, a log entry for an
incoming command will look like this:
[2011-11-30 00:23:34,429 -0500] a4512120 jenkins a/1000001 LOGIN
FROM 69.173.70.227
[2011-11-30 00:23:34,463 -0500] a4512120 jenkins a/1000001 'gerrit
review --project gerrit-sandbox --message hello proton --verified
1 b2426cd54059d62dbf5797a72171d86e4beb9bf3' 15ms 10ms 1
[2011-11-30 00:23:34,504 -0500] a4512120 jenkins a/1000001 LOGOUT
The above log entry shows everything operating correctly: User
"jenkins" logs in via ssh, the "review" command is issued and
completes in 10ms, and then user jenkins logs out.
The following is a log entry for a problematic incoming Gerrit command
on my CentOS server:
[2011-11-30 12:32:27,321 -0500] 3ab9dc78 jenkins a/1000001 LOGIN
FROM 127.0.0.1
[2011-11-30 12:32:44,310 -0500] 3ab9dc78 jenkins a/1000001 LOGOUT
[2011-11-30 12:32:44,312 -0500] 3ab9dc78 jenkins a/1000001 'gerrit
review --project gerrit-sandbox --message '\''Woot'\''
dc2232531701fa0b0f3b05a49585e1bf0b3acf07 --code-review 0 --verified 1'
10ms 16972ms killed
The above shows that the "review" command took 17 seconds before being
killed. (It was killed because I eventually got tired of waiting and
typed ^C.)
This issue has been driving me crazy, because it is preventing our
Jenkins installation from communicating properly with our new Gerrit
installation.
Any ideas?
|>ouglas
Martin Fick
wrote:
> Hi. The Gerrit "review" command is hanging in my
> installation of Gerrit. The "review" command actually
> does what it's told to, but unfortunately it never
> returns, and this lack of returning gums up Jenkins.
I have not really heard of any similar problems. Could you
give us some more details about your setup, version #s...
Do any other commands complete during this hanging (if you
run them from another window), is the WUI still fully
responsive?
-Martin
--
Employee of Qualcomm Innovation Center, Inc. which is a
member of Code Aurora Forum
Alex Blewitt
> On Wednesday, November 30, 2011 11:35:32 am Douglas Alan
> wrote:
>> Hi. The Gerrit "review" command is hanging in my
>> installation of Gerrit. The "review" command actually
>> does what it's told to, but unfortunately it never
>> returns, and this lack of returning gums up Jenkins.
>
> I have not really heard of any similar problems. Could you
> give us some more details about your setup, version #s...
>
> Do any other commands complete during this hanging (if you
> run them from another window), is the WUI still fully
> responsive?
There's an option in ssh to send keepalives; do you have that in ~/.ssh/config ?
https://wiki.archlinux.org/index.php/Secure_Shell#Keep_Alive
Matthias Sohn
Matthias
Douglas Alan
<matthi...@googlemail.com> wrote:
> create a couple of thread dumps when the command is hanging. The
> stack traces in the thread dumps should give some hints on where it's
> hanging
That was a good idea. The thread dump revealed that the "review"
thread in question was waiting on SMTP. So I edited etc/gerrit.config
to point to our centralized SMTP server rather than localhost, and the
problem went away.
Thanks!
|>ouglas
Martin Fick
wrote:
Yikes, that seems like a problematic design. Perhaps the
emails should switched to run asynchronously?
Magnus Bäck
Martin Fick <mf...@codeaurora.org> wrote:
> On Wednesday, November 30, 2011 03:28:34 pm Douglas Alan wrote:
>
> > That was a good idea. The thread dump revealed that the
> > "review" thread in question was waiting on SMTP. So I edited
> > etc/gerrit.config to point to our centralized SMTP server
> > rather than localhost, and the problem went away.
>
> Yikes, that seems like a problematic design. Perhaps the emails
> should switched to run asynchronously?
They definitely should. It's already on my list to look into. Perhaps
the currently available queueing mechanisms for replication and other
asynchronous jobs could be reused for this purpose?
--
Magnus Bäck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson
Saša Živkov
<magnu...@sonyericsson.com> wrote:
> On Wednesday, November 30, 2011 at 23:35 CET,
> Martin Fick <mf...@codeaurora.org> wrote:
>
>> On Wednesday, November 30, 2011 03:28:34 pm Douglas Alan wrote:
>>
>> > That was a good idea. The thread dump revealed that the
>> > "review" thread in question was waiting on SMTP. So I edited
>> > etc/gerrit.config to point to our centralized SMTP server
>> > rather than localhost, and the problem went away.
>>
>> Yikes, that seems like a problematic design. Perhaps the emails
>> should switched to run asynchronously?
>
> They definitely should. It's already on my list to look into. Perhaps
> the currently available queueing mechanisms for replication and other
> asynchronous jobs could be reused for this purpose?
Sounds reasonable, good proposal.
Markward Schubert
I have the same situation, but guess, that it is not the smtp server
in my case.
I updated to current version of gerrit and the current version of
gerrit trigger.
I get my Changes fetched by Jenkins and it builds things. After a
successful build, it does not report back to gerrit.
Tried to issue the above ssh-command on the the cli and it worked
nicely. The review was counted (so no access-issue here, i guess) AND
i received a proper email, notifying me about the change.
Can anybody give me a hint where I can continue searching?
Thanks and Regards!
Markward Schubert
When i filled in explicit values of 0 in all the Verify fields of the
very job, somehow then also the Review +1 is sent to gerrit.
On Dec 8, 6:19 pm, Markward Schubert <markward.schub...@gmail.com>
wrote:
David Raison
David Raison
After posting this, I realized that it was probably the wrong thread, and that I should've posted to the other thread that was linking to this one. Sorry.
We had the same problem, but in our case, the issue was that we had started using gerrit trigger in an earlier version when gerrit was still in version 2.0.x Since then, the command to set verified and code review stati has been renamed from "approve" to "review".In the configuration for gerrit trigger, there's an advanced... button where you will be able to see and change the commands used by the plugin. Make sure it says "gerrit review".
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/77Uobpk99iw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Christopher Alexander
Triggers from Jenkins don't make their way into gerrit. However i can do:
ssh -p 29418 user@localhost gerrit review --project bt_platform --message "'Hello'" --code-review 0 --verified 1 4b87ad77513241b6f8fe21ff3e5bbeba358009a9
Gerrit picks this up.
Gerrit trigger plugin is up to date.
Christopher Alexander
Thanks David for pointing it out.
