pulse not sending buildbot messages

[Jonathan Griffin]

I can connect to pulse normally, but a buildbot consumer receives zero
pulse messages. Normally, I receive them every couple of seconds or so,
now nothing. This is important because OrangeFactor relies on these
messages for various things.

Christian, can you take a look, or would this belong to someone in releng?

Regards,

Jonathan

[Christian Legnitto]

I'm seeing reports of this but everything looks ok from the pulse side. Is buildbot not publishing in anymore catlee?

[Jonathan Griffin]

Pulse doesn't seem so fine right now, attempting to connect yields:

error: [Errno 10061] No connection could be made because the target
machine actively refused
it

Jonathan

[Christian Legnitto]

Ugh, looked like rabbitmq barfed:

=ERROR REPORT==== 1-Sep-2011::14:26:05 ===
** Generic server <0.4066.963> terminating
** Last message in was {'$gen_cast',
{method,
{'queue.declare',0,
<<"w...@mozilla.com|build_log_downloader2_sm-brasstacks01.mozilla.org">>,
false,false,false,true,false,[]},
none}}
** When Server state == {ch,running,1,<0.4035.963>,<0.4065.963>,undefined,
#Fun<rabbit_channel_sup.0.58069862>,none,
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
1,
{[],[]},
{[],[]},
<<"public">>,<<"/">>,<<>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
<0.4033.963>,
{state,fine,
{1314912287274118,#Ref<0.0.333.181907>}}}
** Reason for termination ==
** {{timeout,{gen_server2,call,[<0.4071.963>,{init,false}]}},
[{gen_server2,call,2},
{rabbit_amqqueue,declare,5},
{rabbit_channel,handle_method,3},
{rabbit_channel,handle_cast,2},
{gen_server2,handle_msg,2},

[Jonathan Griffin]

Pulse is working again, and I'm now receiving buildbot messages. Thanks!

Jonathan

[Christian Legnitto]

Hold on, it's going to go down in 5 and then coming back up....upgrading erlang for better performance.

[Bob Clary]

Is pulse located in the phx colo? They are having networking/routing
issues since at least yesterday evening.

[Christian Legnitto]

Yes.

I also took the opportunity to update to the latest erlang and latest rabbitmq. Even at idle rabbit was previously taking all memory previously, it now appears to not do that. We were probably running into an erlang or rabbit bug....it also never seemed to reclaim disk space for its transient message store.

I think we're going to be better going forward and I have a meeting with IT tomorrow about getting some beefier resources (and more of them so we can cluster).

Thanks,
Christian

> _______________________________________________
> tools-pulse mailing list
> tools...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-pulse

[Jonathan Griffin]

Awesome, thanks. If there's anything the a-team can do to help, let me
know.

Jonathan