Azure and latency issues

[Jo Shields]

Hi all,

I'm observing a problem with our new Jenkins master, running on an Ubuntu VM on a high-spec Azure instance - latency between master and workers is *crazy* high. Like 30 plus seconds:  https://jenkins.mono-project.com/computer/. The SSH-based workers can generally survive that, but the JNLP workers time out and kill running jobs (we're using JNLP for non-cloud workers as it makes our network management way simpler). And Jenkins does a pretty dreadful job of keeping track of whether workers are connected or not, so I end up having to restart the master to kick-start things. It's basically a nightmare.

It seems we're not the only people running Jenkins on Azure showing obscene response times, e.g. http://dotnet-ci.cloudapp.net/computer/ - so the question is whether anyone else on the list has observed this, or (even better) resolved it.

The master isn't under high IO, network, memory or CPU load, seems well supplied with entropy; and the failures don't seem to correlate with Java GC pauses; so I'm not sure where to point the finger.

[R. Tyler Croy]

(replies inline)

That is some pretty awful latency. I'm cuhrious whether the master/agents are
on different Virtual Networks? Have you also reached out to Azure support about
this?

Since this is primarily affecting the JNLP agents, I would be skeptical of what
software-defined-networking stuff inside Azure might be doing to that traffic
that it is not doing to SSH traffic.

I would be surprised if the JNLP support in Jenkins were the culprit



- R. Tyler Croy

------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>

% gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
------------------------------------------------------

[Jo Shields]

 Okay, a co-worker may have found a magical fix - Azure times out idle network connections after 4 minutes, but the default value for hudson.slaves.ChannelPinger.pingInterval is 5 minutes...