Chrome Headless Slow Network Requests as Local Jenkins User

[david...@gmail.com]

I am migrating from PhantomJS to ChromeHeadless for use in continuous integration via Karma/Jasmine within a Windows/Jenkins-Slave.

The problem I am seeing is that when running the tests as the local jenkins user with ChromeHeadless, the tests time out. Enabling Karma debugging reveals that network requests are being made very slowly, about every two seconds. When running Chrome w/ GUI, all of the network requests are made very quickly. Finally, if I login to the Windows/Jenkins-Slave with my domain account, and run ChromeHeadless it works very quickly as well.

Each of these tests were done with the same Chrome options, except "--headless --no-sandbox" added to enable Chrome Headless.

Summary:

• Local user "jenkins", ChromeHeadless very slow requests
• Local user "jenkins", Chrome GUI very fast requests
• My Domain user, ChromeHeadless very fast requests

My questions are:

What are the differences between Chrome GUI and Chrome Headless that might affect network request performance?

Do you have any recommendations on how to debug this from the Chrome side? (The remote debugging feature doesn't seem to provide me with much insight into the headless network requests.)

Thanks.

Details

Windows Server 2008 R2 Standard 6.1.7601 SP1 Build 7601

Selenium Standalone Server 3.8.1

Karma 1.7.1

ChromeDriver 2.34.522940 (1a76f96f66e3ca7b8e57d503b4dd3bccfba87af1)

Chrome Version 63.0.3239.132 (Official Build) (64-bit)

[Eric Seckler]

Regarding debugging: Your best bet would be taking a chrome trace - For headless, you can do this via DevTools or via command line flags (--trace-startup and its siblings).

Feel free to share a trace on the mailing list. You could also compare a trace taken from headless with one taken from desktop, to highlight any differences.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/97340131-6a38-457b-8ee6-91b061d4a604%40chromium.org.

[davida...@gmail.com]

Eric,

There has been a development in my investigation. I discovered that one of the Jenkins server works correctly while a couple others do not. I was able to capture apple to apple traces, with the only difference being the server.

I have been looking at these traces for awhile now and trying to figure out what aspects of the trace indicate a problem. The 7 second startup log for the successful server is 22 MB, while the unsuccessful server is 800 KB. I'm a bit concerned about sharing the raw trace at this time, as it may contain confidential details from my company; but I have taken screenshots that may allow you to guide me further.

The first pair of images are from the top of the analysis:

Successful (FAST)

Unsuccessful (SLOW)

The next two screenshots are from the "Resource" section which corresponds to what I can see from the web server log:

Successful (FAST)

Unsuccessful (SLOW)

The question is what is the cause of this? Is the process executing slower? Is Chrome throttling something? Is it proxy related?

Further guidance on how to interpret these traces or on how to get other metadata would be appreciated. Thanks.

- David

[Eric Seckler]

Hmm, it's tricky to make guesses without diving deeper into the trace, the screenshots just show that things are slow :)

I'd look for differences in the BrowserMain and RendererMain threads first, and then go by exclusion. I.e. see until when things progress fine and at what point you see a longer delay between tasks than expected.

If everything is slower in general, maybe the browser doesn't have enough computing resources available?

If it's just network responses that are slow, maybe there's something misconfigured in the OS network stack?

[david...@gmail.com]

One big difference is the slow case shows two PAC threads in below the DnsConfigService, while the fast case doesn’t show any PAC threads at all.

Is this a proxy setting? Why would headless (or sandbox) affect proxy? Any proxy related debug steps? I will continue my investigation.

(Sorry can’t show screenshot. I am at work today and hey block Google Groups.)

[davida...@gmail.com]

Eric,

First, thank you for your responsiveness.

I have spent another day debugging with no resolution. I changed my test scenario away from Karma and simply towards generic browser performance.

I have created 4 chrome traces. All traces were targeting google.com. You can download them from here:

http://intref.com/ll/ChromeHeadlessDebug.zip  (~4MB)

The traces with "Jenkins" in the name were run with the local "jenkins" user.

The traces with "David" in the name were run with my domain user.

The traces with "Headless" were --headless --no-sandbox, while "GUI" ran normally.

To see other flags used, see the last line in either headless trace.

Finally, I'd like to ask: Do these logs make sense?

Although both the GUI traces showed the GUI quickly and rendered google.com relatively quickly; why did the Jenkins trace always result in a ~30 MB file? I'm very curious what caused so many log entries.

Also, if the GUI Jenkins file was larger, why was Headless file smaller; proportionally to the David executions.

- David

[Eric Seckler]

From the traces, it looks like network requests are really slow. I can't really tell what that might be due to. A few pointers:

• GUI JENKINS shows under "NetLog" that even such things as connecting a socket take up to 4 seconds. That's not the case in HEADLESS David, network requests are fast there.
• When comparing HEADLESS vs GUI JENKINS, headless shows two redirect events (Navigation redirectDelay), whereas GUI only shows one. Which might explain why headless takes longer. This might be due to different user agents, try setting it manually using --user-agent in headless mode.
• The reason your GUI traces files are huge is that something seems to be animating in the browser UI, creating lots of events. Headless doesn't have a browser UI to animate.
• The reason the GUI David file is small is that tracing stopped after ~150ms. Not sure why that would be, maybe something crashed / closed / failed?

I would expect your problems to be due to some network configuration issue with your Jenkins user / OS. Chrome seems to receive enough computing resources.

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/09bac4bb-3570-43bd-8457-c1214d2c68b4%40chromium.org.

[davida...@gmail.com]

Eric,

After focusing on network differences I discovered a difference between the "jenkins" (local user) user and my user (domain user). I ran Chrome GUI as myself and as the jenkins user and went to: chrome://net-internals/#proxy

This revealed my domain user was going through a proxy while the jenkins user was not.

I set the proxy in Windows for the jenkins user, and that fixed the problem. I verified further by removing it and seeing the problem come back. So progress is being made!

My IT Administrator is hesitant to set this proxy to this user across the ~100 boxes we have, as there may be unintended consequences. So I need to do some more investigation.

Attempting Manual Proxy Setting

I tried declaring the --proxy-pac-url option via Chrome CLI, but this did not solve the problem. Chrome still didn't use it as verified via net-internals/#proxy:

Windows NO PROXY

Windows NO PROXY, Chrome --proxy-pac-url SET

Windows PROXY

As you can see Chrome is showing three different scenarios.

Questions for you:

Is there something about Chrome Headless that affects how the proxy is used?

Is there something about --no-sandbox that affects how the proxy is used? (A lot of people using Chrome Headless recommend to set --no-sandbox, not sure why.)

Why does setting the --proxy-pac-url not translate into an "Effective proxy setting" and instead only shown as "Original proxy setting"?

Thanks again.

- David

[Alex Clarke]

On 12 January 2018 at 16:25, <davida...@gmail.com> wrote:

Eric,

After focusing on network differences I discovered a difference between the "jenkins" (local user) user and my user (domain user). I ran Chrome GUI as myself and as the jenkins user and went to: chrome://net-internals/#proxy

This revealed my domain user was going through a proxy while the jenkins user was not.

I set the proxy in Windows for the jenkins user, and that fixed the problem. I verified further by removing it and seeing the problem come back. So progress is being made!

My IT Administrator is hesitant to set this proxy to this user across the ~100 boxes we have, as there may be unintended consequences. So I need to do some more investigation.

Attempting Manual Proxy Setting

I tried declaring the --proxy-pac-url option via Chrome CLI, but this did not solve the problem. Chrome still didn't use it as verified via net-internals/#proxy:

Windows NO PROXY

Windows NO PROXY, Chrome --proxy-pac-url SET

Windows PROXY

As you can see Chrome is showing three different scenarios.

Questions for you:

Is there something about Chrome Headless that affects how the proxy is used?

Quite possibly yes, headless is a content embedder (like chrome and third party browsers such as Opera) and it's possible various systems are not initialized properly in the headless code.

 

Is there something about --no-sandbox that affects how the proxy is used? (A lot of people using Chrome Headless recommend to set --no-sandbox, not sure why.)

I'd be surprised if that was the case, although it has various side effects.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/8eb6a40a-9e4e-4866-bbed-e6be81ae6ba8%40chromium.org.

[Alex Clarke]

On 12 January 2018 at 20:38, Alex Clarke <alexc...@google.com> wrote:

On 12 January 2018 at 16:25, <davida...@gmail.com> wrote:

Eric,

After focusing on network differences I discovered a difference between the "jenkins" (local user) user and my user (domain user). I ran Chrome GUI as myself and as the jenkins user and went to: chrome://net-internals/#proxy

This revealed my domain user was going through a proxy while the jenkins user was not.

I set the proxy in Windows for the jenkins user, and that fixed the problem. I verified further by removing it and seeing the problem come back. So progress is being made!

My IT Administrator is hesitant to set this proxy to this user across the ~100 boxes we have, as there may be unintended consequences. So I need to do some more investigation.

Attempting Manual Proxy Setting

I tried declaring the --proxy-pac-url option via Chrome CLI, but this did not solve the problem. Chrome still didn't use it as verified via net-internals/#proxy:

Windows NO PROXY

Windows NO PROXY, Chrome --proxy-pac-url SET

Windows PROXY

As you can see Chrome is showing three different scenarios.

Questions for you:

Is there something about Chrome Headless that affects how the proxy is used?

Quite possibly yes, headless is a content embedder (like chrome and third party browsers such as Opera) and it's possible various systems are not initialized properly in the headless code.

 

Is there something about --no-sandbox that affects how the proxy is used? (A lot of people using Chrome Headless recommend to set --no-sandbox, not sure why.)

I'd be surprised if that was the case, although it has various side effects.

Why does setting the --proxy-pac-url not translate into an "Effective proxy setting" and instead only shown as "Original proxy setting"?

It hasn't been wired up,  I've filed a bug to investigate how hard it is to support this. https://bugs.chromium.org/p/chromium/issues/detail?id=801688