Fwd: Error while running telemetry/catapult with live sites on Android.

[soh...@chromium.org]

On Thursday, March 15, 2018 at 12:06:57 AM UTC+2, soh...@chromium.org wrote:

Hi,

Was trying to run some telemetry benchmarks on Android with live sites, but there seems to be some issues with DevTools backend.

something like,./run_benchmark --browser=android-chromium -v power.typical_10_mobile --use-live-sites  gives error as 

(INFO) 2018-03-14 21:22:03,417 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:03,616 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:03,617 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:03,836 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:03,837 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,078 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:04,079 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,345 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:04,345 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,638 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:04,638 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connect

Even the running it for linux desktop browser is throwing similar errors,

(INFO) 2018-03-14 21:22:03,837 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,078 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:04,079 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,345 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

(INFO) 2018-03-14 21:22:04,345 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:41921/devtools/browser not yet ready: [Errno 111] Connection refused

(INFO) 2018-03-14 21:22:04,638 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:41921/devtools/browser

Is there some new setting to be enabled for running the benchmarks for live sites.

Thanks.

Sohan

[Ned]

Can you give us the command to reproduce the problem on Linux? I can run "./tools/perf/run_benchmark --browser=system loading.desktop --use-live-sites" on my Mac just fine.

For Android, you may need to do some extra setup work: https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark_harnesses/telemetry_device_setup.md

--
You received this message because you are subscribed to the Google Groups "Chrome benchmarking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-d...@chromium.org.
To post to this group, send email to benchmar...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/benchmarking-dev/835fab2f-2de8-40c4-9ff1-bb7bb7764430%40chromium.org.

[Sohan Jyoti Ghosh]

Hi,

Thanks for the info, I will check after doing the extra setup on Android.

My linux command was, ./run_benchmark --browser=system -v smoothness.top_25_smooth --use-live-sites. Looks like the benchmark may not be for desktop ?

I tried loading.desktop and page_cycler_v2.typical_25, and yes i do not see the Devtools errors. There are some other errors though.

Thanks,

Sohan

To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-dev+unsubscribe@chromium.org.

[Ned]

Hi Sohan, 

One can expect finding errors with --use-live-sites. The reason is a many of the test logic can only be applicable to a specific snapshot of the page (e.g: click on a button with dom id === "foo"),  so with the live sites, those logic may no longer hold.

To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-d...@chromium.org.

[Sohan Jyoti Ghosh]

Hi,

I understand the test logic being valid for specific page instance, and using live sites may not be the best option.

But, the benchmark i am trying to check, wouldn't matter much, as i would need power numbers while scrolling through top 10 mobile sites (yes changing content will have an impact, but i am after difference rather than absolute numbers).

Unfortunately, https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark_harnesses/telemetry_device_setup.md doesnt help much.

The categories marked for provision_devices are, 

usage: provision_devices.py [-h] [-v] [-d DEVICES]

                            [--blacklist-file BLACKLIST_FILE]

                            [--adb-path ADB_PATH]

                            [--adb-key-files ADB_KEY_FILES [ADB_KEY_FILES ...]]

                            [--disable-location] [--disable-mock-location]

                            [--disable-network] [--disable-java-debug]

                            [--disable-system-chrome] [--emulators]

                            [--max-battery-temp NUM] [--min-battery-level NUM]

                            [--output-device-blacklist OUTPUT_DEVICE_BLACKLIST]

                            [--reboot-timeout SECS]

                            [--remove-system-apps [SYSTEM_APP_REMOVE_LIST [SYSTEM_APP_REMOVE_LIST ...]]]

                            [--remove-system-packages [SYSTEM_PACKAGE_REMOVE_LIST [SYSTEM_PACKAGE_REMOVE_LIST ...]]]

                            [--remove-system-webview] [--skip-wipe]

The doc suggests to use --disable-network, but thats not i would want to fetch live sites.

So, i basically ran it with empty param list.

$ devil/devil/android/tools/provision_devices.py

But, still the same dev tools inspector error persists,

(INFO) 2018-03-19 11:18:13,779 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:38646/devtools/browser (remote=localabstract:chrome_devtools_remote)

(INFO) 2018-03-19 11:18:13,780 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:38646/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer

(INFO) 2018-03-19 11:18:13,881 chrome_browser_backend._GetDevToolsClient:118  Got devtools config: ws://127.0.0.1:38646/devtools/browser (remote=localabstract:chrome_devtools_remote)

(INFO) 2018-03-19 11:18:13,882 devtools_client_backend._IsInspectorWebsocketReady:127  Websocket at ws://127.0.0.1:38646/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer

Any help ?

Thanks,

Sohan

To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-dev+unsubscribe@chromium.org.

[Sohan Jyoti Ghosh]

Just to add, the error as you might guess is while loading the live website, so the browser window remains empty. So i was wondering is there some new authentication which maybe required now ?

Note: I was able to run the same test fine a couple of months back.

./run_benchmark --browser=android-content-shell -v power.typical_10_mobile --use-live-sites --reset-results

Thanks,

Sohan

[Ned]

Does the error happen to every test case? If it only happen to few of them, I think you maybe right that some new authentication may be required.

To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-d...@chromium.org.

[Sohan Jyoti Ghosh]

Hi,

I havent checked on all benchmarks, but at least the ones i am interested in (related to power, smoothness perf) are having the error.

So, i decided to switch to an old branch (m60), and things have started working fine again.

Just i chanced to see some crbug complaining about similar websocket error in bots, https://bugs.chromium.org/p/chromium/issues/detail?id=720317#c7 , maybe related ?

I'll continue on m60 for the time being.

Thanks,

Sohan

To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-dev+unsubscribe@chromium.org.

[Randy Nephew]

A few quetsions: Is the phone rooted, what version of android is it running, and how do you pull catapult into your checkout? There are two ways to checkout catapult. one just uses the rolled version and the other uses ToT catapult. (The ToT method is pretty rare though, I think.) Depending on how you pull catapult in, things might have gotten out of sync.

It might make sense to add some devtools people as well, since the error seems to be with communicating with it.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/benchmarking-dev/CAOtrQ7yKWVgCyqaOz52%2BmbTrps%2BfKNqeEsz9FBAUjxLnk6etZw%40mail.gmail.com.

[Sohan Jyoti Ghosh]

On Mon, Mar 19, 2018 at 7:15 PM, Randy Nephew <rne...@google.com> wrote:

A few quetsions: Is the phone rooted, what version of android is it running, and how do you pull catapult into your checkout? There are two ways to checkout catapult. one just uses the rolled version and the other uses ToT catapult. (The ToT method is pretty rare though, I think.) Depending on how you pull catapult in, things might have gotten out of sync.

Yes, the phone is rooted and Android version is 8.

I am pulling the rolled version with Chromium repo.

But good news is i am able to run the power test (on ToT Chromium) on live site with chrome public apk (content shell apk still has issues though).

I checked m60 working logs and needed some minor modification at https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/device_utils.py?type=cs&q=http://crbug.com/686716&l=2304 , my device (Huawei Mate10) was not getting detected as Android O.

Thank you all for the help.

Sohan