DevTools benchmark addition

[Alex Chiem]

Hello,

 

We are trying to add a new DevTools benchmark and going over the “Guideline for adding new benchmark”. We believe that the DevTools benchmark doesn’t belong to any existing benchmark or harness.

 

If we were to add it to tools/perf/contrib/, would it run automatically and normally just like the 12 benchmarks in tools/perf/?

 

Thank you,

-Alex

[Caleb Rouleau]

Historically, the primary difference between benchmarks defined in tools/perf/benchmarks and those defined in tools/perf/contrib is that contrib benchmarks are not run automatically on our waterfall ( https://ci.chromium.org/p/chrome/g/chrome.perf/console ). You could however run benchmarks from tools/perf/contrib on some other infrastructure (for example Microsoft's performance infrastructure). It is nice to put these into tools/perf/contrib so that we can try not to break them as we make updates to the Telemetry library.

We need to understand why performance of DevTools is important enough to spend a bunch of bot time on. The current Chrome benchmarking strategy is to only spend time on the very most important use cases. Maybe you could write a one-pager explaining why DevTools performance is important and which use cases in particular are performance-sensitive for DevTools. Then it would be great to get sign off on that one pager from Chrome's devtools team and from me.

Since the bot infrastructure is Google's, the benchmark must be important to Chrome. If the benchmark is important to Edge or some other Chromium-based browser, then other organizations will need to dedicate bot time to running it. For this reason, you will need to have a Googler to sign on as an OWNER of this new benchmark. 

--
You received this message because you are subscribed to the Google Groups "Chrome benchmarking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-d...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/benchmarking-dev/CY1PR00MB0220AC6D8EF11A14F325138A95770%40CY1PR00MB0220.namprd00.prod.outlook.com.

[Alex Chiem]

Hi Caleb,

I am including Yang who is from Google who is working with us to get the DevTools Perf tests upstreamed.

If the benchmark were to be in the tools/perf/contrib/ folder, would Google be able to run bisections manually? Does Google not run any benchmarks in the contrib folder?

The reason why we want to upstream these tests is because we would like to bisect any perf regressions. Microsoft and Google have been contributing to work to DevTools and our perf lab has detected some perf regressions which point to 1 big upstream commit. Both us and Google DevTools team would like to know the performance regressions and what caused it.

Thanks,

-Alex

[Caleb Rouleau]

Yeah, we should be able to run bisections on contrib benchmarks just fine. +Dean to confirm.

It would be much better to just have occasional on-demand pinpoint runs rather than have the waterfall need to use capacity for this regularly.

[Alex Chiem]

Am I able to schedule the occasional on-demand pinpoint runs or does it have to be someone at Google?

 

If we want to measure the performance over time, would we have to schedule these runs manually weekly or daily?

 

@Yang Guo what do you think of not having it in the waterfall and scheduling the on demand runs?

 

Thanks,

-Alex

[Caleb Rouleau]

I mean, if you're gonna just schedule them every day, then it's going to take just as much bot time as putting them on the waterfall, so you should in that case still go through the process of writing that one pager to make sure this is worth the bot time.

[Yang Guo]

Hi,

sorry for not responding sooner. Jet lag and day full of meetings didn't leave much time to answer emails :)

I definitely think that we should run the benchmarks regularly to detect regressions. That is the point of having the benchmark. Perhaps what differs this from regular Chromium benchmarks is that we are not trying to track the performance of Chrome, or regressions caused by changes to Chrome, but rather the performance of DevTools as web app, so that changes to DevTools do not regress its performance, e.g. for starting up. We could of course take this as a reason to set up a separate performance tracking infrastructure, but I'd rather use existing infra if possible.

I can write that one-pager if necessary, once I have some time after BlinkOn :)

Cheers,

Yang

Yang Guo | Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank. This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

[Alex Chiem]

Hey Caleb,

 

So the plan is that Yang will be writing the one-pager and I will continue to get my CLs in a place to be ready to merge after the approval for the DevTools Perf tests goes through.

 

Also, are there more docs on adding a new benchmark besides this:

https://docs.google.com/document/d/1ni2MIeVnlH4bTj4yvEDMVNxgL73PqK_O9_NUm3NW3BA/edit#

 

 

Regarding my CLs, I am blocked by these 2 errors. Could you take a look and give me some advice on how to fix these?

 

#1 I am trying to use generate_perf_data to generate a new benchmark.csv but hitting this error. I followed these instructions: https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark/benchmark_ownership.md

 

 

#2 I am failing presubmit after merging the latest master into my branch. The error seems to be that I don’t have access to the chrome-partner-telemetry after using python depot_tools/gsutil.py config.

(http://www.chromium.org/developers/telemetry/upload_to_cloud_storage)**

 

Presubmit ERRORS **
Script benchmarks\system_health_load_tests_smoke_test.py failed.***************
WARNING:root:Failed to import devil.android.forwarder: No module named fcntl
ERROR:root:You either aren't authenticated or don't have permission to use the archives for this page set.
You may need to run gsutil config.
You can find instructions for gsutil config at: http://www.chromium.org/developers/telemetry/upload_to_cloud_storageTraceback (most recent call last):
 <module> at X:\chromium\depot_tools\src\tools\perf\benchmarks\system_health_load_tests_smoke_test.py:21
   sys.exit(main())
 main at X:\chromium\depot_tools\src\tools\perf\benchmarks\system_health_load_tests_smoke_test.py:17
   system_health_smoke_test.load_tests(None, None, None)
 load_tests at X:\chromium\depot_tools\src\tools\perf\benchmarks\system_health_smoke_test.py:306
   story_names=story_names)
 DownloadArchivesIfNeeded at X:\chromium\depot_tools\src\third_party\catapult\telemetry\telemetry\wpr\archive_info.py:114
   download_if_needed(archive_path)
 download_if_needed at X:\chromium\depot_tools\src\third_party\catapult\telemetry\telemetry\wpr\archive_info.py:89
   cloud_storage.GetIfChanged(path, self._bucket)
 GetIfChanged at X:\chromium\depot_tools\src\third_party\catapult\common\py_utils\py_utils\cloud_storage.py:487
   _GetLocked(bucket, expected_hash, file_path)
 _GetLocked at X:\chromium\depot_tools\src\third_party\catapult\common\py_utils\py_utils\cloud_storage.py:385
   _RunCommand(['cp', url, partial_download_path.name])
 _RunCommand at X:\chromium\depot_tools\src\third_party\catapult\common\py_utils\py_utils\cloud_storage.py:159
   raise GetErrorObjectForCloudStorageStderr(stderr)
PermissionError: Attempted to access a file from Cloud Storage but you don't have permission. To configure your credentials:
 1. Run "X:\chromium\depot_tools\src\third_party\catapult\third_party\gsutil\gsutil config" and follow its instructions.
 2. If you have a @google.com account, use that account.
 3. For the project-id, just enter 0.Locals:
 args       : ['C:\\Users\\alchiem\\.vpython-root\\1481e2\\Scripts\\python.exe', 'X:\\chromium\\depot_tools\\src\\third_party\\catapult\\third_party\\gsutil\\gsutil', 'cp', 'gs://chrome-partner-telemetry/f2584469fe9e114e903bfe67317920fe692a204a', u'X:\\chromium\\depot_tools\\src\\tools\\perf\\page_sets\\data\\tmpuglq7d']
 gsutil     : <subprocess.Popen object at 0x00000000062B94E0>
 gsutil_env : None
 stderr     : 'AccessDeniedException: 403 alex....@microsoft.com does not have storage.objects.list access to chrome-partner-telemetry.\r\n'
 stdout     : ''
***************

 

Thanks,

-Alex

[Caleb Rouleau]

I will answer these questions now, but I want to again emphasize that we should be going through the process to make sure this benchmark is needed before we work on it too deeply. I won't be reviewing changes until after we go through this process.

#1: Please add documentation link as a decorator to your benchmark using the benchmark.Info decorator: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/media.py?q=media.desktop&sq=package:chromium&g=0&l=50 You can add documentation as a .md file inside https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark/harnesses . 

#2: It is a bug that you need this access to pass presubmit. I filed https://bugs.chromium.org/p/chromium/issues/detail?id=1024912

In the meantime, you can use --bypass-hooks to skip presubmit (it will still run on a bot before submission), or you could ask ush...@chromium.org to get you access to gs://chrome-partner-telemetry.

[Yang Guo]

Hi Caleb,

do you have a link to the process so that the one-pager I'll prepare includes everything necessary?

Cheers,

Yang

Yang Guo | Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank. This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

[Caleb Rouleau]

I don't have a template. We need to document why this is needed. What performance-critical things will it measure? Why does this matter to Chrome/Google (since Google is paying for the bots)? Who is willing to sponsor it? Provide examples of regressions that this benchmark would have caught. What was the negative impact of these regressions? If necessary, you could make a Googler-only side of the document if there is non-public information that helps me understand why we need this.

This doesn't need to be something that you spend a ton of time on, but it would be great to write something up quickly rather than doing this over email.

I hope this helps!

Caleb

[Yang Guo]

Got it. I'll prepare that early next week. Let's also maybe catch up during BlinkOn :)

Yang

[Alex Chiem]

Hey Yang,

 

Do you need any data or any other information from us to help you with the one pager?

 

Thank you,

-Alex

 

From: Yang Guo <yan...@google.com>
Sent: Thursday, November 14, 2019 11:47 AM
To: Caleb Rouleau <crou...@google.com>
Cc: Alex Chiem <Alex....@microsoft.com>; benchmar...@chromium.org; j...@chromium.org; Leo Lee <Leo...@microsoft.com>; Dean Berris <dbe...@chromium.org>; Zoher Ghadyali <Zoher.G...@microsoft.com>; James Lissiak <James....@microsoft.com>; Connor Moody <Connor...@microsoft.com>; Tim van der Lippe <tvande...@google.com>
Subject: Re: [EXTERNAL] Re: DevTools benchmark addition

 

Got it. I'll prepare that early next week. Let's also maybe catch up during BlinkOn :)

 

Yang

 

[Yang Guo]

Hi,

it took a few days for me to work down the backlog of things that accumulated while I was at BlinkOn.

I wrote a short document to add details to why we would like to add a benchmark for DevTools frontend. Please let me know if that is sufficient.

Cheers,

Yang

Yang Guo | Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank. This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

[Alex Chiem]

Hey Caleb,

 

My CL is failing benchmark_smoke_unittest:

telemetry_perf_unittests (retry summary)

Failed with patch, succeeded without patch:
benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.devtools
page_sets.story_set_unittest.StorySetUnitTest.testSmoke

 

I was wondering if there are docs somewhere on how to run these? The execution details on the error didn’t tell me much.

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8896405316828311728/+/steps/telemetry_perf_unittests__retry_summary_/0/logs/execution_details/0

[Caleb Rouleau]

Hi, you can run them with 

./tools/perf/run_tests BenchmarkSmokeTest.devtools

I made this change https://chromium-review.googlesource.com/c/chromium/src/+/1925592/ that provides instructions for running unit tests in a markdown doc.

[Caleb Rouleau]

Hi all, I wrote this doc DevTools benchmark: Chromium Speed Infra or its own infra? explaining the benefit of running this new devtools benchmark on infra inside the devtools frontend repo rather than reusing Chromium repo infra. In general, you can mix and match many of our technologies: chromeperf dashboard, tracing metrics (TBMv2 or TBMv3), Telemetry, etc. since they are loosely coupled from each other. They can all work outside of the Chromium repo.

I spoke with Yang and he seemed to agree that this was a better path forward.

I shared the doc with the microsoft email addresses on this thread that have Google accounts associated with them. Please email back after creating a Google account and I can share again.

Thanks!

Caleb

[Yang Guo]

I also copied the document to a chromium account so that more people can access it more easily: https://docs.google.com/document/d/1Fub0R4WqeKdhIko61y-iFPQCq4UsDzgAwTFltvBbOvk/edit#

Yang

[Caleb Rouleau]

Thanks Yang!