Access to chrome-partner-telemetry

[Matt Mallett]

I'm working on getting telemetry_Benchmark tests running on moblab, and I need access to the chrome-partner-telemetry bucket with our test account cros.moblab....@gmail.com . 

The tests are currently failing due to not having access to the bucket while attempting to download dependencies.

Could this account be granted access?

[Ned]

+Ellen Li 

--
You received this message because you are subscribed to the Google Groups "telemetry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

[Ellen Li]

Hi there,

We’re happy to set you up with access––we just need to confirm that the legal entity you're affiliated with is under an active NDA with Google. Would you mind providing which legal entity you are associated with? If we do not currently have an NDA in place, I will generate one for you to sign electronically.

You'll also need to make sure that any email address receiving access has an associated Google account, which you can do here.

Thanks!

-Ellen

[Matt Mallett]

I'm a Google employee, working on getting the telemetry tests to run on a ChromeOS device for internal testing.

The email cros.moblab....@gmail.com is a Google account and is already being used to grant access to some gs buckets.

[Ellen Li]

Great, you should have access now! Please let us know if you encounter any issues.

Thanks!

[Matt Mallett]

I am having some issues actually, is this the best place to discuss them or do you have a different group.

[Ellen Li]

+Ben Henry, who might be able to assist further.

[Matt Mallett]

Thanks Ellen, Ben

I'm having trouble with telemetry dependencies for jetstream and smoothness.top_25_smooth. 

My setup is to use an existing .boto file that authenticates to a few different buckets already (points to the cros.moblab....@gmail.com account). I created a copy of that .boto file and placed it /home/chromeos-test/.boto on the moblab. I also created the chromeos-test user and group. If any test runs as that user, or uses HOME=/home/chromeos-test/.boto, the gsutil authentication should be set up correctly, without needing any gsutil config. There shouldn't be any file permission issues as the moblab user has full permission on /home/chromeos-test/.boto (it is a hard link from /home/moblab/.boto so it retains those permissions), and I believe the tests run as root so should be able to read it regardless.

jetstream looks like it is missing permission for cros.moblab....@gmail.com to access chrome-telemetry. trace below:

```

04/23 12:53:03.805 DEBUG|             utils:0214| Running 'ssh 192.168.231.1 python /mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/fetch_benchmark_deps.py jetstream'

04/23 12:53:07.934 DEBUG|  telemetry_runner:0449| Error occurred getting DEPs: 

 Warning: Permanently added '192.168.231.1' (ED25519) to the list of known hosts.

Warning: Permanently added '192.168.231.1' (ED25519) to the list of known hosts.

WARNING:root:Unable to import cv2 due to: Incorrect cv2 version found, expected 2.4.8 <= version < 3.0.0, found version $Rev: 4557 $

Traceback (most recent call last):

  <module> at /mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/fetch_benchmark_deps.py:115

    main(sys.argv[1:], sys.stdout)

  main at /mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/fetch_benchmark_deps.py:104

    FetchDepsForBenchmark(benchmark, output)

  FetchDepsForBenchmark at /mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/fetch_benchmark_deps.py:75

    _FetchDependenciesIfNeeded(story_set)

  _FetchDependenciesIfNeeded at /mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/fetch_benchmark_deps.py:37

    story_set.wpr_archive_info.DownloadArchivesIfNeeded()

  DownloadArchivesIfNeeded at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/wpr/archive_info.py:107

    download_if_needed(archive_path)

  download_if_needed at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/wpr/archive_info.py:86

    cloud_storage.GetIfChanged(path, self._bucket)

  GetIfChanged at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:451

    _GetLocked(bucket, expected_hash, file_path)

  _GetLocked at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:349

    _RunCommand(['cp', url, partial_download_path.name])

  _RunCommand at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:154

    raise GetErrorObjectForCloudStorageStderr(stderr)

CloudStorageError: AccessDeniedException: 403 AccessDenied

<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>cros.moblab....@gmail.com does not have storage.objects.list access to chrome-telemetry.</Details></Error>

Locals:

  args       : ['/usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil', 'cp', 'gs://chrome-telemetry/5fa90a167110fc63693dcadfafe699965d71518e', u'/mnt/moblab/static/reef-release/R66-10440.0.0/telemetry_src/src/tools/perf/page_sets/data/tmphBjXKl']

  gsutil     : <subprocess.Popen object at 0x7419fb074890>

  gsutil_env : {'SHELL': '/bin/bash', 'SHLVL': '1', 'SSH_CLIENT': '192.168.231.176 42422 22', 'PWD': '/root', 'LOGNAME': 'root', 'USER': 'root', 'MAIL': '/var/spool/mail/root', 'PATH': '/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin', 'SSH_CONNECTION': '192.168.231.176 42422 192.168.231.1 22', 'HOME': '/home/chromeos-test/', '_': '/usr/bin/python'}

  stderr     : "AccessDeniedException: 403 AccessDenied\n<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>cros.moblab....@gmail.com does not have storage.objects.list access to chrome-telemetry.</Details></Error>\n"

  stdout     : ''

```

smoothness.top_25_smooth is not finding the .boto file and is trying to access the bucket as anonymous, any insight here? trace below:

```

(INFO) 2018-04-23 13:04:59,881 cloud_storage._GetLocked:340  Downloading gs://chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c to /usr/local/telemetry/src/tools/perf/page_sets/data/top_25_012.wprgo

(ERROR) 2018-04-23 13:05:02,449 archive_info.download_if_needed:93  You either aren't authenticated or don't have permission to use the archives for this page set.

You may need to run gsutil config.

You can find instructions for gsutil config at: http://www.chromium.org/developers/telemetry/upload_to_cloud_storage

(INFO) 2018-04-23 13:05:02,450 archive_info.DownloadArchivesIfNeeded:110  All WPR archives are downloaded, took 2.76540184021 seconds.

(INFO) 2018-04-23 13:05:02,450 exception_formatter.PrintFormattedException:19  Try printing formatted exception: None None None

Traceback (most recent call last):

  RunBenchmark at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:356

    max_num_values=benchmark.MAX_NUM_VALUES)

  Run at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:183

    story_set.archive_data_file, story_set.wpr_archive_info, stories):

  _UpdateAndCheckArchives at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:413

    wpr_archive_info.DownloadArchivesIfNeeded()

  DownloadArchivesIfNeeded at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/wpr/archive_info.py:107

    download_if_needed(archive_path)

  download_if_needed at /usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/wpr/archive_info.py:86

    cloud_storage.GetIfChanged(path, self._bucket)

  GetIfChanged at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:451

    _GetLocked(bucket, expected_hash, file_path)

  _GetLocked at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:349

    _RunCommand(['cp', url, partial_download_path.name])

  _RunCommand at /usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/cloud_storage.py:154

    raise GetErrorObjectForCloudStorageStderr(stderr)

CredentialsError: Attempted to access a file from Cloud Storage but you have no configured credentials. To configure your credentials:

  1. Run "HOME=/home/chromeos-test/ /usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil config" and follow its instructions.

  2. If you have a @google.com account, use that account.

  3. For the project-id, just enter 0.

Locals:

  args       : ['/usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil', 'cp', 'gs://chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c', u'/usr/local/telemetry/src/tools/perf/page_sets/data/tmpaGH6br']

  gsutil     : <subprocess.Popen object at 0x7f41d611d590>

  gsutil_env : {'SHELL': '/bin/bash', 'SHLVL': '1', 'SSH_CLIENT': '127.0.0.1 35212 2222', 'PWD': '/root', 'LOGNAME': 'root', 'USER': 'root', 'MAIL': '/var/spool/mail/root', 'PATH': '/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin', 'SSH_CONNECTION': '127.0.0.1 35212 127.0.0.1 2222', 'HOME': '/home/chromeos-test/', '_': '/usr/local/bin/python'}

  stderr     : 'ServiceException: 401 Anonymous caller does not have storage.objects.get access to chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c.\n'

  stdout     : ''

```

Thanks!

[Ben Henry]

I love that you're running perf tests on jetstream! I originally helped get jetstream on Chromium infrastructure/build in the very beginning of the project.

error:

"""

"AccessDeniedException: 403 AccessDenied\n<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>cros.moblab....@gmail.com does not have storage.objects.list access to chrome-telemetry.</Details></Error>\n"

  stdout     : ''

smoothness.top_25_smooth is not finding the .boto file and is trying to access the bucket as anonymous, any insight here? trace below:

```

(INFO) 2018-04-23 13:04:59,881 cloud_storage._GetLocked:340  Downloading gs://chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c to /usr/local/telemetry/src/tools/perf/page_sets/data/top_25_012.wprgo

"""

[BGH] This seems to say that the bot running smoothness.top_25_smooth cannot find the .boto file. The point about hardlinks across /home/ user directories has me really concerned - I'm not sure you can do that. Can you add some folks who know moblab really well to see if you've set up the environment in moblab correctly? Maybe +Simran Basi ? Also, have you tried following the advice on the bot: "You may need to run gsutil config."

Ben

[Matt Mallett]

I made /home/chromeos-test/.boto a regular file with an exact copy of /home/moblab/.boto, instead of a symlink, and gave it 777 permissions. I am still seeing the anonymous error in smoothness.top_25_smooth. I don't think gsutil config is the answer as that just creates a different .boto file, and we already have a .boto file that works for multiple other buckets using gsutil. I believe it's something to do with the way this test is set up, and how the HOME and user is configured. I can take the gsutil command that failed and run it directly on the terminal (with HOME=/home/chromeos-test) and it works fine.

I also am still seeing the error with jetstream: cros.moblab....@gmail.com does not have storage.objects.list access to chrome-telemetry. It seems the account needs to be granted access to that bucket.

[Ben Henry]

Wait - you can run gsutil locally with the correct .boto and it works or are you still getting an error locally as well?

[Matt Mallett]

I run gsutil locally with the /home/chromeos-test/.boto and it works.

As root,

HOME=/home/chromeos-test /usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil cp gs://chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c /usr/local/telemetry/src/tools/perf/page_sets/data/tmpCs6j55

Copying gs://chrome-partner-telemetry/bd503b282964e34176b6c617d84d38aee20aa07c...

/ [1 files][ 19.2 KiB/ 19.2 KiB]                                                

Operation completed over 1 objects/19.2 KiB.  

[Matt Mallett]

I figured out the problem with smoothness.top_25_smooth, The gsutil command is run on the DUT, not on the server, so I needed to manually put a .boto key onto the DUT. After doing that, the test works fine. Do you know why this test specifically runs the gsutil command on the DUT? Every other telemetry_Benchmark test I've run has used the .boto file on the server.

I am still seeing this permission issue with the jetstream test:

cros.moblab....@gmail.com does not have storage.objects.list access to chrome-telemetry.

It looks like that email needs access to the chrome-telemetry bucket.

[Ned]

+Achuith Bhandarkar +Chung-yih Wang in cases they know why this benchmark is executing commands on DUT.

[Ellen Li]

I just double checked on our end, and that email address does seem to have access to chrome-telemetry.

Ben, any thoughts for how to troubleshoot this?

[Ben Henry]

We figured out the access part, now it's about how ChromeOS lab setup works. Matt is asking why we cannot put the .boto file on the server (Moblab?) instead of the DUT (device under text, i.e. ChromeOS device in the lab - in this case it's an OnHub/Jetstream) itself. There are some folks in this thread who would know the answers. But I'm also going to add some folks like +Aviv Keshet +Simran Basi because I don't know who does Moblab anymore.

Ben

[Chung-Yih Wang (王崇懿)]

+vovoy

Matt might know it already, recently, we had the issue crbug.com/834659 that all CrOS telemetry benchmarks failed due to the same reason. In CrOS telemetry benchmarks only devservers have the boto file to download wprgo files and copy those wprgo files over to DUTs, so DUTs do not need download those dependencies at all. The logic broke somewhere and Vovo is working on fixing it.