Running Telemetry on production devices

[James Hartley]

Hi,

 

I am trying to run Telemetery across a range of devices – phones and Chromebooks. 

 

There seems to be limitations when trying to run Telmetery based on whether the device is rooted, has adb root access and if it is a production rather than userspace image.  Can you explain which of these configurations are actually supported?

In addition to this broad question I am having specific problems with the page_cycler tests.   As these tests have Trace enabled by default I run into an error on devices such as the Galaxy S7 and Huawei Mate 8.

 

cd chromium/src/tools/perf

./run_benchmarks –browser=android-system-chrome page_cycler_v2.typical_25

 

Error

“system/bin/sh: can’t create /data/local/chrome-tracing-config.json: Permission denied\r\n”

 

Should I expect to be able to run these telemetry tests on production devices?  If this is true then it really limits the effectiveness of Telemetry as a benchmark.  I’m hoping you know of a way around such a limitation.

 

Thanks,

James

 

 

James Hartley | Principal Engineer, UX Performance, Business Line | ARM james....@arm.com | Skype: jameshartleywork Mobile: +44 77205 98345 Visit the new ARM Connected Community | www.arm.com

 

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Ned]

We need root access on the device to setup certain flags for Android Chrome. IIRC, if there is a way for non root user to also modify chrome flags, that would be a security problem with Chrome.

+Juan Antonio Navarro Pérez 

--
You received this message because you are subscribed to the Google Groups "telemetry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

[James Hartley]

Thanks Ned.

 

My question wasn’t clear enough, sorry.  I have rooted the S7 and Mate 8.  I’m comfortable doing that.   The problems relate to what happens next.  Tracing wants to create temporary files within data/local which it cannot do on a rooted, production handset.

 

But I want to come back to the main question – should I expect to be able to use Telemetry with a standard production devices, such as an S7 or Mate 8, even if I root it?  It’s not clear from the code or my experience so far if this is supported.

[Ned]

I think you need a user debug build on Sam Sung to get through the permission problem. 

[James Hartley]

I agree but I just wanted to check.  Practically what is happening is that I am being blocked by the device having a production build.

 

I was just after the ‘do you guys expect Telemetry to work with a production device?’.  Sounds like the answer is that it’s not been considered too much, which is fine.

 

Thanks.

Error! Filename not specified.

James Hartley | Principal Engineer, UX Performance, Business Line | ARM james....@arm.com | Skype: jameshartleywork Mobile: +44 77205 98345 Visit the new ARM Connected Community | www.arm.com

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
You received this message because you are subscribed to the Google Groups "telemetry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Primiano Tucci]

Wait I'm missing something here. If you have a rooted device how come you can't write to /data/local?

Are you restarting adbd (on the device) as root? telemetry needs the shell account to be root for the reasons mentioned above. 

[James Hartley]

On the Mate 8 : adb root  > ‘adbd cannot run as root in production builds’

 

However, you can get round this in most cases using “adb shell su 0 ‘[cmd]’”.

 

Writing to data/local/tmp is fine on most devices, writing to data/local is not always possible.   From what I can see Chrome is hard wired to read a trace file from this location.  Is that right?   Could it be changed to look in data/local/tmp then at least we’d have a fighting chance?

[Ned]

+Zhen Wang 

[Zhen Wang]

We cannot put chrome-tracing-config.json in /data/local/tmp/ for security reasons.

The shell user and group have write access to /data/local/tmp/ on Android. In chrome, after reading the config file content, the JSON parser in Chrome needs to parse this file content. Chrome JSON parser is in general solid. But it does rely on ICU to decode UTF8 and UTF16, so bugs in that would be bugs in the parser. So we can only allow /data/local/, where the developer is responsible for what he/she is doing.

For more details, see https://docs.google.com/document/d/1yRCXhrQ-0rsfUgNHt9T4YdnmJYrXKN6aK56Ozk3kPVc/edit#heading=h.numjn2sdibi1, the security concern section.

Best

-Zhen

+Zhen Wang 

To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+unsubscribe@chromium.org.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
You received this message because you are subscribed to the Google Groups "telemetry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+unsubscribe@chromium.org.

[James Hartley]

Fantastic response.  Thanks Zhen.

 

So we will have to leave this as you can only run Telemetry on root devices with a user-debug build.  Sigh, such a shame L

 

We were trying to profile performance across production devices but sadly that doesn’t look possible for any workloads which have tracing enabled.   I can Octane no problem (with a couple of hacks to disable the production build check).   I’m guessing that the page cycler tests cannot be run with Trace enabled?  Is that correct?

 

James

Best

-Zhen

 

+Zhen Wang 

 

To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
You received this message because you are subscribed to the Google Groups "telemetry" group.

To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Ned]

That's correct. We are investing in better loading metrics such as TimeToFirstMeaningfulPaint and computing those metrics requires using traces.

[James Hartley]

Thanks again guys for your advice and feedback.  We’re going to switch to a Nexus device with AOSP and see if we have more luck.

 

James

[James Hartley]

Hi again guys,

 

I have tried to get Telemetry working with a Nexus 5X with an AOSP userdebug build and I’m still hitting problems.

 

Steps taken

-          Built AOSP and flashed the handset.  Resulting Build number is ‘aosp_bullhead-userdebug 7.0 NYC eng.aleram.201610301.083439 test-keys’

-          Get latest chromium source on an Ubuntu desktop

-          Run ‘adb devices’ which correctly returns the device ID

-          Run ‘cd ~/chromium/src/tools/perf/;./run_benchmark –browser=list

 

This returns the list of browsers on my desktop not on the Nexus device.  At this point it looks like the adb server on the device dies.   I end up having to factory reset the device to get adb back again!

 

I realize I may have a flakey AOSP build (although we have tried two) but is there anything else I am doing wrong here?

Should I be building an older version on Android?

 

Other bits of info:

 

Running ‘adb root’ on the Ubuntu PC never returns.   I can run the same command on my Mac though.   Both are using adb version 1.0.36.

The Chromium check out and telemetry works fine with an Asus Flip Chromebook.  I can run the Octane benchmark, for example with no issues.

 

Regards,

 

James

[Ned]

+John Budorick 

[James Hartley]

Hi guys,

 

Any advice you have would be very useful at this point.  Our Telemetry profiling has effectively stalled in the absence of a platform which we can reliably use.

 

Could you provide any details of the devices and platforms you use day to day?   Is it just Chromebooks and desktop or are there a range of supported handsets as well?