Layout tests on Android

[Łukasz Anforowicz]

Hello,

I was surprised and slightly alarmed that I can't find any steps for running layout test on Android on CQ and/or the Waterfall.  This would mean that some Android-only features (like text-autosizing) have to be compiled on desktop platforms and are tested only on desktop platforms (i.e. not on a platform where end users will actually use the feature).  I am also concerned that some platform-wide settings might not be getting properly exercised (e.g. strict Site Isolation aka site-per-process is the default on desktop platforms but not Android and so it is desirable to have non-site-per-process coverage for Android, but the only way to achieve this today is to have a not-site-per-process-webkit-layout-tests step on a non-Android bot [i.e. on linux_chormium_rel_ng]).

Am I missing something?  FWIW, I see some Android-specific test expectations in third_party/WebKit/LayoutTests/TestExpectations and there are some instructions for manually running layout tests on Android (here and here), but AFAICT layout tests are not covered by CQ and Waterfall, right?  OTOH https://crbug.com/897673 seems to indicate that layout tests are being run on chromium.android/KitKat Phone Tester (dbg), but I can't find layout tests in tryjobs for https://crrev.com/c/1302662/5 ...

This problem seems somewhat similar to the fact that browser_tests are also not run on Android (tracked by https://crbug.com/611756).  

-Lukasz

[Dirk Pranke]

You are correct that we don't currently run the layout tests in the CQ on Android.

Historically this is due to a combination of there not being a compelling need for them (i.e., we haven't found a lot of blink-specific things on Android that would've justified this) and the Android support being under-supported and/or bitrotten as a result. E.g., until recently we couldn't easily run the tests on things other than a Nexus 4 running Android J.

Which, frankly, is kinda sad, but ...

At any rate, John has recently moved things up to newer devices running K, and I think we can run them on M now (I think N+ will still require more work). If there is demand to run the tests in the CQ we can make that happen but we have to scope out how many tests we want to run and what the resource needs will be. I think we'll likely never want to run *all* of the tests on Android, particularly in the CQ, because there's simply too many of them testing only generic functionality and the ROI for that wouldn't be there.

browser_tests is a different set of issues, largely because the architecture for them is fairly different and we haven't invested in what we'd need to do to get the tests to work. There is discussion about fixing that as well, though.

-- Dirk

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7e9219d8-ee9d-4afc-98d2-a74aaddf5b99%40chromium.org.

[John Budorick]

Only the smoke tests are run on Android. I'm not sure which tests beyond that can run on Android, but I am sure that running the entire suite would take an eternity.

As you've noticed, they're running on KitKat Phone Tester (dbg) (which is a waterfall builder; perhaps you're using the term differently?). I'm going to add them to android-kitkat-arm-rel (which is on the CQ) as part of crbug.com/790265 & crbug.com/824539 when I get a chance.

This is a bit different from browser_tests; those aren't supported at all on Android, and will require significant effort to make work. Layout tests do still have issues running on M+ -- they need SELinux disabled until we load fonts from the browser process -- but the harness in general does work.

[Łukasz Anforowicz]

On Wednesday, October 31, 2018 at 11:53:07 AM UTC-7, Dirk Pranke wrote:

You are correct that we don't currently run the layout tests in the CQ on Android.

Historically this is due to a combination of there not being a compelling need for them (i.e., we haven't found a lot of blink-specific things on Android that would've justified this)

I wonder if a similar thing could be said about other platforms - i.e. is there a lot of blink specific things on Mac or ChromeOS?  FWIW, I do see layout tests on Mac bots that are part of CQ, but on ChromeOS...

I also see that Android bots on CQ (e.g. android-marshmallow-arm64-rel) do run some blink tests (e.g. webkit_unit_tests), so it seems to me that lack of layout test coverage is mostly a function of implementation/support-difficulty and lack of investment in running these tests on Android (rather than something originating out of a lack of compelling need).

 

and the Android support being under-supported and/or bitrotten as a result. E.g., until recently we couldn't easily run the tests on things other than a Nexus 4 running Android J. 

Which, frankly, is kinda sad, but ...

At any rate, John has recently moved things up to newer devices running K, and I think we can run them on M now (I think N+ will still require more work). If there is demand to run the tests in the CQ we can make that happen but we have to scope out how many tests we want to run and what the resource needs will be. I think we'll likely never want to run *all* of the tests on Android, particularly in the CQ, because there's simply too many of them testing only generic functionality and the ROI for that wouldn't be there.

Well, I think we do want to run *all* //content-layer and below tests with and without OOPIFs (one reason is that Android won't have strict Site Isolation by default and we want to make sure that layers that are most heavily involved in OOPIF support (e.g. //content and Blink) get sufficient coverage here).  We can always revisit this coverage decision, but I think this was the conclusion of the discussion we had in August.  AFAIK content_browsertests, content_unittests and webkit_unit_tests are run on Android bots in the CQ and therefore give us non-OOPIF coverage.

I think the choice for getting non-OOPIF coverage for layout tests is to run *all* layout tests either 1) in an extra step on linux_chromium_rel_ng (see for example the CL here) VS 2) on one of Android bots.  I hope that #1 is okay for the short-term.  I appreciate the difficulty of supporting some kinds of testing on Android platform, but I think that in the long-term it is best to run tests (layout tests, browser_tests and other tests) on the real platform (rather than only approximate the platform by flipping a few switches [like site-per-process] on another platform).

[Dirk Pranke]

On Thu, Nov 1, 2018 at 12:58 PM, Łukasz Anforowicz <luk...@chromium.org> wrote:

On Wednesday, October 31, 2018 at 11:53:07 AM UTC-7, Dirk Pranke wrote:

You are correct that we don't currently run the layout tests in the CQ on Android.

Historically this is due to a combination of there not being a compelling need for them (i.e., we haven't found a lot of blink-specific things on Android that would've justified this)

I wonder if a similar thing could be said about other platforms - i.e. is there a lot of blink specific things on Mac or ChromeOS?  FWIW, I do see layout tests on Mac bots that are part of CQ, but on ChromeOS...

Yes, the same thing could be said about the other platforms. Android and ChromeOS are special insofar as historically they were an order of magnitude slower and harder to manage.

I would agree that we probably waste a lot of resources running tests redundantly (unnecessarily) across mac/win/linux. The layout tests are not unique in this regard, either. However, we don't have good tooling for figuring out *which* tests are platform-specific or managing which things run on which platforms, so it's easier to just run them everywhere.

I also see that Android bots on CQ (e.g. android-marshmallow-arm64-rel) do run some blink tests (e.g. webkit_unit_tests), so it seems to me that lack of layout test coverage is mostly a function of implementation/support-difficulty and lack of investment in running these tests on Android (rather than something originating out of a lack of compelling need). 

As I said, it's a combination of both things.

 

 

and the Android support being under-supported and/or bitrotten as a result. E.g., until recently we couldn't easily run the tests on things other than a Nexus 4 running Android J. 

Which, frankly, is kinda sad, but ...

At any rate, John has recently moved things up to newer devices running K, and I think we can run them on M now (I think N+ will still require more work). If there is demand to run the tests in the CQ we can make that happen but we have to scope out how many tests we want to run and what the resource needs will be. I think we'll likely never want to run *all* of the tests on Android, particularly in the CQ, because there's simply too many of them testing only generic functionality and the ROI for that wouldn't be there.

Well, I think we do want to run *all* //content-layer and below tests with and without OOPIFs (one reason is that Android won't have strict Site Isolation by default and we want to make sure that layers that are most heavily involved in OOPIF support (e.g. //content and Blink) get sufficient coverage here).  We can always revisit this coverage decision, but I think this was the conclusion of the discussion we had in August.  AFAIK content_browsertests, content_unittests and webkit_unit_tests are run on Android bots in the CQ and therefore give us non-OOPIF coverage.

I think the choice for getting non-OOPIF coverage for layout tests is to run *all* layout tests either 1) in an extra step on linux_chromium_rel_ng (see for example the CL here) VS 2) on one of Android bots.  I hope that #1 is okay for the short-term.  I appreciate the difficulty of supporting some kinds of testing on Android platform, but I think that in the long-term it is best to run tests (layout tests, browser_tests and other tests) on the real platform (rather than only approximate the platform by flipping a few switches [like site-per-process] on another platform).

#1 is far easier and cheaper to do than #2, so we should do #1 if we can and it's sufficient. I agree with your statement in theory but like everything it's a matter of resources and tradeoffs in practice. 

-- Dirk

browser_tests is a different set of issues, largely because the architecture for them is fairly different and we haven't invested in what we'd need to do to get the tests to work. There is discussion about fixing that as well, though.

-- Dirk

On Wed, Oct 31, 2018 at 10:43 AM, Łukasz Anforowicz <luk...@chromium.org> wrote:

Hello,

I was surprised and slightly alarmed that I can't find any steps for running layout test on Android on CQ and/or the Waterfall.  This would mean that some Android-only features (like text-autosizing) have to be compiled on desktop platforms and are tested only on desktop platforms (i.e. not on a platform where end users will actually use the feature).  I am also concerned that some platform-wide settings might not be getting properly exercised (e.g. strict Site Isolation aka site-per-process is the default on desktop platforms but not Android and so it is desirable to have non-site-per-process coverage for Android, but the only way to achieve this today is to have a not-site-per-process-webkit-layout-tests step on a non-Android bot [i.e. on linux_chormium_rel_ng]).

Am I missing something?  FWIW, I see some Android-specific test expectations in third_party/WebKit/LayoutTests/TestExpectations and there are some instructions for manually running layout tests on Android (here and here), but AFAICT layout tests are not covered by CQ and Waterfall, right?  OTOH https://crbug.com/897673 seems to indicate that layout tests are being run on chromium.android/KitKat Phone Tester (dbg), but I can't find layout tests in tryjobs for https://crrev.com/c/1302662/5 ...

This problem seems somewhat similar to the fact that browser_tests are also not run on Android (tracked by https://crbug.com/611756).  

-Lukasz

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7e9219d8-ee9d-4afc-98d2-a74aaddf5b99%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/50452bdd-a296-43f6-ab95-9ae666457f98%40chromium.org.

[Joyer Huang]

Since M69 layout test failed to run on android K. 

Is there a plan to fix this?

I tried forcing layout test to run in single-process mode. It can pass a lot of test cases, but crash immediately after clean up and preparing the next .html case.