Issue 7885 in angleproject: All end2end tests are skipped on iOS

[ynovi… via monorail]

Status: Available
Owner: ----
CC: jmad...@chromium.org
OS: iOS
Priority: Medium
Renderer: DesktopGL
Type: Defect

New issue 7885 by ynov...@chromium.org: All end2end tests are skipped on iOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=7885

E.g. https://ci.chromium.org/ui/p/chromium/builders/try/ios-angle-try-intel/82/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8794971586551905745/+/u/angle_end2end_tests_iPhone_X_16.1_on_Intel_GPU_on_Mac__with_patch__on_Mac-12.5/json.output
{
"interrupted": false,
"num_failures_by_type": {
"SKIP": 5986
},

Some tests used to run, e.g. https://ci.chromium.org/ui/p/chromium/builders/try/ios-angle-try-intel/70/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8803122987246119009/+/u/angle_end2end_tests_iPhone_X_16.0_on_Intel_GPU_on_Mac__with_patch__on_Mac-12.4/json.output
{
"interrupted": false,
"num_failures_by_type": {
"PASS": 3288,
"SKIP": 2475
},

I wonder what changed.
Now I see in output:
https://chromium-swarm.appspot.com/task?id=5f1fc79af21b5310
[07:32:13:INFO] ../../third_party/angle/src/tests/test_utils/ANGLETest.cpp:355: Failure
[07:32:13:INFO] Expected: (nullptr) != (getProcAddress), actual: (nullptr) vs NULL
[07:32:13:INFO] Stack trace:
[07:32:13:INFO] 0 angle_end2end_tests 0x00000001085e164d testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 43
[07:32:13:INFO] 1 angle_end2end_tests 0x00000001085e123e testing::internal::AssertHelper::operator=(testing::Message const&) const + 88
[07:32:13:INFO] 2 angle_end2end_tests 0x000000010902f436 angle::LoadEntryPointsWithUtilLoader(angle::GLESDriverType) + 201
[07:32:13:INFO] 3 angle_end2end_tests 0x0000000108627e09 RegisterContextCompatibilityTests() + 87
[07:32:13:INFO] 4 angle_end2end_tests 0x00000001090331c2 main + 100
[07:32:13:INFO] 5 angle_end2end_tests 0x000000010904ff92 -[AngleUtilAppDelegate runMain] + 22
[07:32:13:INFO] 6 Foundation 0x000000010ee89266 __NSFireTimer + 67
[07:32:13:INFO] 7 CoreFoundation 0x000000010b20d819 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 20
[07:32:13:INFO] 8 CoreFoundation 0x000000010b20d363 __CFRunLoopDoTimer + 812
[07:32:13:INFO] 9 CoreFoundation 0x000000010b20cada __CFRunLoopDoTimers + 243
[07:32:13:INFO] 10 CoreFoundation 0x000000010b207322 __CFRunLoopRun + 2126
[07:32:13:INFO] 11 CoreFoundation 0x000000010b2066f7 CFRunLoopRunSpecific + 560
[07:32:13:INFO] 12 XCTestCore 0x00000001231445b1 -[XCTWaiter waitForExpectations:timeout:enforceOrder:] + 886
[07:32:13:INFO] 13 XCTestCore 0x00000001231138f6 -[XCTFuture _waitForFulfillment] + 716
[07:32:13:INFO] 14 XCTestCore 0x0000000123114cce -[XCTFuture value] + 32
[07:32:13:INFO] 15 XCTestCore 0x000000012313901d -[XCTestDriver _prepareTestConfigurationAndIDESession] + 677
[07:32:13:INFO] 16 XCTestCore 0x0000000123138d48 -[XCTestDriver run] + 23
[07:32:13:INFO] 17 XCTestCore 0x000000012310caa9 _XCTestMain + 125
[07:32:13:INFO] 18 libXCTestBundleInject.dylib 0x000000010a1da9b5 __copy_helper_block_e8_32s + 0
[07:32:13:INFO] 19 CoreFoundation 0x000000010b20ce29 __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
[07:32:13:INFO] 20 CoreFoundation 0x000000010b20c5e2 __CFRunLoopDoBlocks + 406
[07:32:13:INFO] 21 CoreFoundation 0x000000010b206e65 __CFRunLoopRun + 913
[07:32:13:INFO] 22 CoreFoundation 0x000000010b2066f7 CFRunLoopRunSpecific + 560
[07:32:13:INFO] 23 GraphicsServices 0x000000011033228a GSEventRunModal + 139
[07:32:13:INFO] 24 UIKitCore 0x000000011a0e162b -[UIApplication _run] + 994
[07:32:13:INFO] 25 UIKitCore 0x000000011a0e6547 UIApplicationMain + 123
[07:32:13:INFO] 26 dyld 0x000000010a29a2bf start_sim + 10

This is probably the reason.
Seems like https://chromium-review.googlesource.com/c/angle/angle/+/3925426 is the last CL touching this code.

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[ynovi… via monorail]

Updates:
Blockedon: chromium:1399828
Cc: peng...@chromium.org

Comment #1 on issue 7885 by ynov...@chromium.org: All end2end tests are skipped on iOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=7885#c1

Re-setup my local iOS environment after 13.1 macOS upgrade only to find that this doesn't happen anymore.

I didn't track this on CI originally, so now found it was broken in
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16327/overview
and fixed in
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16328/overview
by https://chromium-review.googlesource.com/c/angle/angle/+/4089450.

Then it got reverted in https://chromium-review.googlesource.com/c/angle/angle/+/4105340 = https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16575/overview.

And relanded in https://chromium-review.googlesource.com/c/angle/angle/+/4106348 = https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16634/overview
so everything looks good now.

Something I don't understand is that crrev.com/c/4106348 should have affected only Linux, so it is quire weird that iOS is affected.

[ynovi… via monorail]

Comment #2 on issue 7885 by ynov...@chromium.org: All end2end tests are skipped on iOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=7885#c2

Something very weird is going on with this bot.
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16200/overview is bad
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16199/overview is good
But they have the exact same Chrome and ANGLE revisions.
The CAS input has a different hash, though.

Both of these tasks are deduped from:
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16185/overview bad
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-intel/16184/overview good
which also have same revisions.

The difference seems to be that the good isolate has:
libEGL.framework/
libGLESv1_CM.framework/
libGLESv2.framework/

While the bad one does not:
https://cas-viewer.appspot.com/projects/chromium-swarm/instances/default_instance/blobs/8a0a3df8474a6a2c8c698400fd7a179be276830d92f6108a46e15f06fe655ac2/796/tree bad
https://cas-viewer.appspot.com/projects/chromium-swarm/instances/default_instance/blobs/1ed6031c67cb38c99c95c5dd8f531b0ee282b8e8b94ea59ecfb27396e0dce677/1078/tree good

So, it looks like iOS build is not deterministic and sometimes packs the files in the isolate and sometimes doesn't.

Looking at the parent builds:
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-builder/101275/overview bad
https://luci-milo.appspot.com/ui/p/chromium/builders/ci/ios-angle-builder/101274/overview good

One difference is that the bad build was on Macmini7,1 https://chromium-swarm.appspot.com/bot?id=build86-m9 and the good build on Macmini8,1 https://chromium-swarm.appspot.com/bot?id=mac-91-h525

[ynovi… via monorail]

Comment #3 on issue 7885 by ynov...@chromium.org: All end2end tests are skipped on iOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=7885#c3

No, the version of MacMini doesn't seem to be the deciding factor.
Just ran a trybot https://ci.chromium.org/ui/p/chromium/builders/try/ios-angle-try-intel/83/overview where all tests were skipped and it was built by Macmini8,1 https://chromium-swarm.appspot.com/bot?id=mac-567-g580.