RE: Chromium Layout tests: Imported tests are missing

[Jacques Levy (jacqlevy)]

 

Hi,

 

We are using chromium on STB (set top box) .

Layout tests are run on our CI each night.

We have noticed, since chromium version 53, a number of failed tests greater than usual failures.

Almost all “imported” tests from layout tests are failing

 

 

Suite	Passed Rate	Failed Rate	Total	Run	Passed	Failed	Skipped	Logs
imported	50.4 %	49.5 %	3657	3309	1670	1639	348	link

 

If we look at the failures, we can see that the root cause is that expected files are not present:

 

Tests that had no expected results (probably new) (1351): [flag all] [unflag all]

test	results	image results	actual	expected
+imported/wpt/IndexedDB/idbobjectstore_get4.htm	result		missing
+imported/wpt/IndexedDB/idbdatabase_createObjectStore8-parameters.htm	result		missing
+imported/wpt/IndexedDB/idbdatabase_close2.htm	result		missing
+imported/wpt/IndexedDB/idbcursor_delete_index4.htm	result		missing
+imported/wpt/IndexedDB/cursor-overloads.htm	result		missing
+imported/wpt/IndexedDB/idbdatabase_deleteObjectStore.htm	result		missing
+imported/wpt/IndexedDB/idbfactory_open10.htm	result		missing
+imported/wpt/IndexedDB/idbobjectstore_add8.htm	result		missing
+imported/wpt/IndexedDB/idbobjectstore_put12.htm	result		missing
+imported/wpt/IndexedDB/idbfactory_cmp2.htm	result		missing
+imported/wpt/IndexedDB/keypath_invalid.htm	result		missing

 

 

CONSOLE ERROR: line 8: Uncaught ReferenceError: add_completion_callback is not defined

CONSOLE ERROR: line 11: Uncaught ReferenceError: async_test is not defined

layer at (0,0) size 800x600

  LayoutView at (0,0) size 800x600

layer at (0,0) size 800x8

  LayoutBlockFlow {HTML} at (0,0) size 800x8

    LayoutBlockFlow {BODY} at (8,8) size 784x0

      LayoutBlockFlow {DIV} at (0,0) size 784x0

 

 

Questions :

 

·       Do you know why expected files have been removed ?

·       Are those tests still relevant ?

·       If yes, is the issue corrected in version > 53

 

 

Thanks for your responses

 

Regards

 

From: k...@google.com [mailto:k...@google.com] On Behalf Of Kenneth Russell
Sent: mercredi 7 décembre 2016 10:34
To: Dirk Pranke <dpr...@google.com>
Cc: Jacques Levy (jacqlevy) <jacq...@cisco.com>
Subject: Re: Chromium Layout tests: Imported tests are missing

 

Also: please ask questions like this on blink-dev@. Not only will you receive more replies, but also others can benefit from the answers.

 

 

On Tue, Dec 6, 2016 at 9:57 AM, Dirk Pranke <dpr...@google.com> wrote:

Hi Jacques,

 

Can you provide more information, like which specific tests were missing, what errors you saw, and whether you see them on tip-of-tree?

 

-- Dirk

 

On Tue, Dec 6, 2016 at 5:04 AM, Jacques Levy (jacqlevy) <jacq...@cisco.com> wrote:

Hi ,

 

We are using chromium 53 (it is not the latest version), but we have noticed that “imported test” have not expected results which makes them failing.

In previous version (52) they were present.

 

Suite	Passed Rate	Failed Rate	Total	Run	Passed	Failed	Skipped	Logs

 

imported           100.0 %                                    0 %                                              3543                    3288              3288                        0                       255                         link

 

Do you know :

Why expected have been removed ? and if those tests will be updated in future versions.

 

Thanks

 

 

 

[Joshua Bell]

That looks like the test is failing to load a dependent script, e.g as if <script src="/resources/testharness.js"> failed to resolve.

My guess is that circa 53 we changed how we map URLs like "/resources/..." to a local directory when running tests. You may need to look for changes in our test runner that your CI environment is not picking up for some reason.

 

layer at (0,0) size 800x600

  LayoutView at (0,0) size 800x600

layer at (0,0) size 800x8

  LayoutBlockFlow {HTML} at (0,0) size 800x8

    LayoutBlockFlow {BODY} at (8,8) size 784x0

      LayoutBlockFlow {DIV} at (0,0) size 784x0

 

 

Questions :

 

·       Do you know why expected files have been removed ?

Most imported tests (based on testharness.js) don't have expected files; if every sub-test reports PASS then no file is needed. Only tests where a sub-test contains FAIL need expected files. 

·       Are those tests still relevant ?

Yes.

·       If yes, is the issue corrected in version > 53

This is not a problem that affected normal Chromium development, either locally or on our bots. (Unless we broke it for a revision and fixed it quickly afterwards). This indicates a problem with your CI environment.