help with mojo-related compile issue (url/mojo/url.mojom-shared-internal.h: No such file or directory)

[Paweł Hajdan, Jr.]

Hello folks,

I've encountered a compile issue with chromium-61.0.3163.25, also reported in https://bugs.gentoo.org/show_bug.cgi?id=626684 for 61.0.3163.13. That earlier version compiled successfully for me (and so did .25 on clobber retry). I suspect it could be non-deterministic build issue caused by incomplete dependencies between targets.

FAILED: obj/third_party/WebKit/Source/platform/loader/loader/ResourceRequest.o

x86_64-pc-linux-gnu-g++ -MMD -MF obj/third_party/WebKit/Source/platform/loader/loader/ResourceRequest.o.d -DBLINK_PLATFORM_IMPLEMENTATION=1 -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DNO_TCMALLOC -DDISABLE_NACL -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DBLINK_IMPLEMENTATION=1 -DINSIDE_BLINK -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSK_SUPPORT_GPU=1 -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -I../.. -Igen -I../../third_party/WebKit/Source -I../../third_party/WebKit -Igen/blink -Igen/third_party/WebKit -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -Igen/shim_headers/icui18n_shim -Igen/shim_headers/icuuc_shim -Igen/shim_headers/zlib_shim -Igen/shim_headers/libpng_shim -Igen/shim_headers/harfbuzz_shim -Igen/shim_headers/libjpeg_shim -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/protobuf/src -I../../third_party/boringssl/src/include -I/usr/include/nss -I/usr/include/nspr -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/images -I../../third_party/skia/include/lazy -I../../third_party/skia/include/pathops -I../../third_party/skia/include/pdf -I../../third_party/skia/include/pipe -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/third_party/vulkan -I../../third_party/skia/include/gpu -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../v8/include -Igen/v8/include -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -fPIC -pipe -pthread -m64 -march=x86-64 -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -fomit-frame-pointer -g0 -fvisibility=hidden -std=gnu++11 -Wno-narrowing -fno-rtti -fno-exceptions -fvisibility-inlines-hidden -O2 -pipe -march=native -fno-delete-null-pointer-checks -c ../../third_party/WebKit/Source/platform/loader/fetch/ResourceRequest.cpp -o obj/third_party/WebKit/Source/platform/loader/loader/ResourceRequest.o

In file included from gen/storage/public/interfaces/blobs.mojom-shared.h:25:0,

                 from gen/storage/public/interfaces/blobs.mojom-blink.h:33,

                 from ../../third_party/WebKit/Source/platform/blob/BlobData.h:41,

                 from ../../third_party/WebKit/Source/platform/network/EncodedFormData.h:23,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/ResourceRequest.h:34,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/ResourceRequest.cpp:27:

gen/storage/public/interfaces/blobs.mojom-shared-internal.h:19:48: fatal error: url/mojo/url.mojom-shared-internal.h: No such file or directory

 #include "url/mojo/url.mojom-shared-internal.h"

                                                ^

compilation terminated.

The header not found in above example should be generated from https://cs.chromium.org/chromium/src/url/mojo/url.mojom?q=url/mojo/url.mojom-shared-internal.h&sq=package:chromium (https://cs.chromium.org/chromium/src/url/mojo/BUILD.gn?q=url.mojom+file:build&sq=package:chromium&dr=C&l=7).

What really confuses me is that https://cs.chromium.org/chromium/src/storage/public/interfaces/BUILD.gn?type=cs&q=blobs.mojom+file:build&sq=package:chromium&l=9 which generates the blobs header depends on the url mojom:

mojom("interfaces") {

  sources = [

    "blobs.mojom",

  ]

  public_deps = [

    "//mojo/common:common_custom_types",

    "//url/mojo:url_mojom_gurl",

  ]

  export_class_attribute = "STORAGE_COMMON_EXPORT"

  export_define = "STORAGE_COMMON_IMPLEMENTATION=1"

  export_header = "storage/common/storage_common_export.h"

  export_class_attribute_blink = "BLINK_PLATFORM_EXPORT"

  export_define_blink = "BLINK_PLATFORM_IMPLEMENTATION=1"

  export_header_blink = "third_party/WebKit/public/platform/WebCommon.h"

}

Do you have ideas how to debug this further? Please let me know if you have questions or would like me to provide more info / test results.

Paweł

[Scott Violet]

Might the issue be that the the target that compiles ResourceRequest.cpp (third_party/WebKit/Source/platform/loader) doesn't have a dep on the target that provides EncodedFormData.h (third_party/WebKit/Source/platform) ? The comments indicate the loader target is only intended for consumption by the platform target, yet loader doesn't depend on platform.

  -Scott

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-mojo+unsubscribe@chromium.org.
To post to this group, send email to chromi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAATLsPY3bF5be-Q23Yx%2BdPqDmMbocD5OuHmVG142_6tJ6QWKGg%40mail.gmail.com.

[Ken Rockot]

Yes, looks like a loader dependency issue. There's a bug here btw which is probably the same thing: https://crbug.com/750883

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAKARY_nCx%3DfM8BUXdJmuJ_C6b4SuUg-Oaz9JodQW-ty6KV-ybw%40mail.gmail.com.

[Paweł Hajdan, Jr.]

Thank you for the pointer.

I'm getting a similar error with 62.0.3188.2. Is this expected?

In file included from gen/storage/public/interfaces/blobs.mojom-shared.h:25:0,

                 from gen/storage/public/interfaces/blobs.mojom-blink.h:33,

                 from ../../third_party/WebKit/Source/platform/blob/BlobData.h:42,

                 from ../../third_party/WebKit/Source/platform/network/EncodedFormData.h:23,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/ResourceRequest.h:34,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/Resource.h:40,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/MemoryCache.h:32,

                 from ../../third_party/WebKit/Source/platform/loader/fetch/MemoryCache.cpp:23:

gen/storage/public/interfaces/blobs.mojom-shared-internal.h:19:48: fatal error: url/mojo/url.mojom-shared-internal.h: No such file or directory

 #include "url/mojo/url.mojom-shared-internal.h"

                                                ^

compilation terminated.

Paweł

[Ken Rockot]

On Wed, Aug 23, 2017 at 12:38 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Thank you for the pointer.

I'm getting a similar error with 62.0.3188.2. Is this expected?

Only expected insofar as flakiness is expected when dependencies are underspecified.

It looks like we have a dependency on some internally generated target for blob interfaces, but we're missing a necessary dependency on another internally generated target. +mek who added the line.

If fixing the dependency issue by depending on storage/public/interfaces:interfaces_blink is not an option, we should at least add interfaces_shared__generator to the deps. :(

But... can we just fix the dependency issue somehow? I assume it's an issue of circular deps?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAATLsPYj4kYHZQ0%3Dzh0tZQatBdpUcwJpNPw_LzH5VP5FbeJQ1A%40mail.gmail.com.

[Paweł Hajdan, Jr.]

Looks like this happened again (https://bugs.chromium.org/p/chromium/issues/detail?id=766183).

Is it possible a more systemic fix may be needed?

I wonder if gn --check, or a similar mechanism, could be helpful to flag header dependency issues earlier.

Paweł

[Ken Rockot]

We can certainly whip up a Python script that does build-config-wide validation of mojom imports vs deps, but's going to have to touch a lot of files and thus probably be kinda slow, especially on Windows. Not sure what's the best way to ensure something like that gets run appropriately.

I don't suppose we have any way to efficiently determine what build targets (the target definitions themselves, not the build artifacts they produce) in a given config are affected by a diff?

[Paweł Hajdan, Jr.]

+gn-dev

I'm not sure if this would be slower than gn --check, especially if it could be part of it (not being a GN expert, I don't know if it's feasible).

Either way, we might be able to deal with slowness by only running this on already slow bots, or even an FYI one, but catching the problems reliably, and not as flakes.

I suggest focusing on finding a way to automatically verify mojo dependencies, and finding best place for it in our infrastructure as the next step.

Paweł

[Tomasz Śniatowski]

+chromium-dev (mail bounced; sorry)

On Thu, Sep 21, 2017 at 2:50 PM, Tomasz Śniatowski <tsnia...@vewd.com> wrote:

Hi!

Can I interest you all in https://bugs.chromium.org/p/chromium/issues/detail?id=655123 where I use a patched ninja (pull request at https://github.com/ninja-build/ninja/pull/1331) to catch a lot of build flakes like these? It runs in some 10-15 seconds (on a not-so-high-end linux machine, tests on other platforms welcome) so may be suitable for running alongside the "ninja nothing to do" check that already runs. 

--
You received this message because you are subscribed to the Google Groups "gn-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

--

--
Tomasz Śniatowski

--

--
Tomasz Śniatowski