Clang tools script failure on Windows after clang 7.0.0

[Ramin Halavati]

Hi,

I'm seeing this error on windows, most probably after clang 7.0.0 was rolled. 

When I run the clang tool directly, as "third_party\llvm-build\Release+Asserts\bin\traffic_annotation_extractor.exe" -p out/Release components\spellcheck\browser\spelling_service_client.cc, the tool runs without any error.

But when I ran the clang tool using clang/scripts/run_tool.py, i.e., python tools\clang\scripts\run_tool.py --tool traffic_annotation_extractor --generate-compdb -p out/Release components\spellcheck\browser\spelling_service_client.cc, [at least] one of the includes folder is not found:

Failed to process ../../components/spellcheck/browser/spelling_service_client.cc

In file included from C:\workspace\chromium\src\out\Release\..\..\components\spellcheck\browser\spelling_service_client.cc:5:

../..\components/spellcheck/browser/spelling_service_client.h(8,10):  fatal error: 'map' file not found

#include <map>

         ^~~~~

1 error generated.

Error while processing C:\workspace\chromium\src\out\Release\..\..\components\spellcheck\browser\spelling_service_client.cc.

The script produces this command based on enclosed compile_commands.json:

C:\workspace\chromium\src\third_party\llvm-build\Release+Asserts\bin\traffic_annotation_extractor ../../components/spellcheck/browser/spelling_service_client.cc -- ../../third_party/llvm-build/Release+Asserts/bin/clang-cl.exe --driver-mode=cl /nologo /showIncludes -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....win_sdkInclude10.0.15063.0um -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....win_sdkInclude10.0.15063.0shared -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....win_sdkInclude10.0.15063.0winrt -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....win_sdkInclude10.0.15063.0ucrt -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....VCToolsMSVC14.11.25503include -imsvcC:workspacechromiumsrcthird_partydepot_toolswin_toolchainvs_files1180cb75833ea365097e279efb2d5d7a42dee4b0win_sdkbin....VCToolsMSVC14.11.25503atlmfcinclude -DV8_DEPRECATION_WARNINGS -DUSE_AURA=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION="331747-1" -D_HAS_EXCEPTIONS=0 -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -DDEPRECATEDENUMERATOR(x)=[[deprecated(x)]] -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=1 -DWIN32 -D_SECURE_ATL -D_USING_V110_SDK71_ -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=0x0A000002 -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DWEBP_EXTERN=extern -DUSE_EGL -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=wchar_t -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DV8_USE_EXTERNAL_STARTUP_DATA -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER="gpu/config/gpu_driver_bug_workaround_autogen.h" -DGR_GL_FUNCTION_TYPE=__stdcall -DLEVELDB_PLATFORM_CHROMIUM=1 -DDeleteFile=DeleteFileW -DOS_WIN -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DGTEST_RELATIVE_PATH -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_WIN -DNO_MAIN_THREAD_WRAPPING -I../.. -Igen -I../../third_party/libwebp/src -I../../third_party/wtl/include -I../../third_party/libyuv/include -I../../third_party/khronos -I../../gpu -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/protobuf/src -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/boringssl/src/include -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/gpu -I../../third_party/skia/include/images -I../../third_party/skia/include/lazy -I../../third_party/skia/include/pathops -I../../third_party/skia/include/pdf -I../../third_party/skia/include/pipe -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../third_party/libwebm/source -I../../third_party/blink -Igen/third_party/blink -I../../v8/include -Igen/v8/include -I../../third_party/webrtc_overrides -I../../third_party/webrtc /utf-8 /X -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 /Gy /FS /bigobj /d2FastFail /Zc:sizedDealloc- -fmsc-version=1911 -m64 /W4 -Wimplicit-fallthrough -Wthread-safety /WX /wd4091 /wd4127 /wd4251 /wd4275 /wd4312 /wd4324 /wd4351 /wd4355 /wd4503 /wd4589 /wd4611 /wd4100 /wd4121 /wd4244 /wd4505 /wd4510 /wd4512 /wd4610 /wd4838 /wd4995 /wd4996 /wd4456 /wd4457 /wd4458 /wd4459 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-null-pointer-arithmetic -Wno-ignored-pragma-optimize /O1 /Ob2 /Oy- /Zc:inline /Gw /Oi /Z7 -fno-standalone-debug /MT -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare /wd4267 /TP /wd4577 /GR- /c /Foobj/components/spellcheck/browser/browser/spelling_service_client.obj /Fdobj/components/spellcheck/browser/browser_cc.pdb

Do you have any suggestions on what's wrong or how can I fix the script?

Thanks,

Ramin

[Daniel Cheng]

Can you make sure you're synced past https://chromium-review.googlesource.com/c/chromium/src/+/1049548 before trying to use run_tool.py?

There was a bad version of run_tool.py checked in at one point, and that CL /should/ have fixed it.

Daniel

[Ramin Halavati]

Yes, it's the latest checkout. It's also happening on the trybots, e.g.:

https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.chromium.win%2Fwin7_chromium_rel_ng%2F166308%2F%2B%2Frecipes%2Fsteps%2Fcheck_network_annotations__with_patch_%2F0%2Fstdout

Thanks,

ramin

[Daniel Cheng]

It looks to me like the command-line isn't being properly escaped; that's why the include paths aren't being found.

https://chromium-review.googlesource.com/c/chromium/src/+/1053647 is probably the CL that will need more investigation. +oro...@chromium.org

Daniel

[Ramin Halavati]

Thanks Daniel,

I hadn't noticed before, but it seems that the paths including \\ are not correctly passed. the shlex.split command seems to remove \\.

[Owen Rodley]

Oh yeah, this is because of my change. Sorry. The lack of posix=False in compile_db.py wasn't an issue before, because we didn't read "command" out of the compile db in run_tool. We just read "file" and "directory", and then the clang tool would look up the corresponding entry in the compilation database itself.

I think the correct fix is to check if we're running on Windows, and pass posix=False (in both run_tool.py and compile_db.py) if so.