cxx_indexer failing with compiler errors on translation units that compile fine with clang++

151 views
Skip to first unread message

Paul Seyfert

unread,
Feb 10, 2020, 10:04:01 AM2/10/20
to Kythe
Hi,

I'm trying to run kythe on https://gitlab.cern.ch/gaudi/Gaudi/.

Building it is unfortunately a bit involved, so while originally I tried to build a kythe index for the entire project, I'm focussing in this thread on just one TU that shows that I'm doing something wrong / kythe can't do for me what I expect it to do. I hope the writeup is not too much of a mess and I copied everything correctly together.

What I did (the first few steps are just an explanation how I ended up with - what I think are - the interesting commands towards the end, if you want to save yourself from the mess, skip forward to step 5):

 0. I downloaded kythe 0.0.41 and checked out the corresponding tag from github. It's unpacked to /tmp/tmp.YfWrn8brsT/kythe-v0.0.41.
 1. I did a "regular" clang build. Our default setup for that is that `make` in the top level directory calls `cmake` with the appropriate arguments to create a build directory in which ninja builds with clang. (yes, unconventional, sorry)
   In when running the top level `make` with `-n`, I see that `cmake` should get called with

```sh
```

   That means the instructions from [the docs](https://github.com/kythe/kythe/pull/4173/files) won't work for me as a) unfortunately [here](https://github.com/kythe/kythe/blob/v0.0.41/kythe/go/extractors/config/runextractor/cmakecmd/cmakecmd.go#L112) cmake gets called without `-DCMAKE_TOOLCHAIN_FILE`. (to be fair, I could create a build directory manually, have CMAKE_TOOLCHAIN_FILE set and then run `runextractor cmake`) and b) that sets (invisible from the call) `CMAKE_CXX_COMPILER` to `lcg-clang++-8.0.0`, which would get overridden through `-DCMAKE_CXX_COMPILER=clang++`.

 2. I therefore run my build manually `make; cd /afs/cern.ch/work/p/pseyfert/ofun/Gaudi/build.x86_64-centos7-clang8-opt; cmake --build .`
   `CMAKE_EXPORT_COMPILE_COMMANDS` is ON by default so the `compile_commands.json` gets created and with youcompleteme or clang-tidy of include-what-you-use I can verify that the `json` file is okay.

 3. I check out tag 0.0.41 of kythe, comment out line 112-118 in `cmakecmd.go`, rebuild `runextractor`, and run it as described in the docs, and watch with `ps` what processes get called from `ExtractCompilations`.
 4. I expect the extracted compilations to be slightly incorrect because my clang executable `lcg-clang++-8.0.0` is just an executable shell script:

```sh
#!/bin/sh
LD_LIBRARY_PATH=$(printenv LD_LIBRARY_PATH | sed 's-[^:]*/\(gcc\|llvm\|clang\)/[^:]*:\?--g')
```

    This doesn't do anything spectacular other than setting PATH and LD_LIBRARY_PATH correctly such that the compiler binary works as intended (works around issues that we have with compilers not being linked against the OS's `libstdc++.so`), adds `--gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-centos7`, and passes all arguments on to the compiler binary. I suspect `--gcc-toolchain` should be passed to the cxx_extractor (at least other clang based tools I worked with need it), so after I saw what subprocesses get spawned by `runextractor` I run one of them by hand (step 6).

 5. I set the KYTHE environment variables. I also make sure the output directory exists (https://github.com/kythe/kythe/issues/4174):

```sh
export KYTHE_CORPUS=Gaudi
export KYTHE_ROOT_DIRECTORY=/afs/cern.ch/work/p/pseyfert/ofun/Gaudi
export KYTHE_OUTPUT_DIRECTORY=/tmp/doesexist
mkdir -p $KYTHE_OUTPUT_DIRECTORY
```

 6. I set `PATH` and `LD_LIBRARY_PATH to the values I got from the above clang launcher, `cd` into my build directory and launch the cxx_extractor

```cpp
/tmp/tmp.YfWrn8brsT/kythe-v0.0.41/extractors/cxx_extractor --with_executable /cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7/lcg-clang++-8.0.0 -DBOOST_FILESYSTEM_VERSION=3 -DBOOST_SPIRIT_USE_PHOENIX_V3 -DGAUDI_V20_COMPAT -DGaudiSvc_EXPORTS -DPACKAGE_NAME=\"GaudiSvc\" -DPACKAGE_VERSION=\"v33r0\" -DTBB_USE_GLIBCXX_VERSION=80000 -D_GNU_SOURCE -Df2cFortran -Dlinux -Dunix -IGaudiSvc -I../GaudiSvc -I../GaudiPluginService -I../GaudiKernel -Iinclude -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/clhep/2.4.1.2/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/cppgsl/2.0.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/tbb/2019_U7/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/ROOT/6.18.04/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/Boost/1.70.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/rangev3/0.5.0/x86_64-centos7-clang8-opt/include -march=x86-64 -msse4.2 -fmessage-length=0 -pipe -Wall -Wextra -Werror=return-type -pthread -pedantic -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -std=c++17 -fdiagnostics-color -O3 -DNDEBUG -fPIC -MD -MT GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -MF GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o.d -o GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -c ../GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp  --gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-cent
```

    This finishes successfully ($? is 0) without errors or warnings. The resulting kzip file can be downloaded [here](https://cernbox.cern.ch/index.php/s/Wlbs4ooTPVnuuMc).

 7. When running the `cxx_indexer` with `/tmp/tmp.YfWrn8brsT/kythe-v0.0.41/indexers/cxx_indexer --ignore_unimplemented /tmp/doesexist/* > /tmp/entries` I get the following error messages repeated several times over
```
pseyfert@pseyfert-centos:/tmp/doesexist >> /tmp/tmp.YfWrn8brsT/kythe-v0.0.41/indexers/cxx_indexer --ignore_unimplemented /tmp/doesexist/* > /tmp/entries
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:84:54: error: no type named 'Factory' in 'HepRndm::Generator<Rndm::Bit>'
      template <typename T, typename F = typename T::Factory>
                                         ~~~~~~~~~~~~^~~~~~~
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:85:14: note: in instantiation of default argument for 'DeclareFactory<HepRndm::Generator<Rndm::Bit> >' required here
      struct DeclareFactory {
             ^
```

    The output file `/tmp/entries/` isn't empty. Now it's not clear to me what to expect here. I would've thought if clang can build HepRndmGenerators.cpp then so should the cxx_indexer (at least that's what I'm used to with other clang based tools. btw: my understanding is that the c++ parts of kythe are clang based?). The cxx_indexer doesn't fail on all translation units and when restricting myself to a few translation units that go through the cxx_indexer without errors I reached the end of [the docs](https://raw.githubusercontent.com/kythe/kythe/master/kythe/web/site/examples.md) and could run `http_server -- ....` and look at the code index in my webbrowser (well, that was last time I tried with v0.0.30 anyways). At this point I'm a bit lost how to debug / where to suspect errors on my side or wonder where my tools are just incompatible (NB: I only have the build system and compilers for clang 8 ready, I didn't see if cxx_indexer is clang 9 based and I don't have an ETA for clang 9. Also I know that for include-what-you-use or cpp-insights I often need to make sure clang's system headers are in the right place with [hacks like this](https://gitlab.cern.ch/pseyfert/lb-compiler-explorer/blob/master/Dockerfile#L21) or as [discussed here](https://github.com/include-what-you-use/include-what-you-use/issues/100), no idea if that's relevant for kythe.

Any hint, pointers or comments for me? Maybe a few easy questions from my side:

 * Should kythe just work on a project that builds fine with cmake and clang?
 * Should I be suspicious if I haven't tried building with clang9 yet?
 * Should cxx_indexer be able to digest any tu that clang++ can handle?
 * Are there any other information I could provide or things I should try? (like a docker container with the directories set up would require some work and time from me and would also require running privileged containers for the (world readable) network mounts that are involved, but should be possible)

Thanks in advance,
Paul

Shahms King

unread,
Feb 10, 2020, 12:35:12 PM2/10/20
to Paul Seyfert, Kythe
On Mon, Feb 10, 2020 at 7:04 AM Paul Seyfert <pseyfert.c...@gmail.com> wrote:
Hi,

I'm trying to run kythe on https://gitlab.cern.ch/gaudi/Gaudi/.

Building it is unfortunately a bit involved, so while originally I tried to build a kythe index for the entire project, I'm focussing in this thread on just one TU that shows that I'm doing something wrong / kythe can't do for me what I expect it to do. I hope the writeup is not too much of a mess and I copied everything correctly together.

What I did (the first few steps are just an explanation how I ended up with - what I think are - the interesting commands towards the end, if you want to save yourself from the mess, skip forward to step 5):

 0. I downloaded kythe 0.0.41 and checked out the corresponding tag from github. It's unpacked to /tmp/tmp.YfWrn8brsT/kythe-v0.0.41.
 1. I did a "regular" clang build. Our default setup for that is that `make` in the top level directory calls `cmake` with the appropriate arguments to create a build directory in which ninja builds with clang. (yes, unconventional, sorry)
   In when running the top level `make` with `-n`, I see that `cmake` should get called with

```sh
```

   That means the instructions from [the docs](https://github.com/kythe/kythe/pull/4173/files) won't work for me as a) unfortunately [here](https://github.com/kythe/kythe/blob/v0.0.41/kythe/go/extractors/config/runextractor/cmakecmd/cmakecmd.go#L112) cmake gets called without `-DCMAKE_TOOLCHAIN_FILE`. (to be fair, I could create a build directory manually, have CMAKE_TOOLCHAIN_FILE set and then run `runextractor cmake`) and b) that sets (invisible from the call) `CMAKE_CXX_COMPILER` to `lcg-clang++-8.0.0`, which would get overridden through `-DCMAKE_CXX_COMPILER=clang++`.

 2. I therefore run my build manually `make; cd /afs/cern.ch/work/p/pseyfert/ofun/Gaudi/build.x86_64-centos7-clang8-opt; cmake --build .`
   `CMAKE_EXPORT_COMPILE_COMMANDS` is ON by default so the `compile_commands.json` gets created and with youcompleteme or clang-tidy of include-what-you-use I can verify that the `json` file is okay.


For sufficiently complex CMake builds, I suspect this is in fact the best option, as the CMake extractor itself is just a small wrapper around the compile_commands.json extractor (as you've noticed). 
 
 3. I check out tag 0.0.41 of kythe, comment out line 112-118 in `cmakecmd.go`, rebuild `runextractor`, and run it as described in the docs, and watch with `ps` what processes get called from `ExtractCompilations`.
 4. I expect the extracted compilations to be slightly incorrect because my clang executable `lcg-clang++-8.0.0` is just an executable shell script:

```sh
#!/bin/sh
LD_LIBRARY_PATH=$(printenv LD_LIBRARY_PATH | sed 's-[^:]*/\(gcc\|llvm\|clang\)/[^:]*:\?--g')
```

    This doesn't do anything spectacular other than setting PATH and LD_LIBRARY_PATH correctly such that the compiler binary works as intended (works around issues that we have with compilers not being linked against the OS's `libstdc++.so`), adds `--gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-centos7`, and passes all arguments on to the compiler binary. I suspect `--gcc-toolchain` should be passed to the cxx_extractor (at least other clang based tools I worked with need it), so after I saw what subprocesses get spawned by `runextractor` I run one of them by hand (step 6).

This is potentially going to cause problems during indexing and may in fact be the source of the issues.  Clang uses the executable name and path to configure a variety of options, so if this isn't correct and you have a toolchain which differs from the "default" you're likely to encounter issues (as you've seen).  While I can't definitively say this is the source of the issues it is high on the list of things to check.  
 

 5. I set the KYTHE environment variables. I also make sure the output directory exists (https://github.com/kythe/kythe/issues/4174):

```sh
export KYTHE_CORPUS=Gaudi
export KYTHE_ROOT_DIRECTORY=/afs/cern.ch/work/p/pseyfert/ofun/Gaudi
export KYTHE_OUTPUT_DIRECTORY=/tmp/doesexist
mkdir -p $KYTHE_OUTPUT_DIRECTORY
```

 6. I set `PATH` and `LD_LIBRARY_PATH to the values I got from the above clang launcher, `cd` into my build directory and launch the cxx_extractor

```cpp
/tmp/tmp.YfWrn8brsT/kythe-v0.0.41/extractors/cxx_extractor --with_executable /cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7/lcg-clang++-8.0.0 -DBOOST_FILESYSTEM_VERSION=3 -DBOOST_SPIRIT_USE_PHOENIX_V3 -DGAUDI_V20_COMPAT -DGaudiSvc_EXPORTS -DPACKAGE_NAME=\"GaudiSvc\" -DPACKAGE_VERSION=\"v33r0\" -DTBB_USE_GLIBCXX_VERSION=80000 -D_GNU_SOURCE -Df2cFortran -Dlinux -Dunix -IGaudiSvc -I../GaudiSvc -I../GaudiPluginService -I../GaudiKernel -Iinclude -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/clhep/2.4.1.2/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/cppgsl/2.0.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/tbb/2019_U7/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/ROOT/6.18.04/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/Boost/1.70.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/rangev3/0.5.0/x86_64-centos7-clang8-opt/include -march=x86-64 -msse4.2 -fmessage-length=0 -pipe -Wall -Wextra -Werror=return-type -pthread -pedantic -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -std=c++17 -fdiagnostics-color -O3 -DNDEBUG -fPIC -MD -MT GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -MF GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o.d -o GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -c ../GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp  --gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-cent
```

    This finishes successfully ($? is 0) without errors or warnings. The resulting kzip file can be downloaded [here](https://cernbox.cern.ch/index.php/s/Wlbs4ooTPVnuuMc).

Given that the indexing problems are type-related and not just include-path related, this is unsurprising as the extract only runs the preprocessor to find required files.
 

 7. When running the `cxx_indexer` with `/tmp/tmp.YfWrn8brsT/kythe-v0.0.41/indexers/cxx_indexer --ignore_unimplemented /tmp/doesexist/* > /tmp/entries` I get the following error messages repeated several times over
```
pseyfert@pseyfert-centos:/tmp/doesexist >> /tmp/tmp.YfWrn8brsT/kythe-v0.0.41/indexers/cxx_indexer --ignore_unimplemented /tmp/doesexist/* > /tmp/entries
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:84:54: error: no type named 'Factory' in 'HepRndm::Generator<Rndm::Bit>'
      template <typename T, typename F = typename T::Factory>
                                         ~~~~~~~~~~~~^~~~~~~
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:85:14: note: in instantiation of default argument for 'DeclareFactory<HepRndm::Generator<Rndm::Bit> >' required here
      struct DeclareFactory {
             ^
```

    The output file `/tmp/entries/` isn't empty. Now it's not clear to me what to expect here. I would've thought if clang can build HepRndmGenerators.cpp then so should the cxx_indexer (at least that's what I'm used to with other clang based tools. btw: my understanding is that the c++ parts of kythe are clang based?). The cxx_indexer doesn't fail on all translation units and when restricting myself to a few translation units that go through the cxx_indexer without errors I reached the end of [the docs](https://raw.githubusercontent.com/kythe/kythe/master/kythe/web/site/examples.md) and could run `http_server -- ....` and look at the code index in my webbrowser (well, that was last time I tried with v0.0.30 anyways). At this point I'm a bit lost how to debug / where to suspect errors on my side or wonder where my tools are just incompatible (NB: I only have the build system and compilers for clang 8 ready, I didn't see if cxx_indexer is clang 9 based and I don't have an ETA for clang 9. Also I know that for include-what-you-use or cpp-insights I often need to make sure clang's system headers are in the right place with [hacks like this](https://gitlab.cern.ch/pseyfert/lb-compiler-explorer/blob/master/Dockerfile#L21) or as [discussed here](https://github.com/include-what-you-use/include-what-you-use/issues/100), no idea if that's relevant for kythe.

Any hint, pointers or comments for me? Maybe a few easy questions from my side:

 * Should kythe just work on a project that builds fine with cmake and clang?

It depends substantially on how complex the build process is.  For fairly simple or straightforward projects, yes.  For projects which do more involved toolchain configuration, such as this, I would not expect things to work out of the box.
 
 * Should I be suspicious if I haven't tried building with clang9 yet? 
 * Should cxx_indexer be able to digest any tu that clang++ can handle?

Depending on the versions of each and excluding build processes which use clang/C++20 modules[1], yes. 

 * Are there any other information I could provide or things I should try? (like a docker container with the directories set up would require some work and time from me and would also require running privileged containers for the (world readable) network mounts that are involved, but should be possible)

I think the next thing to try would be changing the command in the produced compile_commands.json to point to the actual clang binary that gets exec'd and includes the `--gcc-toolchain` option.

--Shahms
 

Thanks in advance,
Paul

--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/3dc07640-bdab-49ca-8155-baa5c62f117b%40googlegroups.com.


--
:-P

Shahms King

unread,
Feb 10, 2020, 12:37:33 PM2/10/20
to Paul Seyfert, Kythe
Note that the included (?) http_server is not particularly actively maintained and may or may not work, unfortunately.  There are a few other UIs that have been written which may serve you better, including UnderHood (discussed at https://groups.google.com/g/kythe/c/9XzG5e0JyeQ/m/BRLGTfm-CwAJ)

--Shahms
--
:-P

Paul Seyfert

unread,
Feb 11, 2020, 8:12:49 AM2/11/20
to Kythe
Am Montag, 10. Februar 2020 18:35:12 UTC+1 schrieb Shahms King:


On Mon, Feb 10, 2020 at 7:04 AM Paul Seyfert <pseyfert....@gmail.com> wrote:
…

 * Are there any other information I could provide or things I should try? (like a docker container with the directories set up would require some work and time from me and would also require running privileged containers for the (world readable) network mounts that are involved, but should be possible)

I think the next thing to try would be changing the command in the produced compile_commands.json to point to the actual clang binary that gets exec'd and includes the `--gcc-toolchain` option.

Thanks for the comments,

 So I set PATH and LD_LIBRARY_PATH to the values from the clang launcher and use the binary for compilation in the build directory:
```
cd /afs/cern.ch/work/p/pseyfert/ofun/Gaudi/build.x86_64-centos7-clang8-opt
/cvmfs/lhcb.cern.ch/lib/lcg/releases/clang/8.0.0/x86_64-centos7/bin/clang++ --gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-centos7 -DBOOST_FILESYSTEM_VERSION=3 -DBOOST_SPIRIT_USE_PHOENIX_V3 -DGAUDI_V20_COMPAT -DGaudiSvc_EXPORTS -DPACKAGE_NAME=\"GaudiSvc\" -DPACKAGE_VERSION=\"v33r0\" -DTBB_USE_GLIBCXX_VERSION=80000 -D_GNU_SOURCE -Df2cFortran -Dlinux -Dunix -IGaudiSvc -I../GaudiSvc -I../GaudiPluginService -I../GaudiKernel -Iinclude -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/clhep/2.4.1.2/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/cppgsl/2.0.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/tbb/2019_U7/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/ROOT/6.18.04/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/Boost/1.70.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/rangev3/0.5.0/x86_64-centos7-clang8-opt/include -march=x86-64 -msse4.2 -fmessage-length=0 -pipe -Wall -Wextra -Werror=return-type -pthread -pedantic -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -std=c++17 -fdiagnostics-color -O3 -DNDEBUG -fPIC -MD -MT GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -MF GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o.d -o GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -c ../GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp
```
This compiles fine, no errors/warnings, $? is 0.

I then run the extraction as before, prefixing the command line with /opt/kythe/extractors/cxx_extractor --with_executable:
```
/opt/kythe/extractors/cxx_extractor --with_executable /cvmfs/lhcb.cern.ch/lib/lcg/releases/clang/8.0.0/x86_64-centos7/bin/clang++ --gcc-toolchain=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/8.2.0/x86_64-centos7 -DBOOST_FILESYSTEM_VERSION=3 -DBOOST_SPIRIT_USE_PHOENIX_V3 -DGAUDI_V20_COMPAT -DGaudiSvc_EXPORTS -DPACKAGE_NAME=\"GaudiSvc\" -DPACKAGE_VERSION=\"v33r0\" -DTBB_USE_GLIBCXX_VERSION=80000 -D_GNU_SOURCE -Df2cFortran -Dlinux -Dunix -IGaudiSvc -I../GaudiSvc -I../GaudiPluginService -I../GaudiKernel -Iinclude -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/clhep/2.4.1.2/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/cppgsl/2.0.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/tbb/2019_U7/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/ROOT/6.18.04/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/Boost/1.70.0/x86_64-centos7-clang8-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_96b/rangev3/0.5.0/x86_64-centos7-clang8-opt/include -march=x86-64 -msse4.2 -fmessage-length=0 -pipe -Wall -Wextra -Werror=return-type -pthread -pedantic -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -std=c++17 -fdiagnostics-color -O3 -DNDEBUG -fPIC -MD -MT GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -MF GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o.d -o GaudiSvc/CMakeFiles/GaudiSvc.dir/src/RndmGenSvc/HepRndmGenerators.cpp.o -c ../GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp
```


The cxx_indexer unfortunately fails with the same errors as before
```
cd /afs/cern.ch/work/p/pseyfert/ofun/Gaudi/build.x86_64-centos7-clang8-opt; /opt/kythe/indexers/cxx_indexer --ignore_unimplemented /tmp/doesexist/0d2900e5a936067e7453ed3af896a4e68eafa11d7f1933a7dd594ffe17d8345e.kzip > /tmp/entries 
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:84:54: error: no type named 'Factory' in 'HepRndm::Generator<Rndm::Bit>'
      template <typename T, typename F = typename T::Factory>
                                         ~~~~~~~~~~~~^~~~~~~
/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiPluginService/Gaudi/PluginServiceV2.h:85:14: note: in instantiation of default argument for 'DeclareFactory<HepRndm::Generator<Rndm::Bit> >' required here
      struct DeclareFactory {
             ^
...
14 errors generated.
Error: Errors during indexing.
```
($? is 1).

I figured out that much in the meanwhile that cxx_indexer is at least correct about the fact that HepRndm::Generator<T> doesn't have a ::Factory. Not a type, not an alias, not a member function, not a data member. So it seems clang++ doesn't need to instantiate the default argument for `DeclareFactory<HepRndm::Generator<Rndm::Bit> >` (just spelling out the error message here). Does cxx_indexer do something special about default template instantiation?

DeclareFactory is a templated struct
```
template <typename T, typename F = typename T::Factory>
      struct DeclareFactory{ …
```

While looking that up, I found out that the indexer errors come from the macro invocations [here](https://gitlab.cern.ch/gaudi/Gaudi/blob/master/GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp#L385). Commenting these out and rerunning the extractor leads to a happy, error-free indexer.

The preprocessor turns the macros into
```
typedef HepRndm::Generator<Bit> b1; namespace { ::Gaudi::PluginService::v2::DeclareFactory<b1, ObjFactory> _register_673{ ::Gaudi::PluginService::v2::Details::stringify_id( Bit::typeID() )}; }
```

So in my understanding the macro shouldn't lead to an instantiation of DeclareFactory with F=T::Factory (it should be F = ObjFactory, which is a using alias somewhere else in the includes).

Because that now hints towards template instantiation I looked at the --helpfull of cxx_indexer and flipped the switches -experimental_alias_template_instantiations and -index_template_instantiations to various combinations. This did not lead to disappearance of the errors.

Does that raise any ideas? Thanks again,

Paul

Shahms King

unread,
Feb 11, 2020, 12:44:37 PM2/11/20
to Paul Seyfert, Luke Zarko, Kythe
Luke may have a better idea.  

It's possible that we're instantiating templates for indexing which wouldn't be during compilation and that is the cause of the errors.  It's worth checking to see if there are substantial differences in the output (or even just the size) between the run with errors and the run without.  You may be able to use https://github.com/kythe/kythe/tree/master/kythe/go/storage/tools/triples to get a better sense of the changes between the two.  It's quite possible the indexing errors are benign, although it would be nice to inhibit logging them if that's the case.

--Shahms

--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/6c772b4e-9934-4f85-8e6a-dfa19b2f3343%40googlegroups.com.


--
:-P

Luke Zarko

unread,
Feb 11, 2020, 5:32:08 PM2/11/20
to Shahms King, Paul Seyfert, Kythe
These may be errors that come from the code that tries to generate "nice" signatures for templates, where we do end up calling back into the typechecker. The function that does this is `MarkedSourceGenerator::ReplaceMarkedSourceWithTemplateArgumentList` in `kythe/cxx/indexer/cxx/marked_source.cc`. If we're lucky we can do something there to ignore diagnostics.

Thanks
  Luke

Paul Seyfert

unread,
Feb 12, 2020, 1:46:42 PM2/12/20
to Kythe


Am Dienstag, 11. Februar 2020 18:44:37 UTC+1 schrieb Shahms King:
Luke may have a better idea.  

It's possible that we're instantiating templates for indexing which wouldn't be during compilation and that is the cause of the errors.  It's worth checking to see if there are substantial differences in the output (or even just the size) between the run with errors and the run without.

I would say the outputs differ substantially in size (490MB vs 570MB).
I don't yet have a clue of the output, but I guess it makes sense as the macros I commented out instantiate some templates. (w/o default parameters).

Either way, with the included http_server (the one you said may or may not work) I don't see any content:

2020-02-12-193436_508x162_scrot.png


You may be able to use https://github.com/kythe/kythe/tree/master/kythe/go/storage/tools/triples to get a better sense of the changes between the two.  It's quite possible the indexing errors are benign, although it would be nice to inhibit logging them if that's the case.

--Shahms

To unsubscribe from this group and stop receiving emails from it, send an email to ky...@googlegroups.com.


--
:-P

Shahms King

unread,
Feb 12, 2020, 2:11:13 PM2/12/20
to Paul Seyfert, Kythe
On Wed, Feb 12, 2020 at 10:46 AM Paul Seyfert <pseyfert.c...@gmail.com> wrote:


Am Dienstag, 11. Februar 2020 18:44:37 UTC+1 schrieb Shahms King:
Luke may have a better idea.  

It's possible that we're instantiating templates for indexing which wouldn't be during compilation and that is the cause of the errors.  It's worth checking to see if there are substantial differences in the output (or even just the size) between the run with errors and the run without.

I would say the outputs differ substantially in size (490MB vs 570MB).
I don't yet have a clue of the output, but I guess it makes sense as the macros I commented out instantiate some templates. (w/o default parameters).

Either way, with the included http_server (the one you said may or may not work) I don't see any content:

2020-02-12-193436_508x162_scrot.png


Thanks for the follow up.  If the included http_server fails to display anything in either case, it's unlikely that the indexer error is the problem (or at least not the only one).  Looking at the screenshot, though, I see some content (specifically, the Gaudi: root directory) listed.  Clicking on that should expand to the immediate subdirectories (if any) and so on.

Any other details you can provide in https://github.com/kythe/kythe/issues/4362 would also be helpful.

The expanded macro case *looks* like straightforward case of providing an explicit template argument where the default won't compile, which happens pretty frequently in almost any code base so I would not expect the error to be fatal.

--Shahms
 
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/02f6772f-ecab-4689-9664-7f9acd9727df%40googlegroups.com.


--
:-P

Paul Seyfert

unread,
Feb 12, 2020, 2:27:53 PM2/12/20
to Kythe


On Wed, Feb 12, 2020 at 10:46 AM Paul Seyfert <pseyfert....@gmail.com> wrote:


Am Dienstag, 11. Februar 2020 18:44:37 UTC+1 schrieb Shahms King:
Luke may have a better idea.  

It's possible that we're instantiating templates for indexing which wouldn't be during compilation and that is the cause of the errors.  It's worth checking to see if there are substantial differences in the output (or even just the size) between the run with errors and the run without.

I would say the outputs differ substantially in size (490MB vs 570MB).
I don't yet have a clue of the output, but I guess it makes sense as the macros I commented out instantiate some templates. (w/o default parameters).

Either way, with the included http_server (the one you said may or may not work) I don't see any content:

2020-02-12-193436_508x162_scrot.png


Thanks for the follow up.  If the included http_server fails to display anything in either case, it's unlikely that the indexer error is the problem (or at least not the only one).  Looking at the screenshot, though, I see some content (specifically, the Gaudi: root directory) listed.  Clicking on that should expand to the immediate subdirectories (if any) and so on.

sorry i was too brief there. i tried clicking that in chrome and firefox but nothing expands, as if the directory browser wants to tell me there's the directory but no content.

I do see on the shell running the http_server that the click gets registered by the server

2020/02/12 20:24:05 HTTP server listening on "localhost:8089"
2020/02/12 20:24:11 filetree.CorpusRoots: 43.818517ms
2020/02/12 20:24:13 filetree.Dir: 1.728409ms

Shahms King

unread,
Feb 12, 2020, 3:27:51 PM2/12/20
to Paul Seyfert, Kythe
On Wed, Feb 12, 2020 at 11:27 AM Paul Seyfert <pseyfert.c...@gmail.com> wrote:


On Wed, Feb 12, 2020 at 10:46 AM Paul Seyfert <pseyfert....@gmail.com> wrote:


Am Dienstag, 11. Februar 2020 18:44:37 UTC+1 schrieb Shahms King:
Luke may have a better idea.  

It's possible that we're instantiating templates for indexing which wouldn't be during compilation and that is the cause of the errors.  It's worth checking to see if there are substantial differences in the output (or even just the size) between the run with errors and the run without.

I would say the outputs differ substantially in size (490MB vs 570MB).
I don't yet have a clue of the output, but I guess it makes sense as the macros I commented out instantiate some templates. (w/o default parameters).

Either way, with the included http_server (the one you said may or may not work) I don't see any content:

2020-02-12-193436_508x162_scrot.png


Thanks for the follow up.  If the included http_server fails to display anything in either case, it's unlikely that the indexer error is the problem (or at least not the only one).  Looking at the screenshot, though, I see some content (specifically, the Gaudi: root directory) listed.  Clicking on that should expand to the immediate subdirectories (if any) and so on.

sorry i was too brief there. i tried clicking that in chrome and firefox but nothing expands, as if the directory browser wants to tell me there's the directory but no content.

I do see on the shell running the http_server that the click gets registered by the server

2020/02/12 20:24:05 HTTP server listening on "localhost:8089"
2020/02/12 20:24:11 filetree.CorpusRoots: 43.818517ms
2020/02/12 20:24:13 filetree.Dir: 1.728409ms

Ah, thanks. Pulling down and looking at the kzip you linked earlier it looks broadly okay, although there are a lot of fully-qualified paths in VNames, which can cause problems.  Sometimes they're fine, sometimes they aren't.  Similarly with the data produced when feeding that kzip file to the C++ indexer.

While it would be nice to clean up the error messages, I don't think that's the root of the problem.


--Shahms
 


Any other details you can provide in https://github.com/kythe/kythe/issues/4362 would also be helpful.

The expanded macro case *looks* like straightforward case of providing an explicit template argument where the default won't compile, which happens pretty frequently in almost any code base so I would not expect the error to be fatal.

--Shahms
 

You may be able to use https://github.com/kythe/kythe/tree/master/kythe/go/storage/tools/triples to get a better sense of the changes between the two.  It's quite possible the indexing errors are benign, although it would be nice to inhibit logging them if that's the case.

--Shahms

On Tue, Feb 11, 2020 at 5:12 AM Paul Seyfert <pseyfert....@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.

Shahms King

unread,
Feb 12, 2020, 7:10:49 PM2/12/20
to Paul Seyfert, Kythe
I don't know that the VName paths are the source of the problem, but I know that the graph generally expects the "path" to be relative to the repository root, but it can be pretty easy to end up with absolute paths and paths relative to the build directory instead which can cause issues (and looks to have happened here).

Can you set KYTHE_ROOT_DIRECTORY to the *build* root, rather than the source root and see if that helps?

--Shahms
--
:-P

Shahms King

unread,
Feb 12, 2020, 8:09:18 PM2/12/20
to Paul Seyfert, Kythe
It does appear to be the absolute paths causing issues.  I'm not sure if it's just http_server, but I suspect so.  If I remap the files in the kzip provided, I can get a rudimentary UI working:
image.png

--Shahms
--
:-P

Paul Seyfert

unread,
Feb 13, 2020, 10:41:01 AM2/13/20
to Kythe
:grinning_face_with_star_eyes: amazing, thanks for investigating!

i tried the following setups

 1) KYTHE_ROOT_DIRECTORY set to the source directory and building with ninja (original setup)
 2) KYTHE_ROOT_DIRECTORY set to the build directory and building with ninja
 3) KYTHE_ROOT_DIRECTORY set to the build directory and building with gnu make (iirc cmake creates compile_commands.json with -I relative paths for ninja and absolute paths with gnu make. changing this way doesn't match the needed fix you described but while i have another issue related to that on my desk i figured i just see what happens)

In either case I don't see a UI working as in your screenshot.

Looking at the start of root/units/<file> in the kzip I see
 1) with KYTHE_ROOT_DIRECTORY set to the source tree [{"v_name":{"corpus":"Gaudi","path":"GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp"}
 2) with KYTHE_ROOT_DIRECTORY set to the build tree [{"v_name":{"corpus":"Gaudi","path":"/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp"}

so that seems to go in the opposite of the desired direction.

Do I understand correctly you mean the absolute paths of my dependencies' headers (boost, standard library, …) are problematic? Because as far as I see by eye in the kzip (or with `tools/entrystream --write_format=json -unique`) the files from my source repository appear by relative (to the source top level dir) path.

Cheers,
Paul

Shahms King

unread,
Feb 13, 2020, 11:51:31 AM2/13/20
to Paul Seyfert, Kythe
On Thu, Feb 13, 2020 at 7:41 AM Paul Seyfert <pseyfert.c...@gmail.com> wrote:
:grinning_face_with_star_eyes: amazing, thanks for investigating!

i tried the following setups

 1) KYTHE_ROOT_DIRECTORY set to the source directory and building with ninja (original setup)
 2) KYTHE_ROOT_DIRECTORY set to the build directory and building with ninja
 3) KYTHE_ROOT_DIRECTORY set to the build directory and building with gnu make (iirc cmake creates compile_commands.json with -I relative paths for ninja and absolute paths with gnu make. changing this way doesn't match the needed fix you described but while i have another issue related to that on my desk i figured i just see what happens)

In either case I don't see a UI working as in your screenshot.

Looking at the start of root/units/<file> in the kzip I see
 1) with KYTHE_ROOT_DIRECTORY set to the source tree [{"v_name":{"corpus":"Gaudi","path":"GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp"}
 2) with KYTHE_ROOT_DIRECTORY set to the build tree [{"v_name":{"corpus":"Gaudi","path":"/afs/cern.ch/work/p/pseyfert/ofun/Gaudi/GaudiSvc/src/RndmGenSvc/HepRndmGenerators.cpp"}

so that seems to go in the opposite of the desired direction.

Do I understand correctly you mean the absolute paths of my dependencies' headers (boost, standard library, …) are problematic? Because as far as I see by eye in the kzip (or with `tools/entrystream --write_format=json -unique`) the files from my source repository appear by relative (to the source top level dir) path.

Yeah, that's what I meant.  I took a fairly aggressive approach and used `kzip merge` (with https://github.com/kythe/kythe/pull/4366) to remap all of the fully-qualified paths to a different root and removing the leading '/', which resulted in the results you see.  You can do something similar during extraction by using:

/// vnames.json
[
  {
    "pattern": "^/(.*)",
    "vname": {
      "root": "/",
      "path": "@1@"
    }
]

KYTHE_VNAMES=path/to/vnames.json 

--Shahms


Cheers,
Paul


Am Donnerstag, 13. Februar 2020 02:09:18 UTC+1 schrieb Shahms King:
It does appear to be the absolute paths causing issues.  I'm not sure if it's just http_server, but I suspect so.  If I remap the files in the kzip provided, I can get a rudimentary UI working:
image.png

--Shahms

On Wed, Feb 12, 2020 at 4:10 PM Shahms King <sha...@google.com> wrote:
I don't know that the VName paths are the source of the problem, but I know that the graph generally expects the "path" to be relative to the repository root, but it can be pretty easy to end up with absolute paths and paths relative to the build directory instead which can cause issues (and looks to have happened here).

Can you set KYTHE_ROOT_DIRECTORY to the *build* root, rather than the source root and see if that helps?


--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.

Paul Seyfert

unread,
Feb 18, 2020, 5:10:49 AM2/18/20
to Kythe
Hi,

TL;DR: all fixed. cxx_indexer errors are false alarms. got the http_server serving useful output.

Seems I had my kythe/web/ui broken but now things work (rebuilt it with `bazel build kythe/web/ui`, and then copied kythe/web/ui/resources/public and bazel-bin/kythe/web/ui/resources/public to the same place). From what I see in the browser so far it indeed looks like the indexer errors are false alarms.

for the record

when setting the KYTHE_VNAMES environment variable I get this kzip https://cernbox.cern.ch/index.php/s/CCz1ylUaIqoMbgc
for reference since I reran everything
with KYTHE_ROOT_DIRECTORY pointing to the source directory root w/o KYTHE_VNAMES I get that https://cernbox.cern.ch/index.php/s/ioKIKRw8zmMmtxK
and when using kzip merge on the latter I get this https://cernbox.cern.ch/index.php/s/t3Hu1mC0cN05s0L

In the version that I manipulated with kzip merge there seems to be something wrong (screenshot) but the version with KYTHE_VNAMES set during extraction looks good.

I ran `kzip merge --output root_transformed/test.kzip -rules vnames.json root_with_src/528caaad356838f5dbe4745154b1442df3e497ac65312296c3ee829e00105543.kzip` and had build kzip from kythe commit 59453746275a3e1840ac63cdb5a993e86de96cfc.

2020-02-18-105852_1514x257_scrot.png




Thanks again for your time!

Paul


Am Donnerstag, 13. Februar 2020 17:51:31 UTC+1 schrieb Shahms King:


To unsubscribe from this group and stop receiving emails from it, send an email to ky...@googlegroups.com.


--
:-P

Shahms King

unread,
Feb 19, 2020, 11:11:25 AM2/19/20
to Paul Seyfert, Kythe
I'm glad you got things working and thanks for following up!

--Shahms

To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/7c33afb1-e383-470e-9383-c8be73a6353e%40googlegroups.com.


--
:-P
Reply all
Reply to author
Forward
0 new messages