Re: Digest for kythe@googlegroups.com - 2 updates in 1 topic

18 views
Skip to first unread message

Natraj Kavander

unread,
Mar 17, 2024, 3:15:21 PM3/17/24
to ky...@googlegroups.com
Thank you 🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿

On Sun, Feb 18, 2024, 9:58 AM <ky...@googlegroups.com> wrote:
Eric Amorde <eric....@gmail.com>: Feb 17 02:41PM -0800

I finally narrowed down most of my issues to two causes:
 
1. I was attempting to index a framework target that isn't linked by
default instead of something like an app target. I think it was called out
somewhere in the Kythe docs that it needs to be run on a top-level target.
This was the source of the errors for built-in compiler types and system
frameworks like CoreFoundation / Foundation.
2. Confirmed VFS overlays are the issue of the module import failures.
 
I created an example repo that reproduces the
issue: https://github.com/amorde/kythe-ios-example
 
It relies on a personal fork of Kythe that includes the objc extractor in
the release artifacts and also removes the "-fmodules"
check https://github.com/kythe/kythe/compare/master...amorde:kythe:release-objc-bazel-extractor?expand=1
 
On Friday, February 16, 2024 at 11:36:55 AM UTC-8 Salvatore Guarnieri wrote:
 
Eric Amorde <eric....@gmail.com>: Feb 17 03:26PM -0800

Note that I still see the CoreFoundation / built-in issues and many other
errors when attempting to *index* the extracted compilations.
 
I added a note at the bottom of the README mentioning how to reproduce
those. I'm still stuck with many errors during indexing on my project but
have at least gotten passed some of the errors during compilation
extraction.
 
On Saturday, February 17, 2024 at 2:41:12 PM UTC-8 Eric Amorde wrote:
 
You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page.
To unsubscribe from this group and stop receiving emails from it send an email to kythe+un...@googlegroups.com.

Natraj Kavander

unread,
Mar 17, 2024, 3:17:28 PM3/17/24
to ky...@googlegroups.com
Thank you 🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿🙏🏿👍🏿

On Thu, Feb 15, 2024, 9:58 AM <ky...@googlegroups.com> wrote:
Salvatore Guarnieri <salgua...@google.com>: Feb 14 08:55AM -0800

Just a few thoughts (not much to add that wasn't already covered)
 
The objc extractor is slightly different than the cxx one in that it also
tries to handle the location of the ios sdk. The code for this isn't super
robust so I wouldn't be surprised if that caused problems.
 
The exact objc extractor that is in the repository isn't used by us. We
needed to make some minor tweaks to make it work in our environment. So I
am not surprised that it did not work immediately.
 
I think once you get past extraction though, the indexing should go much
smoother since that code is designed to be mostly agnostic of exactly how
the compilation unit was constructed.
 
What types of new issues are you running into? Are you getting any results
from indexing the codebase?
 
Eric Amorde <eric....@gmail.com>: Feb 14 09:48AM -0800

I am getting some results but am frequently hitting errors during indexing.
The errors are either:
 
 
- Unable to find a header
- Unable to find a module imported via `@import SomeModule;`
- Unable to find compiler built-ins and/or system frameworks like
`Foundation` / `CoreFoundation`
 
 
There are other errors that seem to not actually affect the indexing but
still get printed as an error. These are usually from running the clang
tooling with arguments that it doesn't recognize when running in "analysis"
mode via -fsyntax-only, such as -index-store-path and
 
I've done a bunch of debugging with custom builds of the indexer +
extractor to try to figure out where its going wrong. The exact arguments
passed to the clang tooling invocation work correctly when run outside of
Kythe. Some guesses as to what the issues might be:
 
 
- Mismatch of clang versions - our project currently builds using Xcode
15.1 which includes clang-1500.1.0.2.5. I'm not sure how a version mismatch
would lead to the errors I'm seeing, however.
- VFS overlays - when I run into errors importing modules they are from
our own code that uses VFS overlays to define frameworks and their headers.
The arguments are picked up correctly by the extractor and passed through
the indexer though.
- Execution directory mismatch. When compiling locally outside of Kythe
Bazel runs clang inside its execution root which is something like
`/tmp/some_id/execroot/repo_name`. I've noticed that some of our clang
arguments pass "." as include paths, and our VFS overlays have "../../"
directory traversals that appear to be relative to the execroot. I
attempted to correct for this by running the indexer in the same execroot
but that didn't seem to resolve anything.


> The exact objc extractor that is in the repository isn't used by us. We
needed to make some minor tweaks to make it work in our environment. So I
am not surprised that it did not work immediately.
 
Any plans to generalize the changes and merge them in? I suspect if you
needed something custom maybe others would too :) Happy to help however I
can, though I'm a Bazel and C++ novice. I've definitely got the local
development environment setup for Kythe so I'm familiar enough to try
tweaks on my own.
 
> Are you getting any results from indexing the codebase?
 
Yes - we get some data but its clearly missing a bunch of context. For
example, sometimes the anchor locations are all missing. Other times
certain types are left out, or only their declaration from the header is
indexed but not the implementation (or vice-versa). In all my tests I clean
the Bazel cache and turn off remote caching in an effort to correct any
issues from local state. The results are consistent but the issues I run in
to differ depending on what workarounds I try out.
On Wednesday, February 14, 2024 at 8:56:01 AM UTC-8 Salvatore Guarnieri
wrote:
 
Reply all
Reply to author
Forward
0 new messages