Issues with Objective-C support

[Eric Amorde]

Hello! I'm new to Kythe and am exploring using it to index an iOS project composed of Objective-C & Swift. I'm starting with just the Objective-C code for now.

For the Objective-C code, I'm having a hard time getting things to index fully. Below are some issues I encountered:

• There seems to be no equivalent to the cxx_extractor binary that works with Objective-C. I got a portion of it to work by forwarding `--ObjC` flags through various parts of the process while using the cxx_extractor but I'm guessing this is not the intended workflow.
  
• The Bazel Objc extractor isn't built along with //kythe/release. I was able to add it myself manually. If the maintainers are open to it I'd be happy to put this change into a pull request.
  
• While trying to use the Bazel Objc extractor I saw that any file that requires objc module support via "-fmodules" is skipped. Almost any iOS project requires module support. Is there information on why modules aren't supported and what it would take to support it?
• VFS overlays are not taken into account when indexing extracted compilations and therefore header and module imports fail to resolve.

I assume that Kythe is used internally for iOS repositories, so I'm wondering if there's some workflow that isn't publicized here or perhaps some documentation that I missed.

Regards,

Eric Amorde

[Shahms King]

I'll try to answer inline as best I can, but the folks who best remember the details of Objective-C support are currently OOO for the holidays.

On Mon, Dec 18, 2023 at 1:55 PM Eric Amorde <eric....@gmail.com> wrote:

Hello! I'm new to Kythe and am exploring using it to index an iOS project composed of Objective-C & Swift. I'm starting with just the Objective-C code for now.

For the Objective-C code, I'm having a hard time getting things to index fully. Below are some issues I encountered:

• There seems to be no equivalent to the cxx_extractor binary that works with Objective-C. I got a portion of it to work by forwarding `--ObjC` flags through various parts of the process while using the cxx_extractor but I'm guessing this is not the intended workflow.
  

The C++ and Objective-C/C++ extractor are identical. 

• The Bazel Objc extractor isn't built along with //kythe/release. I was able to add it myself manually. If the maintainers are open to it I'd be happy to put this change into a pull request.
  
• While trying to use the Bazel Objc extractor I saw that any file that requires objc module support via "-fmodules" is skipped. Almost any iOS project requires module support. Is there information on why modules aren't supported and what it would take to support it?

Modules aren't supported because Kythe works by "replaying" compilations and the binary artifacts emitted by the compiler are tightly coupled to the exact version of that compiler. As Kythe generally requires a version of Clang which is close-ish to HEAD, the version used to build a given project is very rarely compatible with the version against which Kythe has been compiled.

 

• VFS overlays are not taken into account when indexing extracted compilations and therefore header and module imports fail to resolve.

I'm curious about this. CompilationUnits should be self-contained and completely describe the filesystem needed to build a particular compilation and as such manually construct the filesystem as seen by Clang during analysis.  Can you go into more detail about how overlays are used?

--Shahms

 

I assume that Kythe is used internally for iOS repositories, so I'm wondering if there's some workflow that isn't publicized here or perhaps some documentation that I missed.

Regards,

Eric Amorde

--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/7e576bd7-11cf-4240-8867-5b53804dbd32n%40googlegroups.com.

--

:-P

[Eric Amorde]

> The C++ and Objective-C/C++ extractor are identical. 

I can't recall exactly what issues I ran into here but it might be related to the binary hard-coding the cpp option here https://github.com/kythe/kythe/blob/7881f8fb2d7ddba79f9291c5e4b0a328f9a1a433/kythe/cxx/extractor/cxx_extractor_main.cc#L50

I know the data ends up being recorded as cxx but the Bazel Objective-C extractor provides the objc value. 

> As Kythe generally requires a version of Clang which is close-ish to HEAD, the version used to build a given project is very rarely compatible with the version against which Kythe has been compiled.

Yes I did realize this and I suspect this is causing issues in my case. We are using the clang version included with Xcode which is much older than what Kythe expects. Does this mean that it's effectively not possible to run the Kythe extractors on an iOS codebase?

Re: VFS overlays - I don't yet understand why it doesn't work, but manually running the exact clang invocation provided in the compilation_commands.json works but when the cxx_extractor runs the same command (I also added some debugging prints to get the final arguments that Kythe provides to the clang tooling) it fails as if the VFS is not included.

[Shahms King]

On Mon, Dec 18, 2023 at 2:35 PM Eric Amorde <eric....@gmail.com> wrote:

> The C++ and Objective-C/C++ extractor are identical. 

I can't recall exactly what issues I ran into here but it might be related to the binary hard-coding the cpp option here https://github.com/kythe/kythe/blob/7881f8fb2d7ddba79f9291c5e4b0a328f9a1a433/kythe/cxx/extractor/cxx_extractor_main.cc#L50

I know the data ends up being recorded as cxx but the Bazel Objective-C extractor provides the objc value. 

> As Kythe generally requires a version of Clang which is close-ish to HEAD, the version used to build a given project is very rarely compatible with the version against which Kythe has been compiled.

Yes I did realize this and I suspect this is causing issues in my case. We are using the clang version included with Xcode which is much older than what Kythe expects. Does this mean that it's effectively not possible to run the Kythe extractors on an iOS codebase?

AFAIK, Kythe is incompatible with an iOS codebase which requires the use of modules, yes.  I know we have some support for that internally and it uses the same indexer/extractor we use externally but am not super familiar with how it works.

 

Re: VFS overlays - I don't yet understand why it doesn't work, but manually running the exact clang invocation provided in the compilation_commands.json works but when the cxx_extractor runs the same command (I also added some debugging prints to get the final arguments that Kythe provides to the clang tooling) it fails as if the VFS is not included.

Ah, given the, uh, convoluted path handling in the extractor I'm not entirely surprised this doesn't work and we definitely don't have any tests of it.  It's certainly something it seems reasonable to support, but I don't know if we can prioritize it much.  I've created https://github.com/kythe/kythe/issues/5979 and if you're able it would be great to get more relevant details of the compiler commands involved.

--Shahms

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/8c12a0c4-0098-43be-a146-7e09bbd0b5b4n%40googlegroups.com.

--

:-P

[Eric Amorde]

I did try commenting out the logic for skipping compilations with "-fmodules" and was able to extract a decent amount of information from some internal Objective-C code. So it seems to at least somewhat work :)

[Eric Amorde]

Curious if any other folks who were on vacation would be willing to share some context here.

I'm still slowly working through some of the issues I'm having attempting to index an iOS codebase (just the objc parts) and might have identified some of the causes but I don't have a simple way to produce an example project that would have the same issue.

[Salvatore Guarnieri]

Just a few thoughts (not much to add that wasn't already covered)

The objc extractor is slightly different than the cxx one in that it also tries to handle the location of the ios sdk. The code for this isn't super robust so I wouldn't be surprised if that caused problems.

The exact objc extractor that is in the repository isn't used by us. We needed to make some minor tweaks to make it work in our environment. So I am not surprised that it did not work immediately.

I think once you get past extraction though, the indexing should go much smoother since that code is designed to be mostly agnostic of exactly how the compilation unit was constructed.

What types of new issues are you running into? Are you getting any results from indexing the codebase?

To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/bf651cab-a160-4428-8270-b3ef7bd65f8fn%40googlegroups.com.

[Eric Amorde]

I am getting some results but am frequently hitting errors during indexing. The errors are either:

• Unable to find a header
• Unable to find a module imported via `@import SomeModule;`
• Unable to find compiler built-ins and/or system frameworks like `Foundation` / `CoreFoundation`

There are other errors that seem to not actually affect the indexing but still get printed as an error. These are usually from running the clang tooling with arguments that it doesn't recognize when running in "analysis" mode via -fsyntax-only, such as -index-store-path and

I've done a bunch of debugging with custom builds of the indexer + extractor to try to figure out where its going wrong. The exact arguments passed to the clang tooling invocation work correctly when run outside of Kythe. Some guesses as to what the issues might be:

• Mismatch of clang versions - our project currently builds using Xcode 15.1 which includes clang-1500.1.0.2.5. I'm not sure how a version mismatch would lead to the errors I'm seeing, however.
• VFS overlays - when I run into errors importing modules they are from our own code that uses VFS overlays to define frameworks and their headers. The arguments are picked up correctly by the extractor and passed through the indexer though.
• Execution directory mismatch. When compiling locally outside of Kythe Bazel runs clang inside its execution root which is something like `/tmp/some_id/execroot/repo_name`. I've noticed that some of our clang arguments pass "." as include paths, and our VFS overlays have "../../" directory traversals that appear to be relative to the execroot. I attempted to correct for this by running the indexer in the same execroot but that didn't seem to resolve anything.
  
  

> The exact objc extractor that is in the repository isn't used by us. We needed to make some minor tweaks to make it work in our environment. So I am not surprised that it did not work immediately.

Any plans to generalize the changes and merge them in? I suspect if you needed something custom maybe others would too :) Happy to help however I can, though I'm a Bazel and C++ novice. I've definitely got the local development environment setup for Kythe so I'm familiar enough to try tweaks on my own.

> Are you getting any results from indexing the codebase?

Yes - we get some data but its clearly missing a bunch of context. For example, sometimes the anchor locations are all missing. Other times certain types are left out, or only their declaration from the header is indexed but not the implementation (or vice-versa). In all my tests I clean the Bazel cache and turn off remote caching in an effort to correct any issues from local state. The results are consistent but the issues I run in to differ depending on what workarounds I try out.

[Salvatore Guarnieri]

Have you unzipped the kzip to look at what it contains for a given compilation unit? While the other problems seem more complex, it should be straightforward to see if a header is present in the required inputs for the compilation unit and then to see if the contents are also present in the kzip. 

On Wed, Feb 14, 2024 at 9:48 AM Eric Amorde <eric....@gmail.com> wrote:

I am getting some results but am frequently hitting errors during indexing. The errors are either:

• Unable to find a header
• Unable to find a module imported via `@import SomeModule;`
• Unable to find compiler built-ins and/or system frameworks like `Foundation` / `CoreFoundation`

There are other errors that seem to not actually affect the indexing but still get printed as an error. These are usually from running the clang tooling with arguments that it doesn't recognize when running in "analysis" mode via -fsyntax-only, such as -index-store-path and

I've done a bunch of debugging with custom builds of the indexer + extractor to try to figure out where its going wrong. The exact arguments passed to the clang tooling invocation work correctly when run outside of Kythe. Some guesses as to what the issues might be:

• Mismatch of clang versions - our project currently builds using Xcode 15.1 which includes clang-1500.1.0.2.5. I'm not sure how a version mismatch would lead to the errors I'm seeing, however.
• VFS overlays - when I run into errors importing modules they are from our own code that uses VFS overlays to define frameworks and their headers. The arguments are picked up correctly by the extractor and passed through the indexer though.
• Execution directory mismatch. When compiling locally outside of Kythe Bazel runs clang inside its execution root which is something like `/tmp/some_id/execroot/repo_name`. I've noticed that some of our clang arguments pass "." as include paths, and our VFS overlays have "../../" directory traversals that appear to be relative to the execroot. I attempted to correct for this by running the indexer in the same execroot but that didn't seem to resolve anything.

* It shouldn't matter where you run the indexer since the indexer *shouldn't* access anything outside of the compilation unit generated during extraction. 

• 
  

> The exact objc extractor that is in the repository isn't used by us. We needed to make some minor tweaks to make it work in our environment. So I am not surprised that it did not work immediately.

Any plans to generalize the changes and merge them in? I suspect if you needed something custom maybe others would too :) Happy to help however I can, though I'm a Bazel and C++ novice. I've definitely got the local development environment setup for Kythe so I'm familiar enough to try tweaks on my own.

No, the changes are very specific to how we build objective c code (e.g. odd SDK path) so there isn't much we could generalize.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/bc3c9ad5-adfb-47fa-a5af-440e5382ea0bn%40googlegroups.com.

[Eric Amorde]

I finally narrowed down most of my issues to two causes:

1. I was attempting to index a framework target that isn't linked by default instead of something like an app target. I think it was called out somewhere in the Kythe docs that it needs to be run on a top-level target. This was the source of the errors for built-in compiler types and system frameworks like CoreFoundation / Foundation.

2. Confirmed VFS overlays are the issue of the module import failures.

I created an example repo that reproduces the issue: https://github.com/amorde/kythe-ios-example

It relies on a personal fork of Kythe that includes the objc extractor in the release artifacts and also removes the "-fmodules" check https://github.com/kythe/kythe/compare/master...amorde:kythe:release-objc-bazel-extractor?expand=1

[Eric Amorde]

Note that I still see the CoreFoundation / built-in issues and many other errors when attempting to index the extracted compilations.

I added a note at the bottom of the README mentioning how to reproduce those. I'm still stuck with many errors during indexing on my project but have at least gotten passed some of the errors during compilation extraction.