Bazel and Kythe

Yunzhao Xu

unread,

Mar 24, 2021, 11:21:12 PM3/24/21

to Kythe

Hi, I am new to use kythe and have some questions about using kythe with bazel.

1. Is there anyway to do the whole process from extraction to build the serving_table using bazel? So far I just noticed the extraction can use bazel, and the rest of jobs was done by script. This is fine, just want to see any better solution:)

2. I have a bazel repo with go and proto. And I tried to build the cross_reference for it. The command I used is like:

2.1 extraction

bazel --bazelrc=$KYTHE_DIR/extractors.bazelrc \ build --override_repository kythe_release=$KYTHE_DIR \ //...

2.2 indexing

find -L bazel-out -name '*.go.kzip' | sort > units

time cat units | parallel -t -L1 /opt/kythe/indexers/go_indexer | /opt/kythe/tools/dedup_stream >> entries

2.3 graphstore and serving table

/opt/kythe/tools/write_entries --graphstore leveldb:/tmp/gs < entries

/opt/kythe/tools/write_tables --graphstore /tmp/gs --out /tmp/ts

So far, everything can run, but all of the import were missing. errors like that.

Unable to resolve import path "github.com/golang/protobuf/jsonpb"

I have no idea what's going on, I tried a go project without bazel and it works fine. Is there any special concern when we using bazel for our system?

3. another question is is there any better way to index a repo with multiple language? Suppose we have C++, Java, Go, and proto. Is there a way to using one script to do all the thing?

Shahms King

unread,

Mar 25, 2021, 12:49:21 PM3/25/21

to Yunzhao Xu, Kythe, Justin Buchanan

Per #1, there isn't any way of doing this from a Bazel rule (if that's what you're asking). Bazel doesn't support dynamic dependencies so you can't have a hypothetical rule which extracts and indexes an entire repository, unfortunately. You can do it on a per-target basis, but that's primarily useful when writing tests. In order to operate on an entire repository, you have to wrap bazel in some kind of script.

Justin should have a better idea on #2 and #3.

--Shahms

--
You received this message because you are subscribed to the Google Groups "Kythe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kythe+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/95bd706c-7891-4964-9035-33fd262ea293n%40googlegroups.com.

--

:-P

Andrew Allen

unread,

Mar 25, 2021, 1:24:42 PM3/25/21

to Shahms King, Yunzhao Xu, Kythe, Justin Buchanan

You might also try the "localrun" script which I was supposed to put into the release package, but forgot to (sorry!)

https://github.com/kythe/kythe/blob/master/kythe/go/localrun/cmd/localrun/main.go

If you've got a version of the kythe webserver in /opt/kythe (or whatever directory you pass in), it'll even start that service for you.

/** ~Andrew Z Allen */

To view this discussion on the web visit https://groups.google.com/d/msgid/kythe/CANeWHAiQOVEkKkXMtdXXLCP4x2GLBKtTG0N%3DODCW%3DTNwM6nZ2w%40mail.gmail.com.

Justin Buchanan

unread,

Mar 25, 2021, 1:46:21 PM3/25/21

to Andrew Allen, Shahms King, Yunzhao Xu, Kythe

For #2, we do support extracting and indexing go code that is built with bazel. I'm not sure offhand what would cause the "unable to resolve import path" errors. Is this for an open source project that I could try to reproduce the issue on? Are you using the latest release version of kythe (0.0.49)?