[ANN] The Go Oracle: a source code comprehension tool for Go programs

[Alan Donovan]

I'm pleased to announce the Go Oracle, a new tool built atop the static analysis libraries under construction in the go.tools repository (https://code.google.com/p/go/source/browse?repo=tools).

The oracle is designed to answer, quickly and precisely, a range of questions about your Go program, such as "where can this dynamic call dispatch to?" or "who sends values on this channel?".

The tool is still in a rough state, but already useful if you're the "early adopter" type.  (I use it all the time.)  The tool has only been tested on linux-amd64 and so far the only editor with which it is integrated is Emacs.  It requires a recent (=tip) version of the GOROOT libraries.

At this stage, the kind of feedback we're most interested in hearing is about the general shape of the tool: its external interfaces and how it should integrate with editors, IDEs and other tools and scripts.  All its interfaces are subject to change---now is your chance to influence that.  Reports of serious bugs are welcome too; send them directly to me.

User manual

https://docs.google.com/a/google.com/document/d/1GLiZOrj0ymGuADcMe6SxAyr7G3r97BzYTNtzS-Ve98k/edit#

Design document

https://docs.google.com/a/google.com/document/d/1kFsml0o_uIcFmj0VPprAe6u6LdZhwVWnxIVnFsRDehY/edit#heading=h.bxe8j4xoq6gk

cheers

alan

[noahca...@gmail.com]

Docs require permission to view.

[Alan Donovan]

I've learned that Google's site-specific Docs configuration won't let me share docs publicly, so I had to resort to posting PDFs (on golang-nuts).  I've reposted them here.

Sorry for the confusion.

[Brad Fitzpatrick]

Use a golang.org docs account.

On Sep 10, 2013 1:12 PM, "Alan Donovan" <adon...@google.com> wrote:

I've learned that Google's site-specific Docs configuration won't let me share docs publicly, so I had to resort to posting PDFs (on golang-nuts).  I've reposted them here.

Sorry for the confusion.

--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[u]

It seems you attached the design doc twice, but forgot the user doc.

[Alan Donovan]

> It seems you attached the design doc twice, but forgot the user doc.

This apparent shambles was in fact a carefully scripted plan to drive away casual users.

For the three of who've persevered this far, behold: the user manual.

[minux]

Thanks Alan, this is really an awesome tool.

I'd like to ask if there is plans to integrate it with godoc? it will make reading Go code on the browser

much easier than before.

--

[Alan Donovan]

On 10 September 2013 16:38, minux <minu...@gmail.com> wrote:

Thanks Alan, this is really an awesome tool.

I'd like to ask if there is plans to integrate it with godoc? it will make reading Go code on the browser

much easier than before.

Thanks.  Nothing concrete yet, but it's an obvious next step and I have some ideas for the UI.

[Alan Donovan]

[this time with working links]

I'm pleased to announce the Go Oracle, a new tool built atop the static analysis libraries under construction in the go.tools repository (https://code.google.com/p/go/source/browse?repo=tools).

The oracle is designed to answer, quickly and precisely, a range of questions about your Go program, such as "where can this dynamic call dispatch to?" or "who sends values on this channel?".

The tool is still in a rough state, but already useful if you're the "early adopter" type.  (I use it all the time.)  The tool has only been tested on linux-amd64 and so far the only editor with which it is integrated is Emacs.  It requires a recent (=tip) version of the GOROOT libraries.

At this stage, the kind of feedback we're most interested in hearing is about the general shape of the tool: its external interfaces and how it should integrate with editors, IDEs and other tools and scripts.  All its interfaces are subject to change---now is your chance to influence that.  Reports of serious bugs are welcome too; send them directly to me.

User manual

https://docs.google.com/a/golang.org/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/edit

Design document

https://docs.google.com/a/golang.org/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/edit#

cheers

[Dan Kortschak]

While reading through the design document I saw a pony. I'm not sure if
it's feasible or necessarily wanted, but thought I'd share.

When refactoring, finding equivalent structures in the code is sometimes
helpful (go fix e.g.). How easy would it be to include the capacity to
find AST subtree matches (structure and type only) to a selection in the
rest of the project?

Dan

[Daniel Morsing]

This is great. Did a bit of playing with it and took note of a couple of things:

- Vast majority of unsafe pointer conversion warnings happen in
"syscall". I'd be ok just whitelisting that package.
- Byte offset is a bit tricky to work around in acme. Using byte
offsets also makes it possible to specify a range that's inside a
UTF-8 sequence, but the tool seems to handle that fine. I don't know
how convenient switching to rune offsets would be, but I'd prefer
that.
- implements mode seems to be broken when called from the command
line. It needs a position, but when I give it a dummy one, it notifies
me that it cannot find it, even if the position exists.
- freevars reports each assignment to field selector. You could clean
up the output by just reporting the value.

Regards,
Daniel Morsing

[Alan Donovan]

On 11 September 2013 08:29, Daniel Morsing <daniel....@gmail.com> wrote:

On Wed, Sep 11, 2013 at 5:44 AM, Alan Donovan <adon...@google.com> wrote:
> [this time with working links]
>
> I'm pleased to announce the Go Oracle, a new tool built atop the static
> analysis libraries under construction in the go.tools repository
> (https://code.google.com/p/go/source/browse?repo=tools).
>
> The oracle is designed to answer, quickly and precisely, a range of
> questions about your Go program, such as "where can this dynamic call
> dispatch to?" or "who sends values on this channel?".
>
> The tool is still in a rough state, but already useful if you're the "early
> adopter" type.  (I use it all the time.)  The tool has only been tested on
> linux-amd64 and so far the only editor with which it is integrated is Emacs.
> It requires a recent (=tip) version of the GOROOT libraries.
>
> At this stage, the kind of feedback we're most interested in hearing is
> about the general shape of the tool: its external interfaces and how it
> should integrate with editors, IDEs and other tools and scripts.  All its
> interfaces are subject to change---now is your chance to influence that.
> Reports of serious bugs are welcome too; send them directly to me.
>
> User manual
> https://docs.google.com/a/golang.org/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/edit
>
> Design document
> https://docs.google.com/a/golang.org/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/edit#
>
> cheers
>

This is great. Did a bit of playing with it and took note of a couple of things:

Thanks for the feedback.

 

- Vast majority of unsafe pointer conversion warnings happen in
"syscall". I'd be ok just whitelisting that package.

Good idea.  https://codereview.appspot.com/13263050

 

- Byte offset is a bit tricky to work around in acme. Using byte
offsets also makes it possible to specify a range that's inside a
UTF-8 sequence, but the tool seems to handle that fine. I don't know
how convenient switching to rune offsets would be, but I'd prefer
that.

We could easily add that as an option.

Rob Pike had a 5-minute hack for oracle+acme.  Though I haven't seen his code, he didn't mention the issue as a problem, so perhaps Acme does provide hooks to access byte offsets?

 

- implements mode seems to be broken when called from the command
line. It needs a position, but when I give it a dummy one, it notifies
me that it cannot find it, even if the position exists.

Thanks; this was a regression I introduced yesterday.

Fix pending in https://codereview.appspot.com/13334050

 

- freevars reports each assignment to field selector. You could clean
up the output by just reporting the value.

Actually I went to extra trouble to implement the documented behaviour. :)

The motivation was that if some code references only x.y.a and x.y.b, you wouldn't extract it into a function that takes x as a parameter; you'd just pass x.y.a and x.y.b, or perhaps their common ancestor x.y.  If this is a nuisance we could disable it or make it an option.

I just sent a fix for another bug in freevars: it wasn't suppressing the printing of free identifiers defined in package scope, only those from file scope. https://codereview.appspot.com/13256050

cheers

alan

[Daniel Morsing]

I've already done my own little acme hack. Turns out to be not that
big of a problem.

>
>>
>> - implements mode seems to be broken when called from the command
>> line. It needs a position, but when I give it a dummy one, it notifies
>> me that it cannot find it, even if the position exists.
>
>
> Thanks; this was a regression I introduced yesterday.
> Fix pending in https://codereview.appspot.com/13334050
>
>
>>
>> - freevars reports each assignment to field selector. You could clean
>> up the output by just reporting the value.
>
>
> Actually I went to extra trouble to implement the documented behaviour. :)
> The motivation was that if some code references only x.y.a and x.y.b, you
> wouldn't extract it into a function that takes x as a parameter; you'd just
> pass x.y.a and x.y.b, or perhaps their common ancestor x.y. If this is a
> nuisance we could disable it or make it an option.
>

You're right. The example I used to test this tool was an
initialization procedure where I wrote to several fields on a single
value. In that case it makes sense to hide field accesses, but not so
much in all the others.

After playing with it some more, I think an option that supresses line
numbers for freevars would be useful. Then I could just do a copy of
all the variables to the function that I'm hoisting the statements
into.

[roger peppe]

On 11 September 2013 15:45, Alan Donovan <adon...@google.com> wrote:
> On 11 September 2013 08:29, Daniel Morsing <daniel....@gmail.com> wrote:
>> - Byte offset is a bit tricky to work around in acme. Using byte
>> offsets also makes it possible to specify a range that's inside a
>> UTF-8 sequence, but the tool seems to handle that fine. I don't know
>> how convenient switching to rune offsets would be, but I'd prefer
>> that.
>
>
> We could easily add that as an option.
>
> Rob Pike had a 5-minute hack for oracle+acme. Though I haven't seen his
> code, he didn't mention the issue as a problem, so perhaps Acme does provide
> hooks to access byte offsets?

It doesn't, but it's easy enough to work around. I also have a 5
minute hack for oracle+acme.

It looks like this:
#!/usr/local/plan9/bin/rc
addr=`{acmedot}
oracle -pos $addr $*

where acmedot prints out the address of the current window.
(http://play.golang.org/p/-KSoTt2Q2_- mostly stolen from godef)

I have also had a little play with oracle. It worked pretty well,
although it took a long time to run.

I'm really looking forward to being able to throw away godef at some
point in the future
and replace it with oracle. Things that should help make this possible:

- the ability to take some code from the editor's buffer rather than from disk.
Currently if the file on disk is different from the editor's version, the
byte offset is probably bogus. In godef I solved this by allowing
the current file to be read from stdin, but I wonder if a package
API to oracle could access all files through some kind of filesystem interface,
so a given editor could define an interface that made currently in-buffer
files available to oracle.

- "best effort" mode. It should be possible to find some information
on a program
even if it doesn't fully type check.

- the ability to find the definition of a symbol even without setting a scope;
the scope is only needed for more sophisticated analysis. This means it
should Perhaps
it might be worth having another mode, "define", which does only the
limited analysis necessary for finding a symbol definition.

- much faster operation - I've seen it take between 5s and 30s to run.
Godef took about 10ms on the same thing (admittedly with much smaller
scope)

- the ability to specify a symbol other than by file offset - for instance, if I
want to find the definition of a function or variable without first finding a
mention of that identifier in the source code, it would be nice to be able
to name it on the oracle command line.

I'll try to file specific issues around the actual oracle output as
they turn up.

This is a marvellous tool - thanks so much for working on it!

[Alan Donovan]

On 15 September 2013 16:15, <freder...@gmail.com> wrote:

On Wednesday, September 11, 2013 4:45:48 PM UTC+2, Alan Donovan wrote:

On 11 September 2013 08:29, Daniel Morsing <daniel....@gmail.com> wrote:

- Byte offset is a bit tricky to work around in acme. Using byte
offsets also makes it possible to specify a range that's inside a
UTF-8 sequence, but the tool seems to handle that fine. I don't know
how convenient switching to rune offsets would be, but I'd prefer
that.

We could easily add that as an option.

I wrote a little web front-end for the oracle:

go get github.com/fzipp/pythia

Unfortunately, the browser provides only rune offsets for selection ranges, no byte offsets. Adding rune offsets as an option would help me.

Awesome!  I have a few UI suggestions:

1. Have the pane of results persist even as you click on lines in it.  (Currently it disappears when you jump to another file.)

2. Highlight the line when you click on a results line (and the de-highight when you do another action).

3. Only show the menu options appropriate for the selected location.  (A little tricky; the ideal implementation requires me to break up the API into smaller chunks.)  e.g. "peers" would be greyed out unless you select a <- token, and freevars would only appear when the selection is nonempty (not on each click).

4. Specialize the call{ers,ees,graph} queries into the UI: fetch the complete callgraph at startup and cache it.  Use it to mark up the source so that all function declarations have a little "jump to caller(s)" button in the margin that performs the "callers" query (from the cached callgraph), and all call sites have a hyperlink on the '(' token that performs a "callees" query (also from the cache).  If the result is unique, go straight there without showing the results pane.

5. Move the "implements" and "callgraph" query buttons to the top of the screen (since they don't need a source selection).

How much time do you want to put into this project going forward?  I would love to develop this approach (perhaps integrated into godoc, not sure) but my powers of GUI are weak.

This is a really nice demo, and I'm impressed by how quickly you put it together.

cheers

alan

[freder...@gmail.com]

On Wednesday, September 11, 2013 4:45:48 PM UTC+2, Alan Donovan wrote:

On 11 September 2013 08:29, Daniel Morsing <daniel....@gmail.com> wrote:

- Byte offset is a bit tricky to work around in acme. Using byte
offsets also makes it possible to specify a range that's inside a
UTF-8 sequence, but the tool seems to handle that fine. I don't know
how convenient switching to rune offsets would be, but I'd prefer
that.

We could easily add that as an option.

[freder...@gmail.com]

On Sunday, September 15, 2013 10:53:47 PM UTC+2, Alan Donovan wrote:

Awesome!  I have a few UI suggestions

Thanks for the positive feedback. All your suggestions make perfect sense and I have added them to the project's issue list.

 

How much time do you want to put into this project going forward?  I would love to develop this approach (perhaps integrated into godoc, not sure) but my powers of GUI are weak.

I can surely spend a couple of hours a week on it.

 

This is a really nice demo, and I'm impressed by how quickly you put it together. 

Go and your oracle API made it really simple.

[Dan Kortschak]

On Mon, 2013-09-16 at 07:32 +0800, Liigo Zhuang wrote:
> The go oracle is a bad name , because there is a famous database named
> this

You do understand the basis for the naming?

http://en.wikipedia.org/wiki/Oracle

It is an excellent name.

[andrey mirtchovski]

> http://en.wikipedia.org/wiki/Oracle
>
> It is an excellent name.

i thought you were going to link to this one:

http://en.wikipedia.org/wiki/Oracle_v._Google

those googlers sure are cheeky :)

[Andrew Gerrand]

On 16 September 2013 10:20, andrey mirtchovski <mirtc...@gmail.com> wrote:

those googlers sure are cheeky :)

To be honest, I've known about this project of Alan's for months and the coincidence never occurred to me!

[andrey mirtchovski]

> To be honest, I've known about this project of Alan's for months and the
> coincidence never occurred to me!

i was only joking, not really suggesting the existence of an insidious
naming scheme. for myself i'm just happy to have 'oracle' be something
closer to its true meaning in my programming life (like that web
frontend!) than just another database :)

[rbere...@gmail.com]

Hi all, 

I have not seen this commented elsewhere (a quick skim of the user manual, this thread,) so just in case someone lands in the same error:

GOPATH/src/code.google.com/p/go.tools/importer/source.go:434: n.Lparen undefined (type *ast.TypeAssertExpr has no field or method Lparen)

GOPATH/src/code.google.com/p/go.tools/importer/source.go:435: n.Lparen undefined (type *ast.TypeAssertExpr has no field or method Lparen)

GOPATH/src/code.google.com/p/go.tools/importer/source.go:436: n.Rparen undefined (type *ast.TypeAssertExpr has no field or method Rparen)

(I've "obscured" my gopath.) So, the oracle only works with the tip version (1.1.2 TypeAssertExpr has no field Lparen, but tip does.) Now I have to decide if I want to tip or not to tip :)

Ruben

[rbere...@gmail.com]

Forget my previous message.

 It requires a recent (=tip) version of the GOROOT libraries.

How could I miss that? Right in the first post by Alan!

Ruben 

[Java Pro]

1. This tool is really cool and much needed!

2. Can we have an option to make the oracle output pane on the left or right rather than at the bottom? This way, 1) more vertical space to view the source code 2) less mouse movement to the bottom.

3. "oracle" is what this tool is. I guess a lot of non-native English speaking people first came to know this word by having heard the Oracle database when Oracle heavily marketed its products in the country. Therefore, oracle does stand for a database. Native English speaking people first knew this word when they were very young and had no clue of what a database is and just remembered the true meaning of this word. 

[freder...@gmail.com]

On Friday, October 18, 2013 5:14:10 PM UTC+2, Java Pro wrote:

2. Can we have an option to make the oracle output pane on the left or right rather than at the bottom?

 In case you are referring to the Pythia web interface: makes sense to me and I've created a ticket: https://github.com/fzipp/pythia/issues/16

[Java Pro]

A little bit syntax highlighting in Pythia for the go files (github or Sublime style) would make it a better standalone analyzer without an IDE. 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/CwdIJZs6Tfc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.