Starting a NetBeans-based IDE
Sam Harwell
- Functional editor with syntax and semantic-aware highlighting for single files (analysis extends only to the file currently being edited)
- Initial release for public review
- Features very likely to be completely missing: multi-file analysis, project/build system, etc.
- Basic project system to keep track of folder/file structure on disk
- Multi-file analysis for all files in the current project
- Context-sensitive code completion (this not your average completion support)
- Editing features very likely to be completely missing: refactoring, code templates, etc.
- Other features still very likely to be missing: build system support, debugging, etc.
rh
Sam Harwell
Pavel Korotkov
create a project: no entity appeared in the dialog box.
On Jan 16, 3:12 am, Sam Harwell <samharw...@gmail.com> wrote:
> Hello,
>
> First I'd like to introduce myself and give a bit of background. I'm Sam
> Harwell, and I'm a Ph. D. student at the University of Texas at Austin
> studying several aspects of the ways developers interact with IDEs to boost
> productivity. Currently I'm leading development of ANTLRWorks 2 (IDE) in
> parallel with Dr. Terence Parr's work on ANTLR 4. ANTLRWorks 2 supports
> development of ANTLR grammars and StringTemplate templates. Due to the
> simplicity of these languages, I've been looking for an additional general
> purpose language to work with so I can exercise some of the more advanced
> features of the framework powering ANTLRWorks 2. After a great deal of
> consideration, I've tentatively chosen Go as the language to bring into the
> fold.
>
> On Friday (1/13) at 4:30 I forked the ANTLRWorks 2 project to make the
> project "GoWorks". Somewhere between running errands and watching Mission
> Impossible, I built a fairly functional editor from the Go Language
> Specification. I give these times to point out the flexibility of ANTLR 4
> and the ANTLRWorks 2 framework in building both a parser and editor for a
> new language. For today I'll give just a screenshot (tease!), along with
> what I'm hoping to accomplish in the short run.
>
> Goal 0: Within the next week
>
> - Features very likely to be completely missing: multi-file analysis,
>
> Goal 1: Within the next month
>
> - Multi-file analysis for all files in the current project
> - Context-sensitive code completion (this not your average completion
> support)
> - Editing features very likely to be completely missing: refactoring,
> code templates, etc.
> - Other features still very likely to be missing: build system support,
>
> Along the way, community feedback will be immensely appreciated.
>
> Thank you,
> Sam Harwell
>
> 137KViewDownload
Sam Harwell
Tom Deakin
Sam Harwell
- Implemented a trivial project system (enough to just show the folder structure in the Projects pane)
- Vastly improved the semantic highlighter and implemented many of the internal data structures required for code completion and other IDE features.
In the previous screenshot I was careful to not have any type references shown to non-builtin types because the resolution functionality was not at all implemented (they would all show as just black). As you can see in today's screenshot, I have resolution working for most types in the current package.
Mikhail Strebkov
Brad Fitzpatrick
Popog
On Jan 18, 8:50 pm, Sam Harwell <samharw...@gmail.com> wrote:
> We're a bit more than halfway through the week now and I wanted to give a
> progress update and another screenshot. Since last time, I have:
>
> features.
>
> In the previous screenshot I was careful to not have any type references
> shown to non-builtin types because the resolution functionality was not at
> all implemented (they would all show as just black). As you can see in
> today's screenshot, I have resolution working for most types in the current
> package.
> For debugging during this early stage, I've enabled a special mode in the
> semantic highlighter that underlines all unresolved items so it's easy to
> see how my progress is coming along. After cross-package type resolution is
> working, I'll add support for constants, then vars, then methods. For this
> week's release, I expect much of the semantic highlighting to be working
> but code completion probably not in place until next week.
>
> 121KViewDownload
Sam Harwell
Sam Harwell
Mikhail Strebkov
Sam Harwell
- Trivial project system is in place - place an empty folder named "nbgoproject" in the folder where your Go package sources are located, and GoWorks will let you open the folder of packages as a project. No build/debug support is in place. For the Go source repository, the path to the empty folder would be "go/src/pkg/nbgoproject".
- Full-project background scanning is in place.
- Context-sensitive code completion is in place for some but definitely not all usages. In the places where it works, it should definitely be fast enough to use.
- I haven't updated the semantic syntax highlighter with the new analysis that's available to code completion, so many things that work for code completion will show as plain identifiers (black) in the highlighter.
Sam Harwell
Kyle Lemons
Sam Harwell
Hi Kyle,
The work I do is more complicated than people realize. Although I've been working on IDEs for various languages for many years, I've only had my current (highly experimental) algorithm operational for a couple of months. Also, the only general purpose programming language for which I've implemented automatic code completion on a broad scale is UnrealScript, which not only lacks a formal specification but tends to mutate from month to month and vary across the various companies who use it. For these reasons, I'll focus on how building an IDE for Go differs from building other analysis tools for Go.
Surprisingly, you may find that once implemented in a high-speed IDE, some features in Go designed for clarity/simplicity will actually reduce the ability of some developers to quickly write code. The primary case-in-point is the short var declaration vs. an assignment statement. If you compare the following two expressions:
someVar := 3
someVar = 3
You see that the `someVar` in the first statement is a variable definition, where the `someVar` in the second statement is a variable reference. For most machine processing applications, the complete text of the statement is available so it's immediately clear whether `someVar` should be treated as a definition or reference. However, in an IDE you might have simply the following, with a | representing the caret position:
s|
In this IDE, typing the `s` triggers an implicit code completion operation (as opposed to a Ctrl+Space triggering an explicit code completion operation). Without the context available to establish whether the user is typing a new definition or referencing something previously defined, the implicit code completion in the IDE must assume the user is typing a definition or risk inadvertently replacing the (valid) identifier of a new definition with that of a reference to a previously defined item. The net result is the short var declaration syntax places a fixed bound on the ability of a code completion algorithm to provide assistance on the first word of a new statement. Given the current nonexistence of IDEs capable of exposing this bound, it’s truly hard to explain how or why it would ever cause a problem. The current attempts to implement implicit code completion of identifiers have resulted in extremely frustrating behavior for a great number of reasons. However, this IDE for Go is designed for extremely predictable behavior and users will quickly figure out how to take advantage of its features without being slowed down by language limitations like this.
I have a few more tweaks to make related to the code completion engine before releasing a new preview within the next week. Hopefully at that point it will be much more apparent what I mean by all of this. :)
Thanks,
--
Sam Harwell
Owner, Lead Developer
Kyle Lemons
The work I do is more complicated than people realize. Although I've been working on IDEs for various languages for many years, I've only had my current (highly experimental) algorithm operational for a couple of months. Also, the only general purpose programming language for which I've implemented automatic code completion on a broad scale is UnrealScript, which not only lacks a formal specification but tends to mutate from month to month and vary across the various companies who use it. For these reasons, I'll focus on how building an IDE for Go differs from building other analysis tools for Go.
Surprisingly, you may find that once implemented in a high-speed IDE, some features in Go designed for clarity/simplicity will actually reduce the ability of some developers to quickly write code. The primary case-in-point is the short var declaration vs. an assignment statement. If you compare the following two expressions:
someVar := 3
someVar = 3
You see that the `someVar` in the first statement is a variable definition, where the `someVar` in the second statement is a variable reference. For most machine processing applications, the complete text of the statement is available so it's immediately clear whether `someVar` should be treated as a definition or reference. However, in an IDE you might have simply the following, with a | representing the caret position:
s|
In this IDE, typing the `s` triggers an implicit code completion operation (as opposed to a Ctrl+Space triggering an explicit code completion operation). Without the context available to establish whether the user is typing a new definition or referencing something previously defined, the implicit code completion in the IDE must assume the user is typing a definition or risk inadvertently replacing the (valid) identifier of a new definition with that of a reference to a previously defined item. The net result is the short var declaration syntax places a fixed bound on the ability of a code completion algorithm to provide assistance on the first word of a new statement. Given the current nonexistence of IDEs capable of exposing this bound, it’s truly hard to explain how or why it would ever cause a problem. The current attempts to implement implicit code completion of identifiers have resulted in extremely frustrating behavior for a great number of reasons. However, this IDE for Go is designed for extremely predictable behavior and users will quickly figure out how to take advantage of its features without being slowed down by language limitations like this.
DisposaBoy
⚛
> Surprisingly, you may find that once implemented in a high-speed IDE, some
> features in Go designed for clarity/simplicity will actually reduce the
> ability of some developers to quickly write code. The primary case-in-point
> is the short var declaration vs. an assignment statement. If you compare the
> following two expressions:
>
> someVar := 3
>
> someVar = 3
>
> You see that the `someVar` in the first statement is a variable definition,
> where the `someVar` in the second statement is a variable reference. For
> most machine processing applications, the complete text of the statement is
> available so it's immediately clear whether `someVar` should be treated as a
> definition or reference. However, in an IDE you might have simply the
> following, with a | representing the caret position:
>
> s|
impossible to recognize programmer's intention.
An example of a situation in which it is possible to identify
programmer's intention:
fmt.Fprintf(os.err, |
A code completion algorithm can *easily* (in the sense that the
algorithm's prediction can be very accurate) convert the above line
into:
fmt.Fprintf(os.Stderr, |
This is much easier to predict than to predict the intended meaning of
s|. In fact, s| is a prime counter-example to code-completion. In my
opinion, a code-completion algorithm should simply stop at cases like
s|.
> In this IDE, typing the `s` triggers an implicit code completion operation
> (as opposed to a Ctrl+Space triggering an explicit code completion
> operation). Without the context available to establish whether the user is
> typing a new definition or referencing something previously defined, the
> implicit code completion in the IDE must assume the user is typing a
> definition or risk inadvertently replacing the (valid) identifier of a new
> definition with that of a reference to a previously defined item. The net
> result is the short var declaration syntax places a fixed bound on the
> ability of a code completion algorithm to provide assistance on the first
> word of a new statement.
as the importance of some other Go language features. Short var
declaration doesn't seem to be special in any way.
> Given the current nonexistence of IDEs capable of
> exposing this bound, it's truly hard to explain how or why it would ever
> cause a problem. The current attempts to implement implicit code completion
> of identifiers have resulted in extremely frustrating behavior for a great
> number of reasons. However, this IDE for Go is designed for extremely
> predictable behavior and users will quickly figure out how to take advantage
> of its features without being slowed down by language limitations like this.
provided by the IDE?
Sam Harwell
Regarding the Fprintf example, it would perform as you mentioned for the reasons described in Part 6 of my latest blog post on the subject [1] (in particular point #6). In fact, it would very quickly learn to always replace "err" with "Stderr" fast enough that you never actually have to type "Stderr" again unless you're working within the os package itself - and you wouldn't have to press Ctrl+Space or slow down your typing for it to take effect. Some of the more advanced scenarios mentioned by Kyle involve a modification of points #8 and 9, similar to my commentary in Part 6.1. When my selection algorithms are used in combination with an even more advanced semantic analysis algorithm, perhaps similar to the one for sorting in [2], all kinds of options open up.
As I mentioned in my previous email, it's extremely difficult to understand or agree that such active behavior of code completion could be beneficial as opposed to extremely irritating. Hopefully people are open to trying new concepts and patient as I work to resolve issues that will surely surface. :)
[1] http://blog.280z28.org/archives/2011/12/105/
[2] http://www.monperrus.net/martin/Learning-from-Examples-to-Improve-Code-Completion-Systems.pdf
--
Sam Harwell
Owner, Lead Developer
⚛
> I don't currently have a list of features provided in this IDE since it's so new (literally less than a month old in its entirety).
>
> Regarding the Fprintf example, it would perform as you mentioned for the reasons described in Part 6 of my latest blog post on the subject [1] (in particular point #6). In fact, it would very quickly learn to always replace "err" with "Stderr" fast enough that you never actually have to type "Stderr" again unless you're working within the os package itself - and you wouldn't have to press Ctrl+Space or slow down your typing for it to take effect.
The idea in my example was based on the assumption that each keypress
is sending some bits of information to the IDE.
Typing "os.err," (including the comma) in the context of the 1st
argument of fmt.Fprintf should be enough data to conclude that the
programmer means "os.Stderr,". There is no learning phase, the
programmer does not need to explicitly teach the IDE anything.
Just "err" isn't enough information to map it to "os.Stderr" without
learning. The user needs to type "os.err," or something similar.
Another example:
var w io.Writer
var r io.Reader
copy(w, r)| --> io.Copy(w, r)|
> Some of the more advanced scenarios mentioned by Kyle involve a modification of points #8 and 9, similar to my commentary in Part 6.1. When my selection algorithms are used in combination with an even more advanced semantic analysis algorithm, perhaps similar to the one for sorting in [2], all kinds of options open up.
>
> As I mentioned in my previous email, it's extremely difficult to understand or agree that such active behavior of code completion could be beneficial as opposed to extremely irritating. Hopefully people are open to trying new concepts and patient as I work to resolve issues that will surely surface. :)
>
> [1]http://blog.280z28.org/archives/2011/12/105/
Sam Harwell
yone098
--
Masaaki YONEBAYASHI
@yone098
Sam Harwell
I finished enough to where it should be somewhat usable. :) Referring back to my “Goal 1: Within the next month” message on Jan. 15, here is a summary of the progress:
· Basic project system (shows file/folder structure in the project pane) is in place.
· Multi-file background analysis for all files in the current project is working.
· Context-sensitive code completion is functional but still has some bugs I’m working out.
· Mark Occurrences is a bonus feature I didn’t originally anticipate having in. :)
· Other items which I said would be missing are still missing. A notable missing feature is Go To Definition, which I plan to implement soon.
Here are some additional notes about code completion in its current state:
· It works in many but not all cases. The missing cases are not always “edge” cases, so there’s a high chance of finding situations where it doesn’t behave as you’d like it to.
· It doesn’t support automatic completion of arguments to function calls.
· It doesn’t track advanced semantics such as suggesting items for the right hand side of an assignment statement that match the type of the item on the left hand side.
· In cross-package references, it doesn’t hide package-private members from the completion popup.
· Background scanning isn’t currently hooked up to the progress bar in NetBeans’ status bar. Most code completion features will work properly after the background scan is complete, and you can watch the CPU usage with a task monitor on your system to know when the background scan is complete.
Here are some additional notes about performance. For reference, the configuration file is located in goworks/etc/goworks.conf.
· If you have a system with enough memory (4G free memory for running NetBeans), I recommend using a 64-bit, 1.7 JVM and adding the following configuration options: “-Xms2g -Xmx4g -XX:+UseG1GC”.
· For now, I recommend having at least 1G free memory when running GoWorks.
· The code completion features currently perform more computation than is ideal, which makes it observably “laggy” on slower computers. On current generation machines (esp. those with Core i5, Core i7 or similar processors), the code completion should perform close to the speed at which you type.
· GoWorks is using a very early build of ANTLR 4 which tends to use a lot of memory. Even though I’ve reduced its demands by nearly 10:1 over the reference implementation and significantly improved its performance, I definitely have a ways left to go. Already on the drawing board are changes that will provide an order of magnitude improvement in many performance categories, and drop an O(2^n) worst-case algorithm in code completion to an O(n³) worst-case algorithm (n tends to be small, but the impact is certainly measurable).
Here are some final notes about me and this project:
· This IDE is a case study in rapid development of an IDE for a general-purpose language. While other platforms for fast/easy integration of new languages (e.g. Xtext) focus on DSLs, I’m interested in scalability of IDE techniques to general purpose languages on large projects. Hopefully I surprised more than a couple people by actually delivering some of these features in the 1-month time span I originally stated. :)
· I’ve never actually written a Go program. I chose Go for this project due to its active development, current lack of other IDE options, and especially the care taken to maintain a proper language specification I can work with.
· In order to prove the true viability of my techniques, over the next couple of months I’m definitely going to polish the current set of features along with a sampling of a spectrum of other features (a couple navigation tools like Go To Definition and Find References, a couple refactoring operations like Rename and Extract Method, etc.). Once these are complete, I’m not sure how much I’ll be able to devote to additional features because they aren’t necessary for my research and the development costs are non-trivial. That said, I’ve really been enjoying this project so I’m hoping to find a way to keep working on it to make it (and Go) competitive in the “big leagues” of IDEs.
The latest “early access/pre alpha” release can be downloaded here:
GoWorks Early Access Preview 3: http://www.tunnelvisionlabs.com/downloads/go/2012-02-08-goworks-ea3.zip
GoWorks EULA (freeware/closed source): http://www.tunnelvisionlabs.com/downloads/go/GoWorksEula.pdf
Thank you,
--
Sam Harwell
Owner, Lead Developer
From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On Behalf Of Sam Harwell
Sent: Tuesday, February 07, 2012 10:33 PM
To: golan...@googlegroups.com
Subject: [go-nuts] Re: Starting a NetBeans-based IDE
Been working hard to iron out remaining bugs. There are definitely a number of prominent bugs remaining, but things seem to be working well enough that I should be able to release a new preview tomorrow. One of the recent additions is Mark Occurrences - currently it only works for some items but soon it should be accurate for whatever's under the caret.
⚛
> GoWorks Early Access Preview 3:http://www.tunnelvisionlabs.com/downloads/go/2012-02-08-goworks-ea3.zip
for i := len(args)-1; i>=0; i-- {
println(1)
}
Sam Harwell
Here's the update that should resolve this. You may get some flickering on occasion which I need to resolve, but at least it won't keep going forever (actually it will stop quickly).
http://www.tunnelvisionlabs.com/downloads/go/2012-02-10-goworks-ea4.zip
I'll set up a page next week for keeping a history of releases.
Thanks for the report (first indication I've had that anyone's even tried this)! :)
--
Sam Harwell
Owner, Lead Developer
-----Original Message-----
From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On Behalf Of ?
Sent: Friday, February 10, 2012 3:20 AM
To: golang-nuts
Subject: [go-nuts] Re: Starting a NetBeans-based IDE
nz
xiaoq...@doublelinepartners.com
On Sunday, January 15, 2012 5:12:22 PM UTC-6, Sam Harwell wrote:
Hello,First I'd like to introduce myself and give a bit of background. I'm Sam Harwell, and I'm a Ph. D. student at the University of Texas at Austin studying several aspects of the ways developers interact with IDEs to boost productivity. Currently I'm leading development of ANTLRWorks 2 (IDE) in parallel with Dr. Terence Parr's work on ANTLR 4. ANTLRWorks 2 supports development of ANTLR grammars and StringTemplate templates. Due to the simplicity of these languages, I've been looking for an additional general purpose language to work with so I can exercise some of the more advanced features of the framework powering ANTLRWorks 2. After a great deal of consideration, I've tentatively chosen Go as the language to bring into the fold.On Friday (1/13) at 4:30 I forked the ANTLRWorks 2 project to make the project "GoWorks". Somewhere between running errands and watching Mission Impossible, I built a fairly functional editor from the Go Language Specification. I give these times to point out the flexibility of ANTLR 4 and the ANTLRWorks 2 framework in building both a parser and editor for a new language. For today I'll give just a screenshot (tease!), along with what I'm hoping to accomplish in the short run.Goal 0: Within the next week
- Functional editor with syntax and semantic-aware highlighting for single files (analysis extends only to the file currently being edited)
- Initial release for public review
- Features very likely to be completely missing: multi-file analysis, project/build system, etc.
Goal 1: Within the next month