Proposal for the next big thing

[Alex Eagle]

I've made some noises about this, now here it is. I think we should design the language around the source repository instead of the filesystem. I think I've made peace with how this can be practical, by implementing just what we have to, and giving the right API's for others to innovate and fill in missing parts.

First thing, we need a source repository. The source repository wraps an abstract semantic graph. It provides:

 * a language model API, things like Project, Library, Class, Method, ReturnStatement, StringLiteral

 * some way to persist these things to disk

 * an API to read and mutate the model

Bobby and Christian, as the guys who are interested in working on a translator, I think this is perfectly suitable for your needs. You'd need a populated source repository, which we can do with the mutation API at first. You'd get a reference to the root element of the program and then you're in the same place you'd be if you got the graph from parsing a program from text files.

Everything else talks to the source repository:

 * a distributed SCM, using the same model as git or hg, which lets you share your local source repository with others. Since the source repository doesn't contain text files, I don't think we can use git or hg directly. I hope that Shawn Pierce will have a little bit of time to consult on this. He wrote jgit and I've talked to him about this already.

 * text editors. These are possible by using a grammar and falling back to traditional means, by formatting out the code into files on disk, coding, then parsing it all back in, detecting diffs, and applying them as mutations. I think there are some downsides to this, but I admit they need to be supported. Jeremie and Shyam, it sounds like you may have a bit of time this quarter, would you like to own this? Note how much complexity of a traditional language lives here :)

 * noop-specific editors and IDE plugins. Here's where I hope Jesse will find some time to mock up an editor that's so much awesomer because it operates directly on the semantic graph. I think online editors are the way of the future, but an Eclipse plugin would be great too, and Matthieu Réjou has already started one (although it's text based, and maybe he'll run away if we go this way :)

 * a compiler is nicely supported by having the semantic graph immediately accessible. Note that the compiler can be read-write wrt. the repository, for instance, adding warnings as comments.

 * a code review tool, which would be much better if the deltas to be reviewed were semantic. I'll mock this up.

While our semantic design for Noop is unchanged, this is pretty disruptive to the way anyone codes today. I realize I need to explain why this is worth doing, and I'll work on assembling all of that into some form. I think verbal is best, so maybe I'll record something? I don't think a slide deck will do it.

Thoughts? I'm sure there are many! :)

-Alex

[Christian Edward Gruber]

Ok, while I want the first step, which is a language I want to program in - I'm more than happy to move this to an environment I want to program in... so long as the language project itself isn't bound in this.  In other words, there's a means to use this in an environment where the below-listed awesomeness isn't practically deployable.  

So, either we have to make it practically deployable anywhere (ideal), or we have to make it so that the language itself and the environment co-evolve, but that alternate expressions are possible.  I think that's a workable goal.

Having said the above, I'm NOT saying this isn't pure win.  Its just insanely ambitious, and my initial needs and desires for noop are well satisfied short of this.  But I'm happy to stay involved in both the language and environment development.

Christian.

[Matthieu Réjou]

Hello,

As I understood it, it's "just" another way of thinking cooperation
between editor, repository and compiler. It does not seems too
ambitious for me, even if it's really disruptive.

The few IDE editors I know already maintain semantic graphs from text
file, using model to generate code, format, search, do some
refactoring... Any programming language tool, in fact (alex was
talking about code review tool and translator, for example). I mean I
think it's coherent the repository does the same, no ?

Otherwise, about the online editor proposal, I'm quite suspicious.
Even if RIA are richer and richer, nowadays IDEs provide powerful
(huge?) functions a browser can't give efficiently, I think..?

-Matthieu

On Apr 7, 1:53 am, Christian Edward Gruber

[Alex Eagle]

Yes, that's just what I was thinking. Most tools need to operate at the semantic level, and starting from text makes each of them work harder and understand the code less accurately. So in a way, it's just pushing the whole text files thing into a particular editing toolset rather than imposing it on the whole language ecosystem. Although I agree with Christian that it's pretty ambitious to rewrite version control along with the language. We can discuss tactics further, so that we aren't on the hook to produce so much stuff just to get the language going.

I mocked up some code for the repository, here's a diagram of hello world represented as a semantic graph (note that edges should be ordered, so that statements in a block remain in the right sequence)

The source is here (maven + java + git :) http://github.com/alexeagle/Noop-SCM/blob/master/src/test/java/noop/operations/OperationsExampleMain.java

I'm modeling the mutations as operations, with the thought that one day we could use operational transformation as in Wave to allow concurrent editing.

+dgrundman

I'm also using Doug's idea that he mailed to noop@ back on 14 Jan, where a block is just a block, and depending on its parent and attributes, it might be a function like the "say hello" above, or a method like "print" above, or associated with an if statement, etc.

-Alex

2010/4/7 Matthieu Réjou <matthie...@gmail.com>

Hello,

As I understood it, it's "just" another way of thinking cooperation
between editor, repository and compiler. It does not seems too
ambitious for me, even if it's really disruptive.

The few IDE editors I know already maintain semantic graphs from text
file, using model to generate code, format, search, do some
refactoring... Any programming language tool, in fact (alex was
talking about code review tool and translator, for example). I mean I
think it's coherent the repository does the same, no ?

Otherwise, about the online editor proposal, I'm quite suspicious.
Even if RIA are richer and richer, nowadays IDEs provide powerful
(huge?) functions a browser can't give efficiently, I think..?

I suppose we'll just have to see if the browser can keep up. Remember that some operations can make a roundtrip to the server, and if the server is efficient, it might be just as fast as waiting for the IDE on your local machine to do that operation. The benefits would be great, though, aside from letting me code on a ChromeBook when I get my hands on one, you'd be able to get into a fully working IDE setup (even when doing a code review) with a single click.

Have you seen code bubbles? I think that's a great model for what a new editor can do.

http://www.cs.brown.edu/people/acb/codebubbles_site.htm

--
To unsubscribe, reply using "remove me" as the subject.

[Christian Edward Gruber]

I'm totally down with all this.  THe main thing for me is that we have a prioritized list of what we're hammering out, and we may need sub-teams with different prioritized subsets of the overall list... mostly because we're all interested in different parts.  But I think this can be really exciting.  

As to the online IDE, given my experience with online tools at Google, both what we offer to the outside world and what we do internally, I'm convinced that an HTML5-capable browser is well and truly able to handle this, provided we get the balance right of what's on the front end and what's on the back-end and what's communicated.  There will be the art.  But we do some really heavy lifting with things like spreadsheets.google.com and gmail that amazed me.  And it was our internal code review and source search systems and the collaborative editing we provide to the world at docs.google.com that made me think this whole rig was possible.  I think Alex has taken the idea even farther, and I am enthused.

But I just want the language first... but that's my priority. :-)

Christian

On Apr 7, 2010, at 10:10 AM, Alex Eagle wrote:

Yes, that's just what I was thinking. Most tools need to operate at the semantic level, and starting from text makes each of them work harder and understand the code less accurately. So in a way, it's just pushing the whole text files thing into a particular editing toolset rather than imposing it on the whole language ecosystem. Although I agree with Christian that it's pretty ambitious to rewrite version control along with the language. We can discuss tactics further, so that we aren't on the hook to produce so much stuff just to get the language going.

I mocked up some code for the repository, here's a diagram of hello world represented as a semantic graph (note that edges should be ordered, so that statements in a block remain in the right sequence)

The source is here (maven + java + git :) http://github.com/alexeagle/Noop-SCM/blob/master/src/test/java/noop/operations/OperationsExampleMain.java

[Matthieu Réjou]

Forgot I was talking with Google guys :)

It's true getting a fully configured and operational workspace in a
single click is very attractive. And code bubbles concept sounds
awesome, I didn't know it, thanks (should test it in production
anyway...).
So after having read your arguments and searched for more informations
on the web, I'm more convinced, and hope others will too.

Anyway, I'll try to finish a few basic editor features on my eclipse
plugin and stop there, waiting for precise priorities and
orientations : don't want to run too far away :)

-Matthieu

On 7 avr, 23:19, Christian Edward Gruber

> > The source is here (maven + java + git :)http://github.com/alexeagle/Noop-SCM/blob/master/src/test/java/noop/o...

[Christian Edward Gruber]

Is your plugin in a public repository? Sorry, I think I lost the
early part of the thread.

Christian.