ceylon.formatter public beta (source code formatter)

[Lucas Werkmeister]

Hi all!

Over the past half year, I’ve been writing the ceylon.formatter, a Ceylon program that formats Ceylon source code and makes it pretty (fixes indentation and whitespace, that sort of stuff). I’m pretty sure it supports all Ceylon 1.1 language features, and there’s some options to customize how the code looks (although there’s definitely room for more here).
As I’ve written the formatter for Ceylon 1.1, which is not compatible with Ceylon 1.0, there’s no point in releasing it right now. Instead, I am now starting a “public beta” sort of thing, where you can test the formatter and tell me what bugs you find :)

To check it out, you will need an installation of Ceylon 1.1, at least the language module (the repository contains the needed SDK modules). Please refer to the ceylon-dist repository for installation instructions on that. Once that is in place, you can run

ceylon run --rep=https://lucaswerkmeister.github.io/ceylon.formatter/modules ceylon.formatter/1.1.0-BETA source

to format all Ceylon files in the source directory, or

ceylon run --rep=https://lucaswerkmeister.github.io/ceylon.formatter/modules ceylon.formatter/1.1.0-BETA source --to source-formatted

if you’re afraid I might break your code (there is built-in error recovery – if any error occurs, the original content should be restored – but better safe than sorry). Please use version control – I don’t want to be responsible for destroying your code. (If you use version control, git diff is also the fastest way to find out what changed.)

If you want to get a bit of help, use ceylon run ceylon.formatter -- --help (ceylon run ceylon.formatter --help doesn’t work due to ceylon-common#38). Otherwise, feel free to ask me any questions! I also hang out in this gitter chat room, if you want to try that out.

Cheers!

Lucas

PS: I suppose I should link to the GitHub repository as well, it’s not in the main text. Anyways, that’s where you can report bugs and, if you’re super awesome, even contribute code!

[Enrique Zamudio]

Awesome! Does it work only with whole files or just blocks of text? If it compiles to js we could use it in the web IDE, client side...

[Lucas Werkmeister]

I believe you could use it to write any AST node, although the UI doesn't encourage this at the moment. To preserve comments, you'd also need a TokenStream that yields exactly the tokens that make up the node.
It doesn't compile to JS because it depends on the Ceylon compiler :-/
if the compiler was rewritten in Ceylon, that would be great! :D

(My stupid phone sent the reply to Enrique instead of the group, so Enrique, sorry if you’re getting this twice.)

[Enrique Zamudio]

Do you really need the compiler, or just the typechecker?

[Tom Bentley]

It'll be the parser he needs, won't it?

On 27 Apr 2014 16:47, "Enrique Zamudio" <alte...@gmail.com> wrote:

Do you really need the compiler, or just the typechecker?

--
You received this message because you are subscribed to the Google Groups "ceylon-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceylon-users...@googlegroups.com.
To post to this group, send email to ceylon...@googlegroups.com.
Visit this group at http://groups.google.com/group/ceylon-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/b93fe6a4-572f-4070-a2a7-d3340643ab90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lucas Werkmeister]

I need:

    shared import com.redhat.ceylon.typechecker "1.1.0";
    shared import com.redhat.ceylon.common "1.1.0";

Parser / typechecker. (It’s implemented as a Visitor that writes the code basically from scratch.)

[Lucas Werkmeister]

So, to get the formatter working on JS, you only really need to write an ANTLR Ceylon backend. This page says that

You'll either get the feeling "Wow, that was easy!" and move on (that happened to me) or "Eeek, what a pain!" and let someone else to the work.

Anyone wants to try it? :)

[Tomáš Hradec]

Hi Lucas,

I wanted to format source code in ceylon.test module, but I got ...

$ ceylon run ceylon.formatter /home/thradec/workspace/ceylon/ceylon-sdk/source/ceylon/test/

Note: JBoss Modules version 1.1.3.GA 

ceylon run: 'other' is different type of Path

java.lang.IllegalArgumentException: 'other' is different type of Path

at sun.nio.fs.UnixPath.relativize(UnixPath.java:416)

at sun.nio.fs.UnixPath.relativize(UnixPath.java:43)

at ceylon.file.internal.ConcretePath.relativePath(ConcretePath.ceylon:87)

at ceylon.formatter.commandLineFiles_$1translate_$1visitor_.file(run.ceylon:137)

at ceylon.file.internal.ConcretePath$1fileVisitor_.visitFile(ConcretePath.ceylon:173)

at ceylon.file.internal.ConcretePath$1fileVisitor_.visitFile(ConcretePath.ceylon:177)

at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:135)

at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)

at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)

at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)

at java.nio.file.Files.walkFileTree(Files.java:2600)

at java.nio.file.Files.walkFileTree(Files.java:2633)

at ceylon.file.internal.ConcretePath.visit(ConcretePath.ceylon:182)

at ceylon.formatter.commandLineFiles_$1translate_.translate(run.ceylon:164)

at ceylon.formatter.commandLineFiles_$1translateToDirectory_.translateToDirectory(run.ceylon:177)

at ceylon.formatter.commandLineFiles_.commandLineFiles(run.ceylon:230)

at ceylon.formatter.run_.run(run.ceylon:274)

at ceylon.formatter.run_.main(run.ceylon)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at ceylon.modules.api.runtime.SecurityActions.invokeRunInternal(SecurityActions.java:58)

at ceylon.modules.api.runtime.SecurityActions.invokeRun(SecurityActions.java:48)

at ceylon.modules.api.runtime.AbstractRuntime.invokeRun(AbstractRuntime.java:85)

at ceylon.modules.api.runtime.AbstractRuntime.execute(AbstractRuntime.java:145)

at ceylon.modules.api.runtime.AbstractRuntime.execute(AbstractRuntime.java:129)

at ceylon.modules.Main.execute(Main.java:69)

at ceylon.modules.Main.main(Main.java:42)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.jboss.modules.Module.run(Module.java:270)

at org.jboss.modules.Main.main(Main.java:294)

at ceylon.modules.bootstrap.CeylonRunTool.run(CeylonRunTool.java:208)

at com.redhat.ceylon.common.tools.CeylonTool.run(CeylonTool.java:343)

at com.redhat.ceylon.common.tools.CeylonTool.execute(CeylonTool.java:283)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.redhat.ceylon.launcher.Launcher.run(Launcher.java:82)

at com.redhat.ceylon.launcher.Launcher.main(Launcher.java:21)

... am I doing something wrong?

And do you plan to integrate it into IDE ?

Regards

Tomas

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/c1bfbada-cfc6-4595-814a-1af1769798d4%40googlegroups.com.

[Lucas Werkmeister]

... am I doing something wrong?

Well you definitely shouldn’t get an exception like this… can you try again with 'ceylon.formatter/1.1.0-BETA-for-Tomáš-Hradec'? I added a small debug message.
Also, I always used relative paths... maybe that’s the reason? (In that case, I’ll of course have to fix it for absolute paths.)

And do you plan to integrate it into IDE ?

Definitely planned, but I’m having trouble getting all the classes loaded… this was my first attempt.

[Lucas Werkmeister]

Oops, Ceylon complains that '1.1.0-BETA-for-Tomáš-Hradec' “contains invalid characters”. Please try 'ceylon.formatter/1.1.0-BETA-for-thradec' instead.

[Lucas Werkmeister]

Actually, nevermind, I got the error as well when I tried it with $(realpath source). I’ll look into it.

[Tom Bentley]

> Ceylon complains that '1.1.0-BETA-for-Tomáš-Hradec' “contains invalid characters”

That sounds like a bug to me -- do you know where the problem arose? Could you open an issue for it?

Thanks,

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/b7d596bc-be1a-4ee5-a091-99c111c55407%40googlegroups.com.

[Lucas Werkmeister]

Which repository would that be? ceylon-common? (I don’t know where it happens.)

[Tomáš Hradec]

Hi Lucas, 

when I use relative path instead of absolute path to module ceylon.test, then I get following exception ...

ceylon.language.Exception "Unexpected token '<', expected '(' instead"

at ceylon.formatter.FormattingWriter$12.$call$(FormattingWriter.ceylon:623)

at ceylon.formatter.FormattingWriter$12.$call$(FormattingWriter.ceylon:606)

at ceylon.formatter.FormattingWriter.fastForward$priv$(FormattingWriter.ceylon:1101)

at ceylon.formatter.FormattingWriter.writeToken$canonical$(FormattingWriter.ceylon:605)

at ceylon.formatter.FormattingWriter.writeToken(FormattingWriter.ceylon:445)

at ceylon.formatter.FormattingVisitor.visitPositionalArgumentList(FormattingVisitor.ceylon:1144)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:497)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$PositionalArgumentList.visit(Tree.java:6311)

at ceylon.formatter.FormattingVisitor.visitInvocationExpression(FormattingVisitor.ceylon:861)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:449)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$InvocationExpression.visit(Tree.java:5657)

at ceylon.formatter.FormattingVisitor.visitExpression(FormattingVisitor.ceylon:547)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:443)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$Expression.visit(Tree.java:5580)

at ceylon.formatter.FormattingVisitor.visitSpecifierExpression(FormattingVisitor.ceylon:1407)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:521)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$SpecifierExpression.visit(Tree.java:6709)

at ceylon.formatter.FormattingVisitor.visitAttributeDeclaration(FormattingVisitor.ceylon:197)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:85)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$AttributeDeclaration.visit(Tree.java:1131)

at com.redhat.ceylon.compiler.typechecker.tree.CustomTree$AttributeDeclaration.visit(CustomTree.java:39)

at ceylon.formatter.FormattingVisitor.visitBody(FormattingVisitor.ceylon:257)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visitBlock(VisitorAdaptor.java:120)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:121)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$Block.visit(Tree.java:1676)

at ceylon.formatter.FormattingVisitor.visitMethodDefinition(FormattingVisitor.ceylon:950)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:93)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$MethodDefinition.visit(Tree.java:1271)

at ceylon.formatter.FormattingVisitor.visitBody(FormattingVisitor.ceylon:257)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visitClassBody(VisitorAdaptor.java:122)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:123)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$ClassBody.visit(Tree.java:1704)

at ceylon.formatter.FormattingVisitor.visitAnyClass(FormattingVisitor.ceylon:139)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visitClassDefinition(VisitorAdaptor.java:70)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:71)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$ClassDefinition.visit(Tree.java:937)

at com.redhat.ceylon.compiler.typechecker.tree.Walker.walkCompilationUnit(Walker.java:30)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$CompilationUnit.visitChildren(Tree.java:30)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visitAny(VisitorAdaptor.java:10)

at ceylon.formatter.FormattingVisitor.visitAny(FormattingVisitor.ceylon:1790)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visitCompilationUnit(VisitorAdaptor.java:12)

at com.redhat.ceylon.compiler.typechecker.tree.VisitorAdaptor.visit(VisitorAdaptor.java:13)

at com.redhat.ceylon.compiler.typechecker.tree.Tree$CompilationUnit.visit(Tree.java:22)

at ceylon.formatter.format_.format(format.ceylon:63)

at ceylon.formatter.run_.run(run.ceylon:293)

Suppressed: ceylon.language.AssertionError "Assertion failed

violated false"

at ceylon.formatter.FormattingWriter.intersectAllowedLineBreaks$canonical$(FormattingWriter.ceylon:409)

at ceylon.formatter.FormattingWriter.intersectAllowedLineBreaks(FormattingWriter.ceylon:381)

at ceylon.formatter.FormattingWriter.writeToken$canonical$(FormattingWriter.ceylon:604)

at ceylon.formatter.FormattingWriter.writeToken(FormattingWriter.ceylon:445)

at ceylon.formatter.FormattingWriter.destroy(FormattingWriter.ceylon:1175)

at ceylon.formatter.FormattingVisitor.destroy(FormattingVisitor.ceylon:1794)

at ceylon.formatter.format_.format(format.ceylon:62)

... 28 more

T.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/f5211294-c782-4e04-9827-3f50bf562b04%40googlegroups.com.

[Lucas Werkmeister]

Alright, I disallowed absolute paths for now (I’ll have to figure out what to do with them later, the meaning when used with --to isn’t quite clear to me – #48), and fixed your other bug (I forgot that invocations can have type arguments – #49). Thanks a lot!

[Tom Bentley]

> I forgot that invocations can have type arguments

That suggests that a worthwhile sanity check would be to throw a whole lot of source code at your formatter to check you've not missed other things -- can you format the source code in ceylon.language and the SDK for instance?

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/93a74c17-bf57-4406-8d4f-42ccbd6de79e%40googlegroups.com.

[Lucas Werkmeister]

Will do. I thought the formatter's source code was extensive enough to catch all such oversights; evidently, I was wrong.

---

Von: Tom Bentley
Gesendet: ‎28.‎04.‎2014 15:26
An: ceylon...@googlegroups.com
Betreff: Re: [ceylon-users] Re: ceylon.formatter public beta (source code formatter)

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/CAMd5Ysw7U8E1DLxfW%3DXwKJcoP-Jw4ihbYwLwjyTREs4mptNx%3Dg%40mail.gmail.com.

[Lucas Werkmeister]

Alright. I fixed a lot more missing things that I found formatting the SDK (only rarely used features like dynamic, setters, type parameters… ahem), and then I found this weird syntax:

variable value map = TreeMap(compare,
        elements map (Element e) => e->present);

What in the world –? You can omit the parentheses around a positional argument list, and use a space instead of a '.' member op? What black magic is this?
I can’t find it in the spec. I don’t see it in the grammar. I’m seriously considering I might be hallucinating…
… and if I’m not, please tell me this is unsupported and I can just ignore it. Please :-/

[Tako Schotanus]

On Tue, Apr 29, 2014 at 12:05 AM, Lucas Werkmeister <lucas.we...@googlemail.com> wrote:

I can’t find it in the spec. I don’t see it in the grammar. I’m seriously considering I might be hallucinating…
… and if I’m not, please tell me this is unsupported and I can just ignore it. Please :-/

See http://www.ceylon-lang.org/documentation/spec/html_single/#operatorstyleexpressions

And the discussion about it https://github.com/ceylon/ceylon-spec/issues/578

At least , that's what I think you're talking about, right?

-Tako

[Lucas Werkmeister]

That's it, yeah. Thanks! FTR, I've always hated this kind of syntax in IIRC Python and Ruby based on a feeling that writing code like this might bite me because of ambiguities I don't understand... but that doesn't really matter now, and if this is restricted to special invocations only, I'll probably be able to implement it (although I do find the way it's implemented in the AST - ghost MemberOperators with a null token and that sort of trickery - quite irritating).
But at least I wasn't seeing things :-)

---

Von: Tako Schotanus
Gesendet: ‎29.‎04.‎2014 00:43

An: ceylon...@googlegroups.com
Betreff: Re: [ceylon-users] Re: ceylon.formatter public beta (source code formatter)

--

You received this message because you are subscribed to the Google Groups "ceylon-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceylon-users...@googlegroups.com.
To post to this group, send email to ceylon...@googlegroups.com.
Visit this group at http://groups.google.com/group/ceylon-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/CAOJRyvpF2BP2c6C6CXTSyDsMq795r%3DTM8z09CocU9Sy85_4sPA%40mail.gmail.com.

[Lucas Werkmeister]

Alright, I fixed even more things, and now the whole Ceylon SDK can be formatted! (I get two failing tests, but they fail with the unmodified sources as well.)

[Lucas Werkmeister]

Fixed a few more things, now ceylon.language works as well.

[Gavin King]

Excellent! Congrats. Now we need to get it integrated into Eclipse!

> --
> You received this message because you are subscribed to the Google Groups
> "ceylon-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ceylon-users...@googlegroups.com.
> To post to this group, send email to ceylon...@googlegroups.com.
> Visit this group at http://groups.google.com/group/ceylon-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ceylon-users/7e004225-50a4-4936-9f24-a350cb0c766c%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
Gavin King
ga...@ceylon-lang.org
http://profiles.google.com/gavin.king
http://ceylon-lang.org
http://hibernate.org
http://seamframework.org

[Lucas Werkmeister]

Has someone tested this on Windows? I’m not sure if the file path handling will work there.

[Lucas Werkmeister]

Here’s a neat thing that you can do with a command-line formatter:
A git pre-push hook to check that your new code is formatted.
Especially important since there’s no IDE integration yet (in Java, I usually have “format on save” enabled so that the problem doesn’t even arise in the first place).