Making Lombok.jar ajc compatible

[Eric B]

I've been chatting a little on the AspectJ mailing list with Andy Clements (lead dev for aspectj) regarding the compatibility issues between Lombok and AJ.

He said the following:

"I had a brief look at this when we added annotation processing support to AspectJ as that is what I thought Lombok was doing. But then I discovered that I think it wanted to be run as an agent when using the Eclipse Java Compiler (on which AspectJ is based). If I recall correctly lombok had hardcoded classnames for ECJ classes in it and in AspectJ we prefix those with "org.aspectj." - that is as far as I got looking though, I'm afraid.  It would be great if someone had a bit more time than me to dig into it.  Possibly you just need a lombok that recognizes this variant of ECJ.

...

I can't say whether lombok would be compatible apart from the package prefix but my gut feeling would be that it would just work.  I wonder if running a jarjar on the lombok code to replace occurrences org org.eclipse.jdt. with org.aspectj.org.eclipse.jdt. might get it into a working state."

I ran a quick test and jarjar'ed lombok from the master branch (1.14.9) to replace all org.eclipse.jdt with org.aspectj.org.eclipse.jdt.  I then copied the resulting jar into my eclipse folder, replacing the original lombok.jar.

Relaunched eclipse, tried to enable AspectJ and it still fails.

I tried looking at the lombok sources, but to be honest, it is a little over my head to understand entry points and even how to debug if/how this is working.  I'd be happy to help out trying to figure this out, but to be honest, I'm a little lost where/how to start.  I'm not even sure of the entire Eclipse/javac/AspectJ compile process so I would need a lot of guidance.

I'm hoping that Andy's quote above can trigger a quick thought for one of the main devs who can better understand how this can work and hopefully provide guidance.

Thanks,

Eric

[Eric B]

I've since tried digging through the Lombok code to see if I can figure out where/how this is failing, but am having a difficult time understand how it is written, and more importantly, even how to initiate debug sessions in eclipse.

Does anyone know how to setup a proper project so that I can try to put in breakpoints and debug it while launching an ajc process?

Thanks,

Eric

[Martin Grajcar]

A fast but maybe useless answer: I do have an eclipse run configuration called LombokizedEclipse.launch, which still works. I created it a long time ago and remember hardly anything. From github:

HINT: If you'd like to develop lombok in eclipse, run 'ant eclipse' first. It creates the necessary project infrastructure and downloads dependencies. Note that, in order to run "LombokizedEclipse.launch", you need to have "Eclipse SDK" installed.

IIRC ant eclipse overwrites eclipse .project and .classpath without any warning.

--
You received this message because you are subscribed to the Google Groups "Project Lombok" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-lombo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric B]

On Fri, Sep 19, 2014 at 5:51 PM, Martin Grajcar <maaar...@gmail.com> wrote:

A fast but maybe useless answer: I do have an eclipse run configuration called LombokizedEclipse.launch, which still works. I created it a long time ago and remember hardly anything. From github:

HINT: If you'd like to develop lombok in eclipse, run 'ant eclipse' first. It creates the necessary project infrastructure and downloads dependencies. Note that, in order to run "LombokizedEclipse.launch", you need to have "Eclipse SDK" installed.

Thanks.  I don't think I was clear in my original post.  I created my eclipse project using `ant eclipse`.  But once compiled, I am having trouble figuring out how to debug this once it is compiled.

The closest I have figured out, is to launch my ant build process on a separate project with remote debugger then attach my remote debugger to the process and put a breakpoint in the Lombok code.:

ANT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=1044

But that seems like a very convoluted way to try to debug Lombok code.

Additionally, when I try to debug the Lombok code, and the process moves into some of the libs (ex: net.java.openjdk.custom-javac6.jar), the debugging line doesn't seem to match the source code line.

So I can only presume that I must be doing something wrong, or there is a better way to launch the procedure.

Any suggestions would be great.

Thanks,

Eric

[Martin Grajcar]

On Sun, Sep 21, 2014 at 4:44 AM, Eric B <eben...@gmail.com> wrote:

On Fri, Sep 19, 2014 at 5:51 PM, Martin Grajcar <maaar...@gmail.com> wrote:

A fast but maybe useless answer: I do have an eclipse run configuration called LombokizedEclipse.launch, which still works. I created it a long time ago and remember hardly anything. From github:

HINT: If you'd like to develop lombok in eclipse, run 'ant eclipse' first. It creates the necessary project infrastructure and downloads dependencies. Note that, in order to run "LombokizedEclipse.launch", you need to have "Eclipse SDK" installed.

Thanks.  I don't think I was clear in my original post.  I created my eclipse project using `ant eclipse`.  But once compiled, I am having trouble figuring out how to debug this once it is compiled.

Don't you have this run configuration?

 

The closest I have figured out, is to launch my ant build process on a separate project with remote debugger then attach my remote debugger to the process and put a breakpoint in the Lombok code.:

ANT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=1044

But that seems like a very convoluted way to try to debug Lombok code.

Indeed.

 

Additionally, when I try to debug the Lombok code, and the process moves into some of the libs (ex: net.java.openjdk.custom-javac6.jar), the debugging line doesn't seem to match the source code line.

This is well-known. Lombok uses just some stubs so it compiles, I guess getting the real sources would help. IIRC it was complicated enough to make me live without them.

 

So I can only presume that I must be doing something wrong, or there is a better way to launch the procedure.

Any suggestions would be great.

All I could remember was said, maybe someone else knows.

[Roel Spilker]

I'll have a look at your debugging problem next week.

Also, did you look at lombok.patcher? That's where the eclipse and ecj magic happens.

Roel

[Eric B]

On Sat, Sep 20, 2014 at 11:28 PM, Martin Grajcar <maaar...@gmail.com> wrote:

On Sun, Sep 21, 2014 at 4:44 AM, Eric B <eben...@gmail.com> wrote:

On Fri, Sep 19, 2014 at 5:51 PM, Martin Grajcar <maaar...@gmail.com> wrote:

A fast but maybe useless answer: I do have an eclipse run configuration called LombokizedEclipse.launch, which still works. I created it a long time ago and remember hardly anything. From github:

HINT: If you'd like to develop lombok in eclipse, run 'ant eclipse' first. It creates the necessary project infrastructure and downloads dependencies. Note that, in order to run "LombokizedEclipse.launch", you need to have "Eclipse SDK" installed.

Thanks.  I don't think I was clear in my original post.  I created my eclipse project using `ant eclipse`.  But once compiled, I am having trouble figuring out how to debug this once it is compiled.

Don't you have this run configuration?

Nope - don't have that run/debug configuration in my project.  Strange.  I tried to clean my folder (removed .project, .settings, etc) and ran ant clean, ant eclipse, but still don't have the runtime configuration show up.  Any chance you can share the config please?

 

Additionally, when I try to debug the Lombok code, and the process moves into some of the libs (ex: net.java.openjdk.custom-javac6.jar), the debugging line doesn't seem to match the source code line.

This is well-known. Lombok uses just some stubs so it compiles, I guess getting the real sources would help. IIRC it was complicated enough to make me live without them.

Great - my problem is that I am not sure where/what to step through, so I figured starting at the entry point and stepping through slowly would point out where I need to go.  But with the wrong sources, it is very difficult to figure this out.

Wonder why not just use the real sources instead of stubs; the org.eclipse libs are all publicly available.

Thanks,

Eric

[Eric B]

On Sun, Sep 21, 2014 at 4:59 AM, Roel Spilker <r.sp...@gmail.com> wrote:

I'll have a look at your debugging problem next week.

Also, did you look at lombok.patcher? That's where the eclipse and ecj magic happens.

I'm not sure if that is a package name, but I don't have that anywhere in my src folder.  The only things I found with "patch" in their class names are in the agent/ folder, but I figured getting the compiler working before dealing with the agent would be easier.

Thanks,

Eric

[Roel Spilker]

It is a separate project on github, next to lombok: https://github.com/rzwitserloot/lombok.patcher

[Martin Grajcar]

On Sun, Sep 21, 2014 at 10:15 PM, Eric B <eben...@gmail.com> wrote:

Thanks.  I don't think I was clear in my original post.  I created my eclipse project using `ant eclipse`.  But once compiled, I am having trouble figuring out how to debug this once it is compiled.

Don't you have this run configuration?

Nope - don't have that run/debug configuration in my project.  Strange.  I tried to clean my folder (removed .project, .settings, etc) and ran ant clean, ant eclipse, but still don't have the runtime configuration show up.  Any chance you can share the config please?

No idea, what's exactly needed, so I packed a bit more:

https://dl.dropboxusercontent.com/u/4971686/lombok/lombokized-eclipse-or-whatever.zip

And maybe a totally wrong thing.

I don't have a clue how I created it.

Additionally, when I try to debug the Lombok code, and the process moves into some of the libs (ex: net.java.openjdk.custom-javac6.jar), the debugging line doesn't seem to match the source code line.

This is well-known. Lombok uses just some stubs so it compiles, I guess getting the real sources would help. IIRC it was complicated enough to make me live without them.

Great - my problem is that I am not sure where/what to step through, so I figured starting at the entry point and stepping through slowly would point out where I need to go.  But with the wrong sources, it is very difficult to figure this out.

Wonder why not just use the real sources instead of stubs; the org.eclipse libs are all publicly available.

No idea. Maybe because they differ between versions, so it won't work for all, anyway? And because of the size? Or I may be talking non-sense...

[Eric B]

On Sun, Sep 21, 2014 at 4:31 PM, Roel Spilker <r.sp...@gmail.com> wrote:

It is a separate project on github, next to lombok: https://github.com/rzwitserloot/lombok.patcher

Now, I'm even more confused.  How does lombok use lombok.patcher during command-line compile?  From what I understand from the Readme, lombok.patcher is used prob by the Eclipse agent to keep classes up to date as you write your code.  But I'm actually just trying to work on the basics at the moment - command-line compilation.  I figured that would be the easiest to start with.

Am I approaching this in the wrong way?

Thanks,

Eric

[Reinier Zwitserloot]

Lombok runs as an annotation processor on javac.

However, in ecj we can't make it work that way, we need more control, so we run as an agent instead.

Unfortunately, the way agents work with JVMs, is that they _MUST_ be in jar form. This is unique in java land (everything else works with classpath or classloaders) and as a result, debuggers get confused.

The just released shadowlauncher stuff is part of an effort to address this, incidentally.

The solution to the aspectJ issue is complicated and requires metaprogramming, I think: Lombok uses org.eclipse.* classes in tons of places, and you'd need to duplicate it all, with the org.aspectj variant of all these classes. You'd need to do this for all eclipse handlers, and then have the patcher also patch the org.aspectj variants to call into TransformEclipseAST, and then have that code be aware of having to call the aspectj variant clones instead of the originals.

lombok.patcher is a framework to write scripts that modify bytecode. For example, to make lombok work in eclipse, we find (amongst many other things) the method that completes a parserun in the ECJ that's part of eclipse, and patch the bytecode to add a call into TransformEclipseAST just before the parser returns a completed AST. This gives lombok the chance to modify that AST by applying all lombok transformations to it. Lombok (the agent) uses this framework to, as an agent, set up scripts to fix eclipse.

The bootup procedure of lombok in eclipse is as follows:

* eclipse.exe starts java.exe, with -javaagent:lombok.jar passed as VM arg.

* The VM fires up a special classloader which loads the agent code. The agent code registers a bytecode transformer and does nothing else.

* Eclipse boots as normal. every single class loaded by the VM (which is, of course, tons and tons of classes) are read in as normal but before the raw bytes are turned into a class definition, the raw bytes are provided to all registered transformers. Each transformer may optionally choose to change the bytecode.

* the lombok bytecode transformer checks every incoming class against the list of patch scripts. If it finds a class that may need to be patched, it'll 'parse' the class file with ASM (a library to read and modify bytecode), and apply the script. It's as if your eclipse is a custom installation with lombok calls injected in the right places, but we fix it all 'live' without actually changing any of jars in your eclipse directory on your harddisk.

* Eclipse continues booting with these slightly modified classes. Because the classes are modified, when eclipse parses source files, the parser actually calls lombok to do its magic before it returns parsed ASTs.

In order to add aspectJ support, you need to:

* Update the patch scripts. The main eclipse patch scripts are in https://github.com/rzwitserloot/lombok/blob/master/src/eclipseAgent/lombok/eclipse/agent/EclipsePatcher.java ; as you can see there, there's LOTS of hardcoded references to eclipse class names, such as "org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration". You'd have to leave these in there, but also add org.aspectj.org.eclispe.jdt.internal.compiler.etcetera.

* The patch scripts patch a call into TransformEclipseAST into various places. This code picks up the value that is about to be returned and does a lot of stuff with it. This code is compiled assuming that the incoming object's type is an "org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration" (which has children, which also have assumed types), but if lombok is being applied to aspectJ compiled results, that won't be the case; they'll be org.aspectj.etceteras. Short of rewriting all of the eclipse-specific lombok code to use reflection, or using metaprogramming (writing a tool that takes compiled lombok code and uses bytecode modification to turn all these calls into reflection, which duplicates all eclipse-specific lombok code and adds a switch that calls into one of the two duplicates based on incomingParam.getClass().getName()), this is impossible.

As you can see, lots of work. If we had the time to do it, we'd do it, but, we don't, and we don't use AspectJ.

Roel

To unsubscribe from this group and stop receiving emails from it, send an email to project-lombok+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Lombok" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-lombok+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Lombok" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-lombok+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Lombok" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-lombok+unsubscribe@googlegroups.com.