Upgrading from Ebean Agent 2.7.4 (to 3, 4, 8, 12 and 13): "ignore class" at compile time, and others at runtime

[sza...@gmail.com]

Hi group, hi Rob!

I'm trying to upgrade the play-ebean module of the old Play! 1 Framework to a newer Ebean release.

It's a nice module which replaces the builtin JPA module with the Ebean ORM.

It enhances itself at compile time with the AntEnhanceTask, and enhances the Play! 1 Framework apps at runtime with the Transformer class.

Upgrading the ORM part was straightforward work, thanks to the awesome developer log documented as issues, categorized with labels! 🙏

With the agent part, I thought many times that the upgrade of the agent would work but the sample app always failed at the end.

Starting at Ebean Agent 4.6.1, the ebeanEnhance task started to ignore my classes, including my Entity class:

[ebeanEnhance] ebean-enhance> transform> pkg: play/modules/ebean
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/Model  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanEnhancer$PlayClassBytesReader  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader$1  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanDataSourceWrapper  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/HSQLPlatform  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanSupport  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader$3  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader$2  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanEnhancer  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader$5  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/PlayAwareTransformer  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanModelAdapter  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin$EbeanModelLoader$4  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPlugin  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanContext  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/EbeanPostLoader  msg: ignore class
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/PlayAwareClassWriter  msg: ignore class

If I move the "last" directory from "classSource" parameter to "packages", like below:

classSource="${basedir}/tmp" packages="play.modules.ebean" ->

classSource="${basedir}" packages="tmp.play.modules.ebean"

then the agent detects my classes again, but it is unable to read superMeta of my Entity class:

[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/Model  msg: read information about superClasses play/modules/ebean/EbeanSupport to see if it is entity/embedded/mappedSuperclass
[ebeanEnhance] ebean-enhance> cls: play/modules/ebean/Model  msg: unable to read superMeta for play/modules/ebean/EbeanSupport

(Newer Agent versions read that superMeta successfully.)

Rob, you wrote that the Maven and Gradle plugins are the recommended way of compile time enhancing.

So I converted the project to Gradle (and also tried the "basic-gradle-java"), but it didn't help: I got the very same "msg: ignore class" messages.

Am I missing something? Adding the "First Entity" (w/o Model) to both packages it shows
 that the class in play.modules.ebean is ignored while the other in org.example.domain is enhanced.

And later it breaks the CustomerTest. 🙈

CustomerTest > queryBean() FAILED
    java.lang.ExceptionInInitializerError
        at io.ebean.DB.<clinit>(DB.java:64)
        at io.ebean.Model.db(Model.java:130)
        at io.ebean.Model.save(Model.java:192)
        at org.example.domain.CustomerTest.queryBean(CustomerTest.java:20)

        Caused by:
        io.ebean.config.BeanNotEnhancedException: Bean class play.modules.ebean.SimpleCustomer is not enhanced? Check packages specified in ebean.mf. If you are running in IDEA or Eclipse check that the enhancement plugin is installed. See https://ebean.io/docs/trouble-shooting#not-enhanced

(I didn't write much about the runtime enhancement problems. I hope fixing the compile time enhancement fixes the runtime enhancement too.)

Thank you,
Szabolcs

[Rob Bygrave]

ebean-agent has a class called IgnoreClassHelper ... and that is ignoring anything that has the top level package of "play" ... this is why it ignores "play.modules.ebean.SimpleCustomer"

See https://github.com/ebean-orm/ebean-agent/blob/master/ebean-agent/src/main/java/io/ebean/enhance/common/IgnoreClassHelper.java#L25

Looks like that change happened Aug 2015 - version 4.6.1 via https://github.com/ebean-orm/ebean-agent/issues/18

Q: Can you move SimpleCustomer and other entities to another package (not a top level package of "play") ?

Note that with respect to play/modules/ebean/Model, that was effectively moved into ebean itself and changed a little (as it had a couple of issues with it).  So in current Ebean, we expect apps to use io.ebean.Model instead of play.modules.ebean.Model.  I don't think the old play Model would work properly.

Q: Are you able to also change to use io.ebean.Model?

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/5189e508-9e34-40ba-b050-adad4409eac5n%40googlegroups.com.

[Szabolcs Hubai]

Hi Rob,

Thanks for the pointer!

And that configurable IgnoreClassHelper which introduced in 4.6.1 with issue #18, got a refactor with the introducing of ebean.mf in version 10.2.1 (with issue #53).

Which in turn got an automatic generation feature, landed in 12.1.8.

Q: Can you move SimpleCustomer and other entities to another package (not a top level package of "play") ?

While most (if not all) Play! 1 Framework modules are in package play.modules.*, of course I could move them into another.

But for a few versions (of this play-ebean module) I'd like to stick at play.modules.ebean while providing an upgrade path.

What options do I have at compile time with the AntEnhanceTask or with the Gradle plugin to force processing my play.modules.ebean package?

In Play! Framework (version 2) both play-ebean modules use some AntEnhanceTask-like magic to enhance their play.db.ebean package.

Q: Are you able to also change to use io.ebean.Model?

Currently my play.modules.ebean.Model extends play.modules.ebean.EbeanSupport which in turn implements play.db.Model to bring the two world together.

If I try to extend io.ebean.Model while keeping play.db.Model compatibility I got type erasure errors, so it won't work.

I plan to move to io.ebean.Model and drop support of play.db.Model, but I like to provide an upgrade path.

At runtime enhancement of the Play! Framework apps, this ebean module adds some Ebean (Query or Finder(??)) related methods with the Transformer class.

If I start using io.ebean.Model in the module, then I still have to call those ctClass.addMethod(ctMethod.make(...))?

Or the Transformer would take care of them?

Thanks,

Szabolcs

[Szabolcs Hubai]

Hi group,

with manual class enhancement I am able to enhance my play.modules.ebean package, but it doesn't enhance correctly.

My build.gradle snippet is:

...

classes.doLast {

    def cl = getClass().getClassLoader()

    new OfflineFileTransform(

        new Transformer(

            new PlayContext(

                new ClassPathClassBytesReader(null),

                'printversion=true;debug=10',

                new AgentManifest(cl)

            )

        ),

        cl,

    sourceSets.main.output.classesDirs.getAsPath()

    ).process("**")

}

...

class PlayContext extends EnhanceContext {

    PlayContext(ClassPathClassBytesReader reader, String agentArgs, AgentManifest manifest) {

        super(reader, agentArgs, manifest)

    }

    @Override

    public boolean isIgnoreClass(String className) {

        !className.startsWith("play/") && super.isIgnoreClass(className)

    }

}

In the build log it enhances, but the tests are failing with "java.lang.InstantiationError: io.ebean.bean.EntityBeanIntercept"

> Task :test FAILED

CustomerTest > queryBean() FAILED
    java.lang.InstantiationError: io.ebean.bean.EntityBeanIntercept
        at org.example.domain.BaseDomain.<init>(BaseDomain.java:13)
        at org.example.domain.Customer.<init>(Customer.java:23)
        at org.example.domain.CustomerTest.queryBean(CustomerTest.java:19)

CustomerTest > saveAndFind() FAILED
    java.lang.InstantiationError: io.ebean.bean.EntityBeanIntercept
        at org.example.domain.BaseDomain.<init>(BaseDomain.java:13)
        at org.example.domain.Customer.<init>(Customer.java:23)
        at org.example.domain.CustomerTest.saveAndFind(CustomerTest.java:32)

2 tests completed, 2 failed

It doesn't matter if I enhance all the project manually or just my ignored package, it fails with these exception.

If I rename the package from play.modules.ebean to play1.modules.ebean then the gradle plugin enhances it happily and no test failure occurs.

So the ultimate solution is to use the gradle plugin in the end. And it would be even better if the plugin would have some parameter to allow ignored packages. 😊

(This is just a FYI, my questions are still open.)

--

Szabolcs

On Wednesday, May 11, 2022 at 11:23:16 AM UTC+2 Szabolcs Hubai wrote:

Hi Rob,

Q: Can you move SimpleCustomer and other entities to another package (not a top level package of "play") ?

While most (if not all) Play! 1 Framework modules are in package play.modules.*, of course I could move them into another.

But for a few versions (of this play-ebean module) I'd like to stick at play.modules.ebean while providing an upgrade path.

What options do I have at compile time with the AntEnhanceTask or with the Gradle plugin to force processing my play.modules.ebean package?

In Play! Framework (version 2) both play-ebean modules use some AntEnhanceTask-like magic to enhance their play.db.ebean package.

Thanks,

Szabolcs

[Szabolcs Hubai]

Another follow-up.

Just read ebean-orm/ebean#2689: "IllegalStateException: Error trying to create the prototypeEntityBean for class".

If I rollback to 12.16.1 from 13.6.1, and enhance manually my own play.modules.ebean classes only and leave to plugin the others, it doesn't break the tests! 🙈

But if I try to enhance all classes manually, I get "property id not found in [....] for type ..." like here.

    10:16:54.437 [Test worker] ERROR io.ebean.internal - Error in deployment
    java.lang.IllegalStateException: If you are running in an IDE with enhancement plugin try a Build -> Rebuild Project to recompile and enhance all entity beans. Error - property id not found in [name, startDate, comments] for type class org.example.domain.Customer

Sure, my hacky build.gradle is as good as the previously linked build.sbt.

[Rob Bygrave]

> with manual class enhancement I am a

Ebean has a gradle plugin and you'd be expected to use that. https://ebean.io/docs/getting-started/gradle ... so that SO answer should be ignored or corrected really.

> While most (if not all) Play! 1 Framework modules are in package play.modules.*, of course I cou

Ebean ONLY cares about enhancing the application models and classes (@Entity entities, query beans and classes with @Transactional).  We don't care about enhancing any play framework classes.

Now that said, given the link you included it seems that at least some versions of playframework expect the @Entity classes to be in the "play/db/ebean/**" package ... and that is a problem with the IgnoreClassHelper ignoring "play".  So pretty much we need to remove that "play" entry from the IgnoreClassHelper it seems.

> In Play! Framework (version 2) both play-ebean modules use some AntEnhanceTask-like magic to enhance their play.db.ebean package.

Yes. Seems like a sbt plugin would be better than that? 

And yes, having to put entity beans into the play.db.ebean package means we need to change IgnoreClassHelper.

> If I start using io.ebean.Model in the module, then I still have to call those ctClass.addMethod(ctMethod.make(...))?

My take is that adding those static methods is a bad idea as it's a concept that doesn't scale as you add more finder methods and clutters up the entity class. The recommended approach has been to put those methods on the finder object instead.

That is, the finder for Fish for example isn't quite right - https://github.com/PromanSEW/ebean_example/blob/main/app/models/Fish.java#L17

... instead, a class FishFinder extends Finder<Integer,Fish> { ... } ... should be created and used instead. Then we can put the finder methods on that FishFinder and not clutter up the entity class with static finder methods.

The javadoc of io.ebean.Finder shows us the recommended way to use Finder.

https://ebean.io/apidoc/13/io.ebean.api/io/ebean/Finder.html

So my recommendation is to do that instead, adding find methods to the finder type and not adding lots of static methods to the entity bean. IMO adding those static methods was always a bad idea really.

> Just read ebean-orm/ebean#2689: "IllegalStateException: Error trying to create the prototypeEntityBean for class".

It seems like ebean-core is not part of the classpath used during the enhancement.  That means the ebean-agent isn't reading /META-INF/ebean-version.mf resource ... to determine that the enhancement should be for ebean 13.6.0 (which has EntityBeanIntercept as an interface).

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/1bbadf43-cadc-4f17-857f-655d6ca1bdc2n%40googlegroups.com.

[Rob Bygrave]

Looking at the gradle snippet ... in particular:

new AgentManifest(cl)

I'm pretty sure that classLoader (cl) ... doesn't include ebean-core-13.6.0.jar,

- so then /META-INF/ebean-version.mf is not read by the AgentManifest

- so then AgentManifest.getEbeanInternalVersion() will return 0 and not 141

- so then the enhancement thinks ... I'm using an older version of ebean where EntityBeanIntercept is a concrete class (but now with 13.6.0 it's an interface).

A hack to kind of prove that would be to get AgentManifest.getEbeanInternalVersion() to return 141 (and not 0).

Background:

ebean-agent needs to be backwards compatible supporting all versions of ebean from 12.x upwards.  To do this, it detects the actual version of ebean being used by reading that /META-INF/ebean-version.mf resource.

[Rob Bygrave]

ebean-agent - remove "play" from ignore - https://github.com/ebean-orm/ebean-agent/issues/181

[Rob Bygrave]

We might be missing some posts to this thread ... due to spam filter weirdness. I need to check that later.

[Rob Bygrave]

Released 13.6.1 with the ebean-agent no longer ignoring the play subpackages so try that out and see if that gets it all working.

Cheers, Rob.

[Szabolcs Hubai]

Hi Rob,

Thank you very much!

I already started to respond to your mails, but this response shouldn't wait!

Thanks again!

Szabolcs

You received this message because you are subscribed to a topic in the Google Groups "Ebean ORM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ebean/t9xdTXgBseM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ebean+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/CAC%3Dts-Gg8ozU7%2BP8gKrH%3DFkgS34BW%2Bx-PeBzWgVDLU%3DUNkuV_w%40mail.gmail.com.

[Szabolcs Hubai]

Hi Rob!

I just read about Gradle's "Targeting a specific Java version" and Java 9's "--release" feature.

As I would like to create a few intermediate releases (of this Play! 1 Ebean module) supporting Java 6 and Java 8 before I migrate  to Java 11, I thought about how could the Ebean Gradle plugin support earlier versions of Ebean.

Let me ask a few questions regarding your answers below ...

On 5/13/22 05:39, Rob Bygrave wrote:

> with manual class enhancement I am a

Ebean has a gradle plugin and you'd be expected to use that. https://ebean.io/docs/getting-started/gradle ... so that SO answer should be ignored or corrected really.

> While most (if not all) Play! 1 Framework modules are in package play.modules.*, of course I cou

Ebean ONLY cares about enhancing the application models and classes (@Entity entities, query beans and classes with @Transactional).  We don't care about enhancing any play framework classes.

Now that said, given the link you included it seems that at least some versions of playframework expect the @Entity classes to be in the "play/db/ebean/**" package ... and that is a problem with the IgnoreClassHelper ignoring "play".  So pretty much we need to remove that "play" entry from the IgnoreClassHelper it seems.

> In Play! Framework (version 2) both play-ebean modules use some AntEnhanceTask-like magic to enhance their play.db.ebean package.

Yes. Seems like a sbt plugin would be better than that? 

And yes, having to put entity beans into the play.db.ebean package means we need to change IgnoreClassHelper.

> If I start using io.ebean.Model in the module, then I still have to call those ctClass.addMethod(ctMethod.make(...))?

My take is that adding those static methods is a bad idea as it's a concept that doesn't scale as you add more finder methods and clutters up the entity class. The recommended approach has been to put those methods on the finder object instead.

That is, the finder for Fish for example isn't quite right - https://github.com/PromanSEW/ebean_example/blob/main/app/models/Fish.java#L17

... instead, a class FishFinder extends Finder<Integer,Fish> { ... } ... should be created and used instead. Then we can put the finder methods on that FishFinder and not clutter up the entity class with static finder methods.

The javadoc of io.ebean.Finder shows us the recommended way to use Finder.

https://ebean.io/apidoc/13/io.ebean.api/io/ebean/Finder.html

So my recommendation is to do that instead, adding find methods to the finder type and not adding lots of static methods to the entity bean. IMO adding those static methods was always a bad idea really.

> Just read ebean-orm/ebean#2689: "IllegalStateException: Error trying to create the prototypeEntityBean for class".

It seems like ebean-core is not part of the classpath used during the enhancement.  That means the ebean-agent isn't reading /META-INF/ebean-version.mf resource ... to determine that the enhancement should be for ebean 13.6.0 (which has EntityBeanIntercept as an interface).

On 5/13/22 05:52, Rob Bygrave wrote:

Looking at the gradle snippet ... in particular:

new AgentManifest(cl)

I'm pretty sure that classLoader (cl) ... doesn't include ebean-core-13.6.0.jar,

- so then /META-INF/ebean-version.mf is not read by the AgentManifest

- so then AgentManifest.getEbeanInternalVersion() will return 0 and not 141

- so then the enhancement thinks ... I'm using an older version of ebean where EntityBeanIntercept is a concrete class (but now with 13.6.0 it's an interface).

A hack to kind of prove that would be to get AgentManifest.getEbeanInternalVersion() to return 141 (and not 0).

Background:

ebean-agent needs to be backwards compatible supporting all versions of ebean from 12.x upwards.  To do this, it detects the actual version of ebean being used by reading that /META-INF/ebean-version.mf resource.

In the end, I would like to use the latest versions of Ebean and Ebean tools. But until then, I had to use Ebean 4 and Ebean 8 too.

You wrote that ebean-agent (currently 13.x) needs to be backwards compatible to Ebean 12.x. Where do those Ebean internal version (which is now enhancement version, ebean-orm/ebean-agent#182) values like 141 and 128 come from?

How could I find out earlier ebean-agent versions' skew to Ebean?

And how tight is the dependency of the Gradle plugin to ebean-agent? What could go wrong ™ if I downgrade the Gradle plugin's ebean-agent's dependency?

As a fallback I could try again with my snippet.

Thanks,

Szabolcs

[Rob Bygrave]

> Where do those Ebean internal version (which is now enhancement version, ebean-orm/ebean-agent#182) values like 141 and 128 come from?

128 means ebean 12.8 or higher is being used

141 meant ebean 14.1 or higher is being used (we expected to bring that feature in with ebean 14.x except we instead pulled that change into 13.x so that doesn't line up per say but that's ok)

These ebean-version indicators do not exist prior to that. It was introduced when we wanted the new behaviour in ebean-agent based on the new version of ebean but keep the old behaviour for people using older ebean. We won't see those versions or anything like them in prior versions of ebean-agent because in theory we didn't need to as we deemed those changes as more bug fixes. That might not be 100% correct if we really should have treated a bug as a behaviour change instead.

> How could I find out earlier ebean-agent versions' skew to Ebean?

Ebean agent from 11.x to 12.8 are expected to work with ebean 11.x to 12.8.

> And how tight is the dependency of the Gradle plugin to ebean-agent?

It's not tight to the ebean-agent but much more around dealing with Gradle specific lifecycle (when to apply the enhancement) and Gradle specific code to obtain the classpath (used for enhancement).

> What could go wrong ™ if I downgrade the Gradle plugin's ebean-agent's dependency?

I expect that to work.

Cheers, Rob.

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ebean/08c90670-8c18-be7d-038c-ab07b4faf1b9%40gmail.com.

[Szabolcs Hubai]

Hi Rob!

Thank you very much for the answers, they help a lot!

Cheers,

Szabolcs