[2.0] Ebean lazy loading

[Timo Hoepfner]

Hi,

I noticed, that lazy loading of attributes wasn't working for Ebean entities when using no explicit getter. I read a bit into the Ebean documentation (Chapter 14 in the user guide). There it says:

14.1 Property Access
With Property Access you MUST have a setter and a getter.
With Property Access ONLY the getter and setter are intercepted. If you use the field in other methods then access to that field is not intercepted/managed (See the getFullName() method as an example).
 
14.2 Field Access
Field access in general is implemented via bytecode Enhancement (Also refered to as Weaving and Transformation).
...
Unlike property access field access does not require the getter or setter to even exist – they can be removed if you want.
Essentially with Field access you can write whatever code you like (within reason) and it will behave as expected (ORM interception occuring as and how you would expect).
 
14.2.2 Ebean – Field vs Property access
... With Ebean you get Field Access when you use Enhancement and you get Property Access when you use Subclassing.

Obviously, play currently doesn't do byte code enhancement of Ebean entities, so explicit getters and setters must be used to get lazy loading and dirty checking functionality. 

Is adding bytecode enhancement already on the radar? Should I file a ticket for it?

As long as it isn't, I would like to suggest changing the play sample code to use Property access.

Timo

[Guillaume Bort]

The sample applications are fine since they don't use lazy loading. I
suggest to not use lazy loading any way.

> --
> You received this message because you are subscribed to the Google Groups
> "play-framework" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/play-framework/-/ZY75hnvMeVAJ.
> To post to this group, send email to play-fr...@googlegroups.com.
> To unsubscribe from this group, send email to
> play-framewor...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/play-framework?hl=en.

--
Guillaume Bort

[Timo Hoepfner]

Lazy loading is not the only reason why one would want to ensure that one either uses property access or bytecode enhancement:

15.1 Why are these techniques needed?
All the Java ORM's that I am aware of need this to support “Lazy Loading” and that includes Ebean.
In addition to that Ebean (and some other ORMs) use these techniques to support “Dirty Checking” / “Optimistic Concurrency Checking”. That is, detecting when an entity bean has been modified (made dirty) and maintaining original values to support Optimistic Concurrency Checking.

From reading the docs (chapter 15), the bytecode enhancement approach would also be more desirable than the subclass approach which is implicitly used when using property access, as the auto-generated sub-classes will come from a different class loader.

Timo

[Timo Hoepfner]

I was wrong, Play already does bytecode enhancement of ebean classes (see PlayCommands.scala arund line 450). Now I just have to find out why the lazy loading doesn't work as expected.

Timo

[Guillaume Bort]

It isn't enabled in Scala class (including the templates). 

On 21 févr. 2012, at 16:24, Timo Hoepfner <timo.h...@gmail.com> wrote:

I was wrong, Play already does bytecode enhancement of ebean classes (see PlayCommands.scala arund line 450). Now I just have to find out why the lazy loading doesn't work as expected.

Timo

--
You received this message because you are subscribed to the Google Groups "play-framework" group.

To view this discussion on the web visit https://groups.google.com/d/msg/play-framework/-/DmjFKJgJnkIJ.

[Timo Hoepfner]

Thanks Guillaume, your comment brought me on the right track.

TL;DR

Why is the play.core.enhancers.PropertiesEnhancer.rewriteAccess enhancement not applied to Scala code?

Long story:

In case someone else is wondering, this is what is happening:

After Play has compiled classes, it will enhance the compiled byte code. This is happening in PostCompile in PlayCommands.scala.

1. It will add getters and setters for fields, if there aren't any in place yet (play.core.enhancers.PropertiesEnhancer.generateAccessors)

2. It will will rewrite classes that directly access fields to use the accessors insted (play.core.enhancers.PropertiesEnhancer.rewriteAccess)

3. If using Ebean, the Ebean enhancer will be applied to the classes configured via application.conf (e.g. "models.*")

The Ebean enhancer will scan classes markes with @Entity and add its own getters and setters which call additional code before the actual value is read from or set to the field.

This additional code enables lazy loading, dirty checking and optimistic concurrency checking.

Then it will (also) replace each direct field access with calls to the Ebean getters/setters.

As play already replaced all occurences of direct field access with a call to getters and setters in step 2, this should effectively only modify the getters and setters created manually or in step 1.

Sample:

Employee employee=Employee.find.byId(1);

Company company=employee.company;

After Step 1&2, this will be converted to

Company company=employee.getCompany();

With Employee#getCompany() beeing something like

@PropertiesEnhancer.GeneratedAccessor

public Company getCompany(){

return this.company;

}

After step 3, the getter will be modified to be something like

@PropertiesEnhancer.GeneratedAccessor

public Company getCompany(){

return _ebean_get_company();

}

protected Company _ebean_get_company() {

  this._ebean_intercept.preGetter("company");

  return this.company;

}

The unexpected behaviour I came across, was that when I did something like employee.company.name in Java, lazy loading worked, but when doing @employee.company.name in a view template it didn't.

Why?

Because the steps 1&2 above are only executed for classes that have Java source code that lives below the "app" folder of the project.

The view templates are converted to Scala sources by play and are put into target/scala-2.9.1/src_managed before compilation. The enhancement is not done for them.

What to do right now?

When using direct field access:

- Java code just works

- Don't rely on lazy loading in Scala code (including views!) 

- Never ever change values on Ebean entities from Scala code as there will be no dirty checking in place

Alternatively use property access:

- The bytecode enhancement will be done after the view templates are already compiled

- That means you cannot use the acessors generated in step 1 of the enhancement process in view templates, as they are not there yet

- So manually add getters and setters to the Ebean entities (tendious) and only use these, at least from Scala code including view templates (ugly)

How to improve that?

Enable step 2 of the enhancement process for the view templates or scala classes in general. E.g. by adding the following to PostCompile in PlayCommands.scala:

val viewClassesDir = new File(srcManaged, "views")

val viewClasses = ( viewClassesDir ** "*.scala").get.map { sourceFile =>

  analysis.relations.products(sourceFile)

}.flatten.distinct

viewClasses.foreach(play.core.enhancers.PropertiesEnhancer.rewriteAccess(classpath, _))

I did only test that very briefly. It appears to work, but I don't know if it has unwanted side effects.

So the big question is, are there reasons why this currently isn't done?

I do think, that lazy loading has its place, especially when combined with Ebeans auto query tuning. In one test I had a page that generated >400 SQL queries per view due to lazy loading. Within a few minutes, Ebean's query tuning converted that to a single query per page view.

Thanks for reading this far,

Timo

[Guillaume Bort]

I try to be conservative regarding byte code modification, especially
for Scala generated bytecode. It is very easy to introduce strange
bugs while doing this.

Using getters access still work and solve the problem. But for me I
think it is better to fetch every needed data during the query, and to
avoid lazy loading during the template rendering.

We should probably update the documentation however.

> --
> You received this message because you are subscribed to the Google Groups
> "play-framework" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/play-framework/-/wPdpB6Hm_4QJ.

>
> To post to this group, send email to play-fr...@googlegroups.com.
> To unsubscribe from this group, send email to
> play-framewor...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/play-framework?hl=en.

--
Guillaume Bort

[Daniel Berndt]

I definitely would welcome an in-depth description of how models get enhanced in the docs because I guess this will be a stumbling block for many people starting to integrate more complex entities into their templates.

[GrailsDeveloper]

Hi Guillaume,

just to get it clear: Play2.0 doesn't use either the subclass approach
nor the enhancing of an Ebean-model? I wondering that this is work,
because it's totally other than I understand the documentation of
Ebean.

Niels

On 22 Feb., 12:18, Guillaume Bort <guillaume.b...@gmail.com> wrote:
> I try to be conservative regarding byte code modification, especially
> for Scala generated bytecode. It is very easy to introduce strange
> bugs while doing this.
>
> Using getters access still work and solve the problem. But for me I
> think it is better to fetch every needed data during the query, and to

> avoidlazyloading during the template rendering.

>
> We should probably update the documentation however.
>
>
>
> On Wed, Feb 22, 2012 at 12:01 PM, Timo Hoepfner <timo.hoepf...@gmail.com> wrote:
> > Thanks Guillaume, your comment brought me on the right track.
>
> > TL;DR
>
> > Why is the play.core.enhancers.PropertiesEnhancer.rewriteAccess enhancement
> > not applied to Scala code?
>
> > Long story:
>
> > In case someone else is wondering, this is what is happening:
>
> > After Play has compiled classes, it will enhance the compiled byte code.
> > This is happening in PostCompile in PlayCommands.scala.
>
> > 1. It will add getters and setters for fields, if there aren't any in place
> > yet (play.core.enhancers.PropertiesEnhancer.generateAccessors)
> > 2. It will will rewrite classes that directly access fields to use the
> > accessors insted (play.core.enhancers.PropertiesEnhancer.rewriteAccess)

> > 3. If usingEbean, theEbeanenhancer will be applied to the classes

> > configured via application.conf (e.g. "models.*")
>

> > TheEbeanenhancer will scan classes markes with @Entity and add its own

> > getters and setters which call additional code before the actual value is
> > read from or set to the field.

> > This additional code enableslazyloading, dirty checking and optimistic

> > concurrency checking.
> > Then it will (also) replace each direct field access with calls to theEbean
> > getters/setters.
> > As play already replaced all occurences of direct field access with a call
> > to getters and setters in step 2, this should effectively only modify the
> > getters and setters created manually or in step 1.
>
> > Sample:
>
> > Employee employee=Employee.find.byId(1);
> > Company company=employee.company;
>
> > After Step 1&2, this will be converted to
>
> > Company company=employee.getCompany();
>
> > With Employee#getCompany() beeing something like
>
> > @PropertiesEnhancer.GeneratedAccessor
> > public Company getCompany(){
> > return this.company;
> > }
>
> > After step 3, the getter will be modified to be something like
>
> > @PropertiesEnhancer.GeneratedAccessor
> > public Company getCompany(){
> > return _ebean_get_company();
> > }
>
> > protected Company _ebean_get_company() {
> >   this._ebean_intercept.preGetter("company");
> >   return this.company;
> > }
>
> > The unexpected behaviour I came across, was that when I did something like

> > employee.company.name in Java,lazyloadingworked, but when doing

> > @employee.company.name in a view template it didn't.
>
> > Why?
>
> > Because the steps 1&2 above are only executed for classes that have Java
> > source code that lives below the "app" folder of the project.
> > The view templates are converted to Scala sources by play and are put into
> > target/scala-2.9.1/src_managed before compilation. The enhancement is not
> > done for them.
>
> > What to do right now?
>
> > When using direct field access:
> > - Java code just works

> > - Don't rely onlazyloadingin Scala code (including views!)
> > - Never ever change values onEbeanentities from Scala code as there will

> > be no dirty checking in place
>
> > Alternatively use property access:
> > - The bytecode enhancement will be done after the view templates are already
> > compiled
> > - That means you cannot use the acessors generated in step 1 of the
> > enhancement process in view templates, as they are not there yet

> > - So manually add getters and setters to theEbeanentities (tendious) and

> > only use these, at least from Scala code including view templates (ugly)
>
> > How to improve that?
>
> > Enable step 2 of the enhancement process for the view templates or scala
> > classes in general. E.g. by adding the following to PostCompile in
> > PlayCommands.scala:
>
> > val viewClassesDir = new File(srcManaged, "views")
> > val viewClasses = ( viewClassesDir ** "*.scala").get.map { sourceFile =>
> >   analysis.relations.products(sourceFile)
> > }.flatten.distinct
> > viewClasses.foreach(play.core.enhancers.PropertiesEnhancer.rewriteAccess(classpath,
> > _))
>
> > I did only test that very briefly. It appears to work, but I don't know if
> > it has unwanted side effects.
>
> > So the big question is, are there reasons why this currently isn't done?
>

> > I do think, thatlazyloadinghas its place, especially when combined with

[Guillaume Bort]

It uses the enhancer model. But as stated in the ebean documentation
the enhancement doesn't work with direct field access. When you use
ebean in Play it will work with direct field access however because
Play will rewrite direct field access to getter/setter. But then it
doesn't work when called from Scala code.

So to be clear, if you want to rely on magic and lazy loading it is
surely better to keep field privates and to use getter and setter to
access values.

[Lev Tverdokhlebov]

Thanks a lot Timo!

This should be written in red at the beginning of the PlayFramework Java manual. I wonder why this is not yet there.

Helped me to solve not reproducible NPE error which appeared only on the server in PROD.