Hibernate prematurely flushing within RequestFactoryServlet while building object

[Eric Andresen]

I have a domain object that has references to several other domain objects, and also uses Hibernate and hibernate validation annotations.

I'm running into an issue where the order of the RF Servlet's loadDomainObject and setProperty calls is causing Hibernate to flush the entity before it is fully constructed, causing validation errors.

The scenario I am seeing is where you set a couple properties and also change a couple of relationships on an object and submit it.

On the server side I see:

  loadDomainObject calls for all of the original object and related objects.

  setProperty calls for some of the changed fields

  setProperty calls for one of the changed relationships

  loadDomainObject call for the changed relationship's new selected object.

That last loadDomainObject call is triggering Hibernate to flush the primary object, because it is doing a hibernate query and our environment doesn't support nested transactions.  

Does anyone know if there is a way to force the servlet to load all of the objects it needs, prior to dirtying any of them?   

Based on http://code.google.com/p/google-web-toolkit/wiki/RequestFactoryMovingParts#Flow it looks like that is what is supposed to happen, but I am seeing the newly selected objects being queried after some of the setProperty calls are already made.

Alternately, is there a way to get a list of the ".with(...)" properties at the time of the find() call?  If I had that, I could initialize the object and detach it from the session and avoid this problem.

Thanks,

Eric

[Thomas Broyer]

Can you trace which methods from the SimpleRequestProcessor or Resolver are calling the loads? that'd greatly help on understanding the problem.

[StefanR]

Hi Eric,

is setting the hibernate flush mode to COMMIT an option? That would mean that no flush is executed before running a query.

Regards,

Stefan.

[Eric Andresen]

Thomas,  here is the trace of what is going on :

The changes involved in this update are:

  Change PrimaryObject.alias to "TEMPLATE"

  Change PrimaryObject.field1 to 10

  Change PrimaryObject.field2 to 0

  Change PrimaryObject.subObjectC from SubObjectC(ID 2) to SubObjectC(ID 1)

Browser request:

{ "F" : "com.my.package.gwt.shared.MyRequestFactory",

  "I" : [ { "O" : "j_yEHCJ9k$WUFHgn__u$bEm7haU=",

        "P" : [ { "S" : "IjEi",

              "T" : "AK9V76Rtc6gMKi9lbRBXpR_Vfsc="

            } ],

        "R" : [ "subobjectb.subobjecta",

            "subobjectc.subobjecta",

            "subobjecta",

            "subobjectb",

            "subobjectc"

          ]

      } ],

  "O" : [ { "O" : "UPDATE",

        "P" : { "alias" : "TEMPLATE",

            "field1" : 10,

            "field2" : 0,

            "subobjecta" : { "S" : "IjMi",

                "T" : "xSFY59dKciM_bQDRhOYtCU94gBw="

              },

            "subobjectb" : { "S" : "IjEi",

                "T" : "HtxC4B1dhRC9YcFi2VSkpbkVpcY="

              },

            "subobjectc" : { "S" : "IjEi",

                "T" : "VJdvF9mvxcVn$KJmLjACkBkrjhw="

              }

          },

        "S" : "IjEi",

        "T" : "AK9V76Rtc6gMKi9lbRBXpR_Vfsc=",

        "V" : "IjEzMjg2Mjc0MzI2Mzki"

      },

      { "O" : "UPDATE",

        "S" : "IjMi",

        "T" : "xSFY59dKciM_bQDRhOYtCU94gBw=",

        "V" : "IjEzMjM0NDUxNDMxODci"

      },

      { "O" : "UPDATE",

        "P" : { "subobjecta" : { "S" : "IjEi",

                "T" : "xSFY59dKciM_bQDRhOYtCU94gBw="

              } },

        "S" : "IjEi",

        "T" : "HtxC4B1dhRC9YcFi2VSkpbkVpcY=",

        "V" : "IjEzMTk1NzUyOTgzNDgi"

      },

      { "O" : "UPDATE",

        "S" : "IjEi",

        "T" : "xSFY59dKciM_bQDRhOYtCU94gBw=",

        "V" : "IjEzMTk1NzUyOTA1NTQi"

      },

      { "O" : "UPDATE",

        "P" : { "subobjecta" : { "S" : "IjEi",

                "T" : "xSFY59dKciM_bQDRhOYtCU94gBw="

              } },

        "S" : "IjIi",

        "T" : "VJdvF9mvxcVn$KJmLjACkBkrjhw=",

        "V" : "IjEzMjg2MjczOTk4NzQi"

      }

    ]

}

Object Graph:

  PrimaryObject

   |

   +--SubObjectA

   +--SubObjectB

   |  |

   |  +--SubObjectA

   |

   +--SubObjectC

      |

      +--SubObjectA  

Order of operations seen in the ServiceLayerDecorator:

1. Load PrimaryObject(ID 1)

2. Load SubObjectA(ID 3)

3. Load SubObjectB(ID 1)

4. Load SubObjectA(ID 1)

5. Load SubObjectC(ID 2)

6. PrimaryObject.setAlias(TEMPLATE)

7. PrimaryObject.setSubObjectA(SubObjectA(ID 3))

8. PrimaryObject.setField1(10)

9. PrimaryObject.setField2(0)

10. PrimaryObject.setSubObjectB(SubObjectB(ID 1))

11. LoadDomainObject SubObjectC(ID 1)  <<-- This load after dirtying the PrimaryObject causes the flush

Stack trace at breakpoint 11:

ServiceLayerCache(ServiceLayerDecorator).loadDomainObject(Class<T>, Object) line: 121

ReflectiveServiceLayer.loadDomainObjects(List<Class<?>>, List<Object>) line: 221

LocatorServiceLayer(ServiceLayerDecorator).loadDomainObjects(List<Class<?>>, List<Object>) line: 126

MyServiceLayerDecorator(ServiceLayerDecorator).loadDomainObjects(List<Class<?>>, List<Object>) line: 126

ServiceLayerCache(ServiceLayerDecorator).loadDomainObjects(List<Class<?>>, List<Object>) line: 126

RequestState.getBeansForIds(List<SimpleProxyId<?>>) line: 267

RequestState.getBeansForPayload(List<IdMessage>) line: 147

RequestState.getBeanForPayload(Splittable) line: 124

EntityCodex.decode(EntitySource, Class<?>, Class<?>, Splittable) line: 101

MySimpleRequestProcessor$1.visitReferenceProperty(String, AutoBean<?>, PropertyContext) line: 547

ProxyAutoBean<T>.traverseProperties(AutoBeanVisitor, AbstractAutoBean$OneShotContext) line: 324

ProxyAutoBean<T>(AbstractAutoBean<T>).traverse(AutoBeanVisitor, AbstractAutoBean$OneShotContext) line: 166

ProxyAutoBean<T>(AbstractAutoBean<T>).accept(AutoBeanVisitor) line: 101

MySimpleRequestProcessor.processOperationMessages(RequestState, RequestMessage) line: 537

MySimpleRequestProcessor.process(RequestMessage, ResponseMessage) line: 210

MySimpleRequestProcessor.process(String) line: 127

MyRequestFactoryServlet.doPost(HttpServletRequest, HttpServletResponse) line: 153

MyRequestFactoryServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 637

MyRequestFactoryServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 717

ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 290

ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206

PageFilter(SiteMeshFilter).obtainContent(ContentProcessor, SiteMeshWebAppContext, HttpServletRequest, HttpServletResponse, FilterChain) line: 129

PageFilter(SiteMeshFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 77

ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 235

ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206

OpenEntityManagerInViewFilter.doFilterInternal(HttpServletRequest, HttpServletResponse, FilterChain) line: 113

OpenEntityManagerInViewFilter(OncePerRequestFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 76

ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 235

ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 368

FilterSecurityInterceptor.invoke(FilterInvocation) line: 109

FilterSecurityInterceptor.doFilter(ServletRequest, ServletResponse, FilterChain) line: 83

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

ExceptionTranslationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 97

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

SessionManagementFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 100

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

AnonymousAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 78

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

SecurityContextHolderAwareRequestFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 54

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

RequestCacheAwareFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 35

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

UsernamePasswordAuthenticationFilter(AbstractAuthenticationProcessingFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 187

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

UsernamePasswordAuthenticationFilter(AbstractAuthenticationProcessingFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 187

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

UsernamePasswordAuthenticationFilter(AbstractAuthenticationProcessingFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 187

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

LogoutFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 105

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

SecurityContextPersistenceFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 79

FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 380

FilterChainProxy.doFilter(ServletRequest, ServletResponse, FilterChain) line: 169

DelegatingFilterProxy.invokeDelegate(Filter, ServletRequest, ServletResponse, FilterChain) line: 237

DelegatingFilterProxy.doFilter(ServletRequest, ServletResponse, FilterChain) line: 167

ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 235

ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206

StandardWrapperValve.invoke(Request, Response) line: 233

StandardContextValve.invoke(Request, Response) line: 191

StandardHostValve.invoke(Request, Response) line: 127

ErrorReportValve.invoke(Request, Response) line: 102

StandardEngineValve.invoke(Request, Response) line: 109

CoyoteAdapter.service(Request, Response) line: 298

Http11Processor.process(Socket) line: 857

Http11Protocol$Http11ConnectionHandler.process(Socket) line: 588

JIoEndpoint$Worker.run() line: 489

Thread.run() line: 662

(The custom Decorator, RequestProcessor, and Servlet are all identical to the 2.4 version except for a bit more error logging)

[Eric Andresen]

Stefan,

  In the past we have investigated FlushMode.COMMIT and FlushMode.MANUAL, but everything we have read about this is that Spring/Hibernate treats it only as a suggestion and not a rule, and it may still flush at inopportune times.  

  We did bring in some consultants from SpringSource and asked them this question a few months ago, but they recommended against changing the FlushMode.  They instead suggested either maintaining a strict order of operations or working with detached objects.

Thanks,

Eric
---------------------------------------------------
Eric Andresen

--
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-web-toolkit/-/iw6cs7bEpbIJ.

To post to this group, send email to google-we...@googlegroups.com.
To unsubscribe from this group, send email to google-web-tool...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.

[Jesse Hutton]

Eric,

By flush you mean the primary object has it's values rest to the
persistent state in the db?

What does your Locator#find() method look like? When using Hibernate
with RequestFactory, you should always use EntityManager.find() (or
the Hibernate Session equivalents) in Locator#find() because that will
prevent hitting the db multiple times for a given entity, which I'm
guessing might be part of the problem.

Jesse

[Eric Andresen]

Jesse,

  Currently my find() methods are calling into some spring-managed DAOs, which in turn run a named query on the database to retrieve the object.  I will try changing these to use the EntityManager directly and see if that helps.

Thanks,

Eric
---------------------------------------------------
Eric Andresen

[Jesse Hutton]

Unless you implement some caching layer yourself, you will most likely
want to use the mentioned methods, and of course a request scoped (or
larger) entity manager or session. Once an entity is in the current
persistence context, a call to em.find() or session.get() will just
pull the value from the context. And, you should note that RF will
call Locator.find() on every entity it proxies back to the client,
including all those that were looked up as list when showing a page of
records, for example. So, for a simple list query, you end up with n +
1 db lookups (the original query and one for each entity in
Locator.find()) if you're not careful.

You might have to also be careful if you have FetchType.LAZY
attributes with such a locator. I do:

if (HibernateProxy.class.isAssignableFrom(clazz)) {
clazz = (Class<? extends VersionableEntity>) clazz.getSuperclass();
}
return entityManagerProvider.get().find(clazz, id);

Another gotcha, uncovered when dealing with RF, is that BeanValidation
can have issues with validating Hibernate proxies if the validation
annotations are applied to fields as opposed to getters. Beware,
always apply to getters!

Jesse

[Eric Andresen]

Jesse,

  I switched my locator to use the EntityManager.find() interface, and it no longer flushes the session, even when it does query the database.    Thanks for the suggestion!  

  I admit I'm a bit of a n00b with Hibernate, do you know what the difference is between letting the EntityManager find the object and calling the javax.persistence.Query.getSingleResult() to find it that causes the flush to happen for the second and not the first?  Something to do with the Spring voodoo or the @Transactional annotations on the DAO perhaps?  I ask because these stray flushes have been a thorn in our heel for ages now.

Thanks,

Eric

[Jesse Hutton]

Eric,

So, I guess you mean session state is flushed to the db? In that case,
perhaps Hibernate can know apriori whether a call to
EntityManager#find(clazz, id) could load entities from the db that
will overwrite modifications in the current session, and that it would
be harder to determine that for arbitrary queries. But, I'm not really
sure. I'd ask on the hibernate mailing list or in #hibernate on
irc.freenode.org.

It might also help to narrow the scope of your @Transactional methods,
because session flushing shouldn't happen without an active
transaction to begin with.

Jesse

> --
> You received this message because you are subscribed to the Google Groups
> "Google Web Toolkit" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-web-toolkit/-/xbC0paEFeecJ.

[Eric Andresen]

Thanks for the information.  I updated all my locators and no longer have the flushing problem.

It still seems like the behavior I'm seeing in the RF Servlet is differing from what is documented at http://code.google.com/p/google-web-toolkit/wiki/RequestFactoryMovingParts#Flow where it says:

• The accumulated state is transmitted to the server, where the following operations take place:
• All domain objects referred to by the payload will be loaded.
• Proxy objects are created for each referenced domain object. These proxies are used later in the request processing to minimuze the amount of data returned to the client.
• All accumulated operations will be applied to the domain objects by traversing properties of the proxies.
• All method invocations in the payload are executed.

It seems like steps 1 and 3 are getting mixed together here.  I don't know if the wiki is an authoritative source or not, so I'm not sure what the actual expected behavior is. If the design is that the objects should be loaded first, we should probably log an issue.  If not, maybe update the wiki.

Thanks,
Eric

[Thomas Broyer]

I believe it's really supposed to happen the way it's documented in the wiki.

Looking at it very closely, your request seems strange though: the PrimaryObject ("S" : "IjEi", "T" : "AK9V76Rtc6gMKi9lbRBXpR_Vfsc="; note: T=proxy type, S=server ID) references a "subobjectc" "S" : "IjEi", "T" : "VJdvF9mvxcVn$KJmLjACkBkrjhw=" which we don't find in the top-level "O" (operations) list. The instance of SubObjectC we find here has a different ID: "S" : "IjIi".

In processOperationMessages, the flow is to load all the objects referenced in the top-level "O" list (if you look into com.google.web.bindery.requestfactory.shared.messages, this is the List<OperationMessage> property of the RequestMessage) and then process their "P" (property map, or using the same terms as in the wiki "all accumulated operations"). When traversing the "subobjectc" property of the loaded PrimaryObject, an EntityCodex.decode() is made that's supposed to simply get the proxy out of the list of proxies previous decoded. In your case though, it's referencing an object that's not yet been seen, so it triggers the loadDomainObject code-path.

This is not supposed to happen as, on the client-side, as soon as you edit() an object, you automatically edit() all referenced objects recursively, and then when building the request, one OperationMessage is made for each edited object (this is done in AbstractRequestContext). So your S=IjEi SubObjectC should have been in the 'editedProxies' list and should have resulted in an OperationMessage. On the server-side, it would then have been loaded at the same time as other domain objects, and then simply reused from the "MySimpleRequestProcessor$1.visitReferenceProperty(String, AutoBean<?>, PropertyContext) line: 547" instead of triggering a loadDomainObject.

Are you doing anything tricky on the client-side? If so, blame yourself, but if you're only doing "standard" things, then please try to make a reduced test case and file an issue.

[Eric Andresen]

I guess "tricky" is a relative term.  My primary editor references SubObjectC using a LeafValueEditor<SubObjectCProxy>, and in this scenario my LeafValueEditor is changing to a different  SubObjectC instance that wasn't included in the original object. 

What I see in the server trace is that in the top-level object, it loads the original instance of subObjectC.  Later in the setProperty methods it loads the newly-selected subObjectC.  

My best guess is that since the new SubObjectC was not "edited" in the original context (it was added in later), that is why it isn't included in that list. It is understandable if the RF servlet doesn't scan through all the operations to find other objects when doing the initial object load.

If this is the expected behavior in this scenario I can accept that, since Jesse's suggested server-side change removed the negative side-effect I was seeing.  I was really just looking for clarification.

Thanks,

Eric

[Thomas Broyer]

On Monday, February 13, 2012 3:34:37 PM UTC+1, Eric Andresen wrote:

I guess "tricky" is a relative term.  My primary editor references SubObjectC using a LeafValueEditor<SubObjectCProxy>, and in this scenario my LeafValueEditor is changing to a different  SubObjectC instance that wasn't included in the original object.

So basically the equivalent of:

ctx.edit(primaryObject); // now, the "original" SubObjectC is edit()ed

primaryObject.setSubobjectc(newSubObjectC); // where newSubObjectC has *not* been edit()ed in this RequestContext

ctx.save(primaryObject).fire();

Nothing "tricky" indeed.

What I see in the server trace is that in the top-level object, it loads the original instance of subObjectC.  Later in the setProperty methods it loads the newly-selected subObjectC.  

My best guess is that since the new SubObjectC was not "edited" in the original context (it was added in later), that is why it isn't included in that list. It is understandable if the RF servlet doesn't scan through all the operations to find other objects when doing the initial object load.

If this is the expected behavior in this scenario I can accept that, since Jesse's suggested server-side change removed the negative side-effect I was seeing.  I was really just looking for clarification.

If I'm right in the above "simplification", could you file an issue? I think the newSubObjectC should be edit()ed when the setter is called. Or alternatively allow un-edited proxies (could reduce the request payload in some circumstances) but then make sure they're loaded before setters are called on the server-side.

[Eric Andresen]

Yes, your simplification is correct.  I'm doing it inside the Editor with getValue/setValue, but I think the result should be the same.

I have logged the issue as http://code.google.com/p/google-web-toolkit/issues/detail?id=7189 .