QueryDSL in a multithreaded web application - stuck threads

[Grant Lewis]

Hi,

I have not seen this topic discussed in much detail so thought I'd ask for help before we abandon queryDSL and return to JPA criteria objects. We are observing stuck threads immediately in a clustered WebLogic container when deploying our application EAR file with queryDSL. I pasted an example of our usage pattern that is causing the issues. From our perspective it looks fine but it causes threading issues consistently. Any insight on what we need to do to fix  the threading issues would be greatly appreciated? I can't find much information on the subject in the documentation or this group discussion.

thanks,

Grant

Usage Sample from our Application:

public class ClaimAssignmentTileResultHandler extends AbstractResultHandler<ClaimAssignmentTileRequest, AssignmentResult> {

    @Override

    public AssignmentResult getResult(ClaimAssignmentTileRequest claimAssignmentTileRequest) {

        // create the reference date for our calculations

        LocalDate now = LocalDate.now();

        // create the result object

        AssignmentResult assignmentResult = new AssignmentResult();

        // create and execute query for auto assigned claims

        QClaimAssignmentFact autoAssigned = new QClaimAssignmentFact("claimAssignmentFact");

        BooleanExpression assignmentType = autoAssigned.assignmentType.type.eq(ClaimAssignmentTileRequest.AUTO);

        BooleanExpression factDate = createDateBooleanExpression(autoAssigned.factDate, now);

        JPAQuery query = getJpaQuery().from(autoAssigned).where(assignmentType, factDate);

        assignmentResult.setTodayCount(Metric.AUTO_ASSIGNED, query.singleResult(autoAssigned.claim.count()));

        // create and execute query for returned claims

        QClaimAssignmentFact recalled = new QClaimAssignmentFact("claimAssignmentFact");

        assignmentType = recalled.assignmentType.type.eq(ClaimAssignmentTileRequest.RECALL);

        factDate = createDateBooleanExpression(recalled.factDate, now);

        query = getJpaQuery().from(recalled).where(assignmentType, factDate);

        assignmentResult.setTodayCount(Metric.RETURNED, query.singleResult(recalled.claim.count()));

        // return result

        return assignmentResult;

    }

}

[Timo Westkämper]

Hi.

It might be related to classloading. Here is a troubleshooting entry for it http://www.querydsl.com/static/querydsl/3.6.2/reference/html/ch04s02.html

Did you investigate where it gets stuck?

Br,

Timo

--
You received this message because you are subscribed to the Google Groups "Querydsl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to querydsl+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grant Lewis]

Here is a thread dump...

[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f19f800f800 nid=0x4f16 in Object.wait() [0x00007f19de5d8000]
  java.lang.Thread.State: RUNNABLE
at com.mysema.query.types.path.BeanPath.createNumber(BeanPath.java:257)
at gov.va.vba.vbms.analytics.etl.model.QClaimAssignmentTypeDim.<init>(QClaimAssignmentTypeDim.java:22)
at gov.va.vba.vbms.analytics.etl.model.QClaimAssignmentTypeDim.<clinit>(QClaimAssignmentTypeDim.java:20)
at gov.va.vba.vbms.analytics.etl.model.QClaimAssignmentFact.<init>(QClaimAssignmentFact.java:57)
at gov.va.vba.vbms.analytics.etl.model.QClaimAssignmentFact.<init>(QClaimAssignmentFact.java:40)
at gov.va.vba.vbms.analytics.etl.model.QClaimAssignmentFact.<clinit>(QClaimAssignmentFact.java:23)
at gov.va.vba.vbms.analytics.nwq.ClaimAssignmentTileResultHandler.getResult(ClaimAssignmentTileResultHandler.java:22)
at gov.va.vba.vbms.analytics.nwq.DashboardServiceImpl.getAssignmentResult(DashboardServiceImpl.java:45)
at gov.va.vba.vbms.analytics.nwq.DashboardServiceEjb.getAssignmentResult(DashboardServiceEjb.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:100)
at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:117)

    "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f19e4006000 nid=0x4f13 in Object.wait() [0x00007f19eda68000]
  java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:140)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1057)
at java.lang.reflect.Field.getFieldAccessor(Field.java:1038)
at java.lang.reflect.Field.get(Field.java:379)
at com.mysema.query.types.OperatorImpl.<clinit>(OperatorImpl.java:42)
at com.mysema.query.types.Ops$DateTimeOps.<clinit>(Ops.java:253)
at com.mysema.query.types.expr.DateTimeExpression.currentDate(DateTimeExpression.java:58)
at com.mysema.query.types.expr.DateTimeExpression.<clinit>(DateTimeExpression.java:37)
at com.mysema.query.types.path.BeanPath.createDateTime(BeanPath.java:212)
at gov.va.vba.vbms.analytics.etl.model.QDeferralEventFact.<init>(QDeferralEventFact.java:33)
at gov.va.vba.vbms.analytics.etl.model.QDeferralEventFact.<init>(QDeferralEventFact.java:42)
at gov.va.vba.vbms.analytics.etl.model.QDeferralEventFact.<clinit>(QDeferralEventFact.java:23)
at gov.va.vba.vbms.analytics.nwq.DeferralTileResultHandler.getResult(DeferralTileResultHandler.java:22)
at gov.va.vba.vbms.analytics.nwq.DashboardServiceImpl.getDeferralResult(DashboardServiceImpl.java:98)
at gov.va.vba.vbms.analytics.nwq.DashboardServiceEjb.getDeferralResult(DashboardServiceEjb.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)

On Wednesday, March 4, 2015 at 3:44:08 PM UTC-5, timowest wrote:

Hi.

It might be related to classloading. Here is a troubleshooting entry for it http://www.querydsl.com/static/querydsl/3.6.2/reference/html/ch04s02.html

Did you investigate where it gets stuck?

Br,

Timo

[Timo Westkämper]

Hi.

You should definitely try the ClassPathUtils based initialization. It might solve the issue.

Timo

[Grant Lewis]

I think we tried to use ClassPathUtils and ran into some problems with it from an EAR file. I've asked one of my lead engineers to join our discussion and comment on his findings with trying to use ClassPathUtils. I have someone else writing up his findings on the problem. Seems related to some of the static initialization code we are finding.

Grant

On Wednesday, March 4, 2015 at 3:57:31 PM UTC-5, timowest wrote:

Hi.

You should definitely try the ClassPathUtils based initialization. It might solve the issue.

Timo

[Grant Lewis]

Okay, so we know the problem with ClassPathUtils. It will solve our problem with one little issue. When calling classLoader.getResources(packagePath) WebLogic returns "zip" as the protocol and your code only handles "jar" as of right now and will not call the scanJar method.

Grant

[Timo Westkämper]

Hi.

Could you open a ticket for the zip protocol support in ClassPathUtils?

Br,

Timo

[Grant Lewis]

yeah, we can do that. In the meantime we are just going to roll our own class initialization code until you can fix it.

On Wednesday, March 4, 2015 at 4:36:16 PM UTC-5, timowest wrote:

Hi.

Could you open a ticket for the zip protocol support in ClassPathUtils?

Br,

Timo

[Timo Westkämper]

Alternatively enabling entityAccessors like documented here should also make the class initialization safer http://www.querydsl.com/static/querydsl/3.6.2/reference/html/ch03s03.html#d0e2233

[Grant Lewis]

Cool, that might actually help us. I opened an issue. Issue #1235.

On Wednesday, March 4, 2015 at 4:42:51 PM UTC-5, timowest wrote:

Alternatively enabling entityAccessors like documented here should also make the class initialization safer http://www.querydsl.com/static/querydsl/3.6.2/reference/html/ch03s03.html#d0e2233

[Grant Lewis]

For reference purposes I thought I'd explain in a little more detail how the problem manifests itself w/o the single threaded class initialization. Basically we have a web page with a series of user interface tiles. Each tile maps to a result handler on the backend with similar logic to what I pasted in my original post. The tiles are loaded concurrently by ajax requests resulting in multiple threads to the backend handlers. The static initialization code in the framework is getting called concurrently by multiple threads resulting in deadlocks.

Grant

[Grant Lewis]

We are still observing threading issues and stuck threads even after implementing your suggestions. We implemented a static block in our abstract handler class that replicates the code in your ClassPathUtils with some modifications to support the peculiarities of WebLogic. I pasted that code below. Even with that code, if we attempt to instantiate a Q-type outside the static block in one of our concrete handlers instead of using the Q-type's static default variable we consistently observe stuck threads. Not sure what to make of this and hoping you can shed some light on the matter. Based on our observations it is not safe to instantiate Q-types in a multithreaded environment unless all Q-types are initialized in a single thread as statics which is not feasible in our application.

public abstract class AbstractResultHandler<T extends FactRequest, V extends FactResult> implements FactResultHandler<T, V> {

    static {

        try {

            QueryDslConfig.scanPackage(

                    Thread.currentThread().getContextClassLoader(),

                    "gov.va.vba.vbms.analytics.etl.model");

        } catch (IOException e) {

        }

    }

    @PersistenceContext(unitName = "...")

    protected EntityManager entityManager;

    public JPAQuery getJpaQuery() {

        return new JPAQuery(entityManager);

    }

}

[Timo Westkämper]

Hi.

Are you aware of this issue https://github.com/querydsl/querydsl/issues/1237

There is already a pending PR that you could maybe test. I will add a few more tested in the weekend.

At least the OperatorImpl initialization which was also seen in one of your thread should be safe now.

Br,

Timo

[Grant Lewis]

I did not see that issue but I can confirm we are observing deadlocks in the same code of OperationImpl. I'm not a big fan of our temporary solution below but we opted for a synchronized factory method for q-types in our abstract handler. I eagerly await 4.0 if it fixes this issue.

public synchronized final <QType> QType getQueryType(Class<QType> qTypeClass, String variable) {

        try {

            Constructor<QType> qTypeConstructor = qTypeClass.getConstructor(String.class);

            return qTypeConstructor.newInstance(variable);

        } catch (NoSuchMethodException |

                 InstantiationException |

                 IllegalAccessException |

                 InvocationTargetException e) {

            return null;

        }

}

On Friday, March 6, 2015 at 2:44:18 PM UTC-5, timowest wrote:

Hi.

Are you aware of this issue https://github.com/querydsl/querydsl/issues/1237

There is already a pending PR that you could maybe test. I will add a few more tested in the weekend.

At least the OperatorImpl initialization which was also seen in one of your thread should be safe now.

Br,

Timo

[Timo Westkämper]

Hi.

4.0.0 definitely fixes the OperatorImpl/Ops loading issue, but also the next release in the 3.* line: 3.6.3.

Timo