Re: [hippo-community] Lock Wait time out Leading to CMS going down

[Jasper Floor]

Hi,

I only took a glance at your code so this may not be it, but it seems you aren't closing your session every time. The session.logout is only called in the finally of a try/catch which is itself in a try/catch. Unless you are generating exceptions on purpose to enter those blocks the session will not be logged out.

mvg,

Jasper

On Tue, Jan 5, 2016 at 3:01 PM, <rekhaga...@gmail.com> wrote:

Hi,

I have written a CMS Scheduler job (PFA sample code) which fetches data from an external API and creates CMS documents corresponding to the records fetched. It also updates the CMS documents if already present and if any data from external API is different from what is stored in CMS for the corresponding document. It also unpublishes modified CMS documents if they are in Published state.

This Job extends the RepositoryJob from Hippo and the code resides in the CMS WAR.

The code seems to work fine and is serving the purpose.

But the problem is faced when I configure the scheduler to run frequently say every 5 minutes or so and at the same time multiple users access the CMS and try to update the documents randomly.

Things suddenly hang up and CMS becomes unresponsive and after sometime it starts giving proxy error.

If I check the logs then I can find following exceptions: (For details please refer log snapshot in attachment)

05.01.2016 08:01:45 WARN  Hippo JCR Quartz Job Scheduler_Worker-2 [ClusterNode.setRevision:874] Unable to set current revision to 5943.

org.apache.jackrabbit.core.journal.JournalException: Failed to update local revision.

Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction

05.01.2016 08:03:00 WARN  jackrabbit-pool-8 [LockManagerImpl$TimeoutHandlerVisitor.elementVisited:197] Unable to expire the lock. NodeId 00ee2298-79b9-41b7-924f-0cc7bd75441d

javax.jcr.RepositoryException: Unable to remove lock properties.

05.01.2016 08:03:02 WARN  jackrabbit-pool-3 [LockManagerImpl$TimeoutHandlerVisitor.elementVisited:197] Unable to expire the lock. NodeId 00ee2298-79b9-41b7-924f-0cc7bd75441d

javax.jcr.RepositoryException: Unable to remove lock properties.

I have to restart the server in order to get the CMS working.

If I stop the CMS scheduler job some how then CMS works fine.

So I suspect something in the CMS code is wrong and causing this.

Can you please provide some guidance as to what could be causing this problem.

Thanks,

Santanu

--
Hippo Community Group: The place for all discussions and announcements about Hippo CMS (and HST, repository etc. etc.)
 
To post to this group, send email to hippo-c...@googlegroups.com
RSS: https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
---
You received this message because you are subscribed to the Google Groups "Hippo Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hippo-communi...@googlegroups.com.
Visit this group at https://groups.google.com/group/hippo-community.
For more options, visit https://groups.google.com/d/optout.

[Santanu]

Hi Jasper,

My understanding is that the finally block is always executed except for following scenarios in which they may be skipped

1. Another thread calls an interrupt on the current threat

2. JVM crash

3. System.exit()

So not sure if that is the culprit.

Can you may be have a look and check if in the steps for creating  / editing a document , if I am missing / doing anything wrong.?

Is it possible that when my scheduler Job is executing the document workflow and if another users logs in to CMS and tries to makes changes  to the same Document can a RACE condition arise?

Cheers!

SG

[Santanu]

And piece of information which may be useful is that earlier when I used "admin" user and credentials to create session this issue was never reproduced.

In the recent revision I have used "workflowuser" as recommended by Hippo.

So not sure if this has any part to play in this issue.

[Jasper Floor]

On Fri, Jan 8, 2016 at 2:11 PM, Santanu <rekhaga...@gmail.com> wrote:

Hi Jasper,

My understanding is that the finally block is always executed except for following scenarios in which they may be skipped

1. Another thread calls an interrupt on the current threat

2. JVM crash

3. System.exit()

A finally is always called if you enter the try/catch block. If you don't enter it isn't called. Your finally is in a catch block of another try/catch. So if that catch block is never entered you will never close your session.

[Shantanu Ganguly]

Hi Jasper,

So if session doesn't get closed properly won't I get a warning in the CMS logs?

I had observed that the CMS does throw session not closed type of warnings if it is not closed properly, for example in a loop.

Additionally I have observed that the lock is acquired on the hipposched:triggers node. 

For e.g.

Path:

/hippo:configuration/hippo:modules/scheduler/hippo:moduleconfig/AimiaRepositoryJobGroup/RewardJob/hipposched:triggers/RewardSchedule

And if I am able to manually make hippo:locakable under Mixin Types for this Node to false then the exception goes away. Although I am not 100% sure but it worked once.

So not able to pinpoint where should I focus my investigation on.

Any inputs would be welcome.

Thanks,

SG

[Jasper Floor]

To begin with move your finally one block up. That way you are at least guaranteed to close your session.

Also see how long your jobs are taking to run. It could be your job is blocking during execution.

I don't think you need to use synchronized either.

mvg,

Jasper