Re: mysterious scheduled job triggers heap size spike in a few min

[sarahpeng]

I also saw these log entries suggesting communication between the two pool
members:

[07/05/17 21:52:35:278 MDT] INFO quartz.DotJobStore: ClusterManager:
detected 1 failed or restarted instances.
[07/05/17 21:52:35:278 MDT] INFO quartz.DotJobStore: ClusterManager:
Scanning for instance "2"'s failed in-progress jobs.
[07/05/17 21:53:52:501 MDT] INFO business.DotGuavaCacheAdministratorImpl:
Method view: Cluster View is : MergeView::[serverb-5555|4275] [serverb-5555,
servera-3758], subgroups=[[serverb-5555|4274] [serverb-5555],
[servera-3758|4273] [servera-3758]]
[07/05/17 21:53:52:501 MDT] INFO business.DotGuavaCacheAdministratorImpl:
viewAccepted + Cluster View is : MergeView::[serverb-5555|4275]
[serverb-5555, servera-3758], subgroups=[[serverb-5555|4274] [serverb-5555],
[servera-3758|4273] [servera-3758]]
[07/05/17 21:54:09:040 MDT] INFO business.DotGuavaCacheAdministratorImpl:
Method suspect: There is a suspected member : serverb-5555
[07/05/17 21:54:09:040 MDT] INFO business.DotGuavaCacheAdministratorImpl:
suspect + There is a suspected member : serverb-5555
[07/05/17 21:54:09:474 MDT] INFO business.DotGuavaCacheAdministratorImpl:
Method view: Cluster View is : [servera-3758|4276] [servera-3758]
[07/05/17 21:54:09:474 MDT] INFO business.DotGuavaCacheAdministratorImpl:
viewAccepted + Cluster View is : [servera-3758|4276] [servera-3758]
[07/05/17 21:54:35:294 MDT] INFO quartz.DotJobStore: ClusterManager:
detected 1 failed or restarted instances.
[07/05/17 21:54:35:295 MDT] INFO quartz.DotJobStore: ClusterManager:
Scanning for instance "2"'s failed in-progress jobs.
[07/05/17 21:54:35:300 MDT] INFO quartz.DotJobStore: ClusterManager:
......Freed 1 acquired trigger(s).



--
View this message in context: http://forumarchive.dotcms.com/mysterious-scheduled-job-triggers-heap-size-spike-in-a-few-min-tp5733229p5733230.html
Sent from the dotCMS Users Group mailing list archive at Nabble.com.

[sarahpeng]

We have the application code(version 2.5.1) running three years without
issue. Starting from last weekend, both app servers experienced heap usage
spiked every evening at 9pm while cpu usage was minimal. I suspect there is
a scheduled job running somewhere as the log has these:
[07/05/17 22:23:12:819 MDT] INFO quartz.DotJobStore: Handling 1 trigger(s)
that missed their scheduled fire-time.
[07/05/17 22:23:12:829 MDT] ERROR quartz.DotJobStore: ClusterManager: Error
managing cluster: Failure updating scheduler state when checking-in:
ORA-00001: unique constraint (DOTCMS.SYS_C009531) violated

org.quartz.JobPersistenceException: Failure updating scheduler state when
checking-in: ORA-00001: unique constraint (DOTCMS.SYS_C009531) violated
[See nested exception: java.sql.SQLIntegrityConstraintViolationException:
ORA-00001: unique constraint (DOTCMS.SYS_C009531) violated
]
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.clusterCheckIn(JobStoreSupport.java:3373)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3216)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3847)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3883)
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-00001:
unique constraint (DOTCMS.SYS_C009531) violated

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)

but I couldn't find any scheduled job at 9pm. The only place to check
scheduled job is from CMS Admin->site search->view all jobs...where else can
this mysterious job stored?



--
View this message in context: http://forumarchive.dotcms.com/mysterious-scheduled-job-triggers-heap-size-spike-in-a-few-min-tp5733229.html