[2.3.2] Overnight error.

[Nathan Keiter]

Saturday night into Sunday morning I am seeing thousands of these errors in both server logs (every few milliseconds):

[19/10/13 21:45:28:354 EDT] WARN reindex.ReindexThread: can't dele dist_reindex records. Will try again later java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:769) at java.util.SubList.size(AbstractList.java:645) at com.dotmarketing.common.reindex.ReindexThread.run(ReindexThread.java:144)
Any idea the cause? Our network experts say the network share is fine.

The error went away on each server after restarting each server's dotCMS service.

It did cause the servers' backend performance to slow down. The index is still showing a healthy green on both servers.

Clustered windows environment, dotCMS 2.3.2.

Nathan

Nathan I. Keiter
Lead Network Applications Programmer
DataSystems
Gettysburg College
Campus Box 2453
717-337-6993

Measure your success by the level of service you provide.
Si hoc legere scis nimium eruditionis habes.

[Falzone, Chris]

The only thing I can think is, it was in the middle of reindexing some records, got stuck, and then the reindex thread ran again.  Because 2 threads were trying to modify or access the same data at the same time you get a ConcurrentModificationException.  It might just be a fluke, but if it happens again, take a thread dump before you restart and post that in a github issue for the dotcms guys to take a look and figure out if it is a bug or something they could code to avoid against.  

--
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.
Visit this group at http://groups.google.com/group/dotcms.
For more options, visit https://groups.google.com/groups/opt_out.

--

Christopher Falzone

Interactive Developer

A Q U E N T

Digital, Creative, and Marketing Talent

aquent.com

cfal...@aquent.com

[Nathan Keiter]

Thanks for the feedback Chris. We'll watch for it to happen again and do that if it does.

Nathan

Nathan I. Keiter
Lead Network Applications Programmer
DataSystems
Gettysburg College
Campus Box 2453
717-337-6993

Measure your success by the level of service you provide.
Si hoc legere scis nimium eruditionis habes.

From: <Falzone>, Chris <cfal...@aquent.com<mailto:cfal...@aquent.com>>
Reply-To: "dot...@googlegroups.com<mailto:dot...@googlegroups.com>" <dot...@googlegroups.com<mailto:dot...@googlegroups.com>>
Date: Monday, October 21, 2013 8:18 AM
To: "dot...@googlegroups.com<mailto:dot...@googlegroups.com>" <dot...@googlegroups.com<mailto:dot...@googlegroups.com>>
Subject: Re: [dotcms] [2.3.2] Overnight error.

The only thing I can think is, it was in the middle of reindexing some records, got stuck, and then the reindex thread ran again. Because 2 threads were trying to modify or access the same data at the same time you get a ConcurrentModificationException. It might just be a fluke, but if it happens again, take a thread dump before you restart and post that in a github issue for the dotcms guys to take a look and figure out if it is a bug or something they could code to avoid against.

On Sun, Oct 20, 2013 at 10:56 AM, Nathan Keiter <nke...@gettysburg.edu<mailto:nke...@gettysburg.edu>> wrote:
Saturday night into Sunday morning I am seeing thousands of these errors in both server logs (every few milliseconds):

[19/10/13 21:45:28:354 EDT] WARN reindex.ReindexThread: can't dele dist_reindex records. Will try again later java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:769) at java.util.SubList.size(AbstractList.java:645) at com.dotmarketing.common.reindex.ReindexThread.run(ReindexThread.java:144)
Any idea the cause? Our network experts say the network share is fine.

The error went away on each server after restarting each server's dotCMS service.

It did cause the servers' backend performance to slow down. The index is still showing a healthy green on both servers.

Clustered windows environment, dotCMS 2.3.2.

Nathan

Nathan I. Keiter
Lead Network Applications Programmer
DataSystems
Gettysburg College
Campus Box 2453

717-337-6993<tel:717-337-6993>

Measure your success by the level of service you provide.
Si hoc legere scis nimium eruditionis habes.

--
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com<mailto:dotcms%2Bunsu...@googlegroups.com>.
To post to this group, send email to dot...@googlegroups.com<mailto:dot...@googlegroups.com>.

Visit this group at http://groups.google.com/group/dotcms.
For more options, visit https://groups.google.com/groups/opt_out.



--

Christopher Falzone

Interactive Developer


A Q U E N T

Digital, Creative, and Marketing Talent

aquent.com<http://aquent.com>

cfal...@aquent.com<mailto:cfal...@aquent.com>

--
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com<mailto:dotcms+un...@googlegroups.com>.
To post to this group, send email to dot...@googlegroups.com<mailto:dot...@googlegroups.com>.