Index blocked for some reason

18 views
Skip to first unread message

Xander Steinmann

unread,
Aug 23, 2017, 11:00:34 AM8/23/17
to dotCMS User Group
Hi all,

I have a very strange issue in one of our environments. When running a reindex the reindex is blocked on 0% (I can see records in dist_reindex_journal), but when I copy the database to my local pc and run a reindex then it indexes without issues. What could be causing this reindexing problem? There are no errors in the logs...

It's a dotCMS 3.7.1 with an Oracle DB.

Kind regards,

Xander

Will Ezell

unread,
Aug 23, 2017, 11:12:44 AM8/23/17
to dot...@googlegroups.com
So, very roughly, it seems like you are missing the stored proc that builds the list of records to index. I would make sure that the 1) the stored procedure exists on the remote environ (and all of the stored procedures) and 2) that the db user has the proper permissions on it.  I know my nomenclature is off for Oracle but the ideas are the same.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+unsubscribe@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/7f67f67b-f4d4-4c60-8c32-697dc0fc1b3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--






3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 
305-900-2001 | Direct: 978.294.9429

   

Xander Steinmann

unread,
Aug 23, 2017, 11:18:26 AM8/23/17
to dotCMS User Group
Thank you Will, I will investigate the stored procedures and user permissions.

Kind regards,

Xander

Op woensdag 23 augustus 2017 17:12:44 UTC+2 schreef Will Ezell:
So, very roughly, it seems like you are missing the stored proc that builds the list of records to index. I would make sure that the 1) the stored procedure exists on the remote environ (and all of the stored procedures) and 2) that the db user has the proper permissions on it.  I know my nomenclature is off for Oracle but the ideas are the same.
On Wed, Aug 23, 2017 at 11:00 AM, Xander Steinmann <xan...@isaac.nl> wrote:
Hi all,

I have a very strange issue in one of our environments. When running a reindex the reindex is blocked on 0% (I can see records in dist_reindex_journal), but when I copy the database to my local pc and run a reindex then it indexes without issues. What could be causing this reindexing problem? There are no errors in the logs...

It's a dotCMS 3.7.1 with an Oracle DB.

Kind regards,

Xander

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

To post to this group, send email to dot...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/7f67f67b-f4d4-4c60-8c32-697dc0fc1b3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nathan Keiter

unread,
Aug 23, 2017, 12:41:29 PM8/23/17
to dot...@googlegroups.com

Also try clearing the Indices cache and/or restarting the bad node (clears memory cache for sure.)

 

We just recovered from a recent index issue ourselves.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

Xander Steinmann

unread,
Aug 24, 2017, 6:57:31 AM8/24/17
to dotCMS User Group
Hi Will and Nathan,

I was able to fix the issue for my local db when I added two missing types: reindex_record and reindex_record_list. I'm not sure why they were missing. Is it possible that the database user does not have enough permissions to create them? And are they created on the fly?
After adding them and restarting dotCMS (thanks Nathan :)) the reindex worked fine.

Kind regards,

Xander

Will Ezell

unread,
Aug 24, 2017, 7:58:49 AM8/24/17
to dot...@googlegroups.com
Great Xander - good to hear.  And yes, probably the user did not have enough permissions to create them.  They are only created on initial startup and data load.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+unsubscribe@googlegroups.com.

To post to this group, send email to dot...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Xander Steinmann

unread,
Sep 5, 2017, 2:29:10 AM9/5/17
to dotCMS User Group
Hi Will,

After a lot of back-and-forth a DBA investigated the load_records_to_index and told me that it causes an exception: ORA-06502: PL/SQL: numeric or value error: character to number conversion error.
It's in this line of the load_records_to_index: ;data_ret(data_ret.Last) := reindex_record(dj.id,dj.inode_to_index,dj.ident_to_index,dj.priority,dj.dist_action)

Can this be because of some bad record in the dist_reindex_journal or one of the types it uses? Or maybe the load_records_to_index was not updated after an upgrade?

Kind regards,

Xander 

Jason Tesser

unread,
Sep 5, 2017, 8:05:20 AM9/5/17
to dotCMS User Group

Most likely the db restore is missing something imo


--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

To post to this group, send email to dot...@googlegroups.com.

Xander Steinmann

unread,
Sep 6, 2017, 9:00:14 AM9/6/17
to dotCMS User Group
Is there some class we can override to add logging of some sort? There must be an exception somewhere that is not logged...

Kind regards,

Xander

Nathan Keiter

unread,
Sep 6, 2017, 9:06:20 AM9/6/17
to dot...@googlegroups.com

In our experience, restarting dotCMS usually solves this kind of “errorless” index issue.

 

Our theory is something in the JVM memory relating to index cache gets corrupted occasionally, interfering with the index process. Oddly, dumping the caches won’t resolve it, only a fresh JVM instance.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

From: dot...@googlegroups.com [mailto:dot...@googlegroups.com] On Behalf Of Xander Steinmann
Sent: Wednesday, September 06, 2017 9:00 AM
To: dotCMS User Group
Subject: Re: [dotcms] Index blocked for some reason

 

Is there some class we can override to add logging of some sort? There must be an exception somewhere that is not logged...

 

Kind regards,

 

Xander

--

http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.

Jason Tesser

unread,
Sep 6, 2017, 9:33:59 AM9/6/17
to dot...@googlegroups.com
Well you have the error at the DB.  It is something with the DB.  

The class that calls that function is ESDistributedJournalFactoryImpl 

But I am pretty sure this is an issue with the restore of the DB somehow. 

Xander Steinmann

unread,
Sep 6, 2017, 11:24:57 AM9/6/17
to dotCMS User Group
Hi Jason,

You were right. After adding a lot of logging in the ESDistributedJournalFactoryImpl we noticed that an exception wasn't logged, which was a "character to number conversion error" in LOAD_RECORDS_TO_INDEX line 14. After looking into that we noticed that the order of the parameters in the type REINDEX_RECORD was wrong. We were able to fix this by recreating the type:

create or replace TYPE reindex_record AS OBJECT (

  ID INTEGER,

  INODE_TO_INDEX varchar2(36),

  IDENT_TO_INDEX varchar2(36),

  priority INTEGER,

  dist_action INTEGER

)


This solved our headache. It was probably caused by a database restore or maybe an upgrade? We can't think of anything else.


Thanks all for the help!


Xander

Note: Why is the exception in https://github.com/dotCMS/core/blob/3.7.1/src/com/dotcms/journal/business/ESDistributedJournalFactoryImpl.java line 458 wrapped and not logged?

Reply all
Reply to author
Forward
0 new messages