Frequent Hung/Pending Tasks with 3 and 5 node clusters

[jimmyd]

Hello all,

I wanted to share a bit of knowledge we obtained from a recent troubleshooting marathon with 3 and 5 node CoprHD clusters (deployed from OVF).  We were frequently seeing tasks which would enter a "pending" state and never complete.  This would happen during storage system discovery and in the middle of the provisioning activities.  We traced the issue back to a time sync problem between the nodes.  Even though you define NTP servers as part of the CoprHD setup/install wizard, I found that the NTP services were not actually functioning properly (the coprhd service is called "syncntp", I believe).  As a result, we were seeing the following in the syssvc.log file every 15 minutes:

2016-08-24 06:42:05,654 [pool-37-thread-1]  WARN  AlertsLogger.java (line 75) NTP: [UNCONFIGURED].

2016-08-24 06:57:00,654 [pool-37-thread-1]  WARN  AlertsLogger.java (line 75) NTP: [UNCONFIGURED].

My workaround was to simply install the NTP package on each node using zypper and setup time synchronization outside of CoprHD.  

1. zypper install ntp
   
2. vi /etc/ntp.conf
   
3. Add a line for each of your NTP servers to the end of the file 'server <ntpserver>'
   
4. systemctl enable ntpd
5. systemctl start ntpd

This showed positive results almost immediately.  Our provisioning activities started to complete without issue.  I'm not clear why NTP isn't functioning out of the box.  I'll see if a Jira issue exists on the topic.

Thanks!

Jim DeWaard

[jimmyd]

Also.  Here is an EMC article on the same topic:

https://emcservice.force.com/CustomersPartners/kA2j0000000kCHPCA2