dotCMS startup issues after assets rsync

[Michael Bishop]

Upon performing an rsync of the assets and db restore from production to a new test instance (clustered environment with 2 app servers and separate db and assets servers), the app servers are no longer able to start.  

It is very odd because there are no errors in dotcms.log, but it looks like there's a lot less going on that what I would find in a healthy startup log (no license sniffing, starting service coyote, etc.)

Attached is the full dotCMS log

[Nathan Keiter]

There’s not much in that log.  Are you sure you should be clustering?  You might want to turn off the cluster config if this is a new standalone server.  Or did you want to add this new server to your cluster?

 

Need to see more of the log file.

 

Nathan

 

Nathan I. Keiter

Lead Network Applications Programmer

DataSystems

Gettysburg College

Campus Box 2453

717-337-6993

 

Measure your success by the level of service you provide.

Si hoc legere scis nimium eruditionis habes.

--
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.
Visit this group at http://groups.google.com/group/dotcms.
For more options, visit https://groups.google.com/d/optout.

[Michael Bishop]

I should mention that this is an upgrade from 2.5.4 > 2.5.7  Initial upgrade consisted of

• cloning all servers
  
• mounted the new nfs server
  
• downloaded and uncompressed 2.5.7 build
• copied plugins to new folder
• changed ip references for the database in config plugin
• changed cluster names and ip references in config plugin
• copied esdata
• started server and reindexed

This worked perfectly, the instance worked fine for testing.  We wanted to refresh assets from the productions servers, so we:

• rsynced the assets
• performed a backup of the production db
• restored on the new db server
• removed esdata on app servers

I started up and reindexed.  The reindex worked fine, but it didn't seem like the rsync worked properly, so I

• resynced overnight
  
• removed esdata on app servers

Upon this attempt to restart, I started to get the issues I was describing before.

I've made gists of the Full 2.5.7 startup log and Full (working) 2.5.4 startup log

There's obviously quite a few differences, but the ones that I'm worried about the most are:

• lines 1-6 aren't on the 2.5.7 log (coyote initialization and catalina startup)
  
• no license sniffing (lines 148-151 on the 2.5.4 log)
• no db connection (lines 185-190)

[Michael Bishop]

Nathan,

Yes, there should be clustering, and there are 2 app servers.  And that is the full log file, I've posted another comment that provides more information that may help

[Michael Bishop]

If helpful, here is the catalina.out for a failed server

[Nathan Keiter]

Remove the broken OSGI plugin or fix the dependency list?  I don’t know if this would kill the entire server, but OSGI can do unexpected stuff.

 

ERROR: Bundle com.dotcms.actionlet.email [8] Error starting/stopping bundle. (org.osgi.framework.BundleException: Unresolved constraint in bundle com.dotcms.actionlet.email [8]: Unable to resolve 8.0: missing requirement [8.0] osgi.wiring.package; (osgi.wiring.package=org.apache.felix.http.api.ExtHttpService))

org.osgi.framework.BundleException: Unresolved constraint in bundle com.dotcms.actionlet.email [8]: Unable to resolve 8.0: missing requirement [8.0] osgi.wiring.package; (osgi.wiring.package=org.apache.felix.http.api.ExtHttpService)

        at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3974)

        at org.apache.felix.framework.Felix.startBundle(Felix.java:2037)

        at org.apache.felix.framework.Felix.setBundleStartLevel(Felix.java:1483)

        at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:334)

        at java.lang.Thread.run(Thread.java:744)

 

 

Nathan

 

Nathan I. Keiter

Lead Network Applications Programmer

DataSystems

Gettysburg College

Campus Box 2453

717-337-6993

 

Measure your success by the level of service you provide.

Si hoc legere scis nimium eruditionis habes.

 

--

[Michael Bishop]

Removing the plugin didn't work either.  I updated the gist with the new catalina.out

[Jason Tesser]

I would start with that yes

[Michael Bishop]

Jason,

I thought that may be an issue too, but removing that plugin (and all osgi plugins at one point) did not solve the issue.

[Michael Bishop]

Figured out the issue guys.  After the rsync, the nfs mount went stale and had to restart the nfs server and remount on the app servers.  I find it odd that there were no errors leading to this, but I'm on 2.5.7, so it's possible that more verbose logging pertaining to this has arose in later versions.