Jenkins not starting and killing Tomcat on Solaris VM

[Andrew]

Hi

We are going through an infrastructure migration at the moment; and the latest move was the server with our jenkins on it.

Basic problem is that when you start Tomcat (and Jenkins), it dies.

Remove Jenkins, and tomcat starts fine and you can log in with manager.

Because I cannot log start Jenkins, I cannot log in and enable any additional logging.  I tried to add a groovy script as suggested somewhere as $JENKINS_HOME/init.groovy.d/extralogging.groovy, but that does not seem to do anything.

Looking for ideas on what to try next / what to change to fix this problem.

More details below...

Working configuration is a Solaris 10 VM running on a Solaris 10 Host

VM: % uname -a

SunOS melz*****03 5.10 Generic_150400-09 sun4v sparc SUNW,T5440

Host: %uname -a

SunOS melu*****05 5.10 Generic_150400-09 sun4v sparc SUNW,T5440

Tomcat 7.0.42

Java 1.6.0_38

Jenkins 1.532.2

New (non-working setup) is the (imaged) Solaris 10 VM

• The hosting platform is Oracle VM Sun SPARC (OVMSS).
• The CPU cores, RAM, virtual switch and virtual storage is hard partitioned to LDOMs.  (V3)
• A global zone is an OS running within the LDOM, and its purpose is to soft partition resources to the non-global zones.  
• A branded non-global zone is one that is at leas major version below the release of the global zone. This is the agreed OVMSS architecture, as otherwise the global zone must be of exactly the same version as the non-global zone(s).
• In this case, a single LDOM runs a Solaris 11 global zone for the Solaris 10 branded non-global zones.

Tomcat 7.0.42 and 7.0.57  (ignore the 42, it is a local build package)

Java 1.6.0_38, and tried both of 1.7.0_67 and 1.8.0_11

Jenkins 1.532.2, 1.552 and 1.580.3

I have reinstalled JRE, TC and Jenkins wars, including the latest LTS 1.580.3  (this one gives warning about IllegalStateException: Property HttpOnly which I am ignoring at the moment)

The following is from a JRE 8, Jenkins 1.580.3 run

Catalina Setup

Running with CATALINA_OPTS of [-Denv=sit -Dhostname=melz*****03 -Xms512m -Xmx2G -Djava.awt.headless=true -XX:+MaxFDLimit -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/jenkins/server/current/logs/javaGC.log -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true]

Setting TC Server profile

Using CATALINA_BASE:   /opt/jenkins/server/7.0.57.42

Using CATALINA_HOME:   /opt/jenkins/server/7.0.57.42

Using CATALINA_TMPDIR: /opt/jenkins/server/7.0.57.42/temp

Using JRE_HOME:        /usr/jre1.8.0_11

Using CLASSPATH:       /opt/jenkins/server/7.0.57.42/bin/bootstrap.jar:/opt/jenkins/server/7.0.57.42/bin/tomcat-juli.jar

Using CATALINA_PID:    /opt/jenkins/server/7.0.57.42/temp/catalina.pid

Existing PID file found during start.

Removing/clearing stale PID file.

Tomcat started.

% cat logs/catalina.out

Feb 05, 2015 9:30:17 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/opt/sfw/lib:/usr/sfw/lib:/usr/jdk/packages/lib/sparcv9:/lib/64:/usr/lib/64

Feb 05, 2015 9:30:21 AM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-bio-8280"]

Feb 05, 2015 9:30:21 AM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["ajp-bio-8209"]

Feb 05, 2015 9:30:21 AM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 5981 ms

Feb 05, 2015 9:30:21 AM org.apache.catalina.core.StandardService startInternal

INFO: Starting service Catalina

Feb 05, 2015 9:30:21 AM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/7.0.57

Feb 05, 2015 9:30:21 AM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive /opt/jenkins/server/webapps/jenkins.war

Jenkins home directory: /opt/jenkins/server/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")

Feb 05, 2015 9:31:02 AM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deployment of web application archive /opt/jenkins/server/webapps/jenkins.war has finished in 40,213 ms

Feb 05, 2015 9:31:02 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/jenkins/server/webapps/manager

Feb 05, 2015 9:31:02 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deployment of web application directory /opt/jenkins/server/webapps/manager has finished in 798 ms

Feb 05, 2015 9:31:02 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/jenkins/server/webapps/host-manager

Feb 05, 2015 9:31:05 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deployment of web application directory /opt/jenkins/server/webapps/host-manager has finished in 2,331 ms

Feb 05, 2015 9:31:05 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory /opt/jenkins/server/webapps/ROOT

Feb 05, 2015 9:31:09 AM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deployment of web application directory /opt/jenkins/server/webapps/ROOT has finished in 3,863 ms

Feb 05, 2015 9:31:09 AM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-bio-8280"]

Feb 05, 2015 9:31:09 AM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["ajp-bio-8209"]

Feb 05, 2015 9:31:09 AM org.apache.catalina.startup.Catalina start

INFO: Server startup in 47782 ms

Feb 05, 2015 9:31:13 AM jenkins.InitReactorRunner$1 onAttained

INFO: Started initialization

Feb 05, 2015 9:31:14 AM jenkins.InitReactorRunner$1 onAttained

INFO: Listed all plugins

Feb 05, 2015 9:31:15 AM jenkins.InitReactorRunner$1 onAttained

INFO: Prepared all plugins

Feb 05, 2015 9:31:15 AM jenkins.InitReactorRunner$1 onAttained

INFO: Started all plugins

Feb 05, 2015 9:31:15 AM jenkins.InitReactorRunner$1 onAttained

INFO: Augmented all extensions

Feb 05, 2015 9:31:15 AM jenkins.InitReactorRunner$1 onAttained

INFO: Loaded all jobs

Feb 05, 2015 9:31:31 AM jenkins.model.JenkinsLocationConfiguration updateSecureSessionFlag

WARNING: Failed to set secure cookie flag

java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

...

Caused by: java.lang.IllegalStateException: Property HttpOnly can not be added to SessionCookieConfig for context /jenkins as the context has been initialised

        at org.apache.catalina.core.ApplicationSessionCookieConfig.setHttpOnly(ApplicationSessionCookieConfig.java:107)

        ... 48 more

and shortly thereafter it dies.

I have run truss over the process with

truss -aflr all -o logs/truss.log ./bin/startup.sh

And a 158Mb later the last bit of the log says

28912/71:       stat("/opt/jenkins/server/webapps/jenkins/WEB-INF/classes/com/sun/jna/CallbackResultContext.class", 0xFFFFFFFF468F6C78) Err#2 ENOENT

28912/71:       stat("/opt/jenkins/server/webapps/jenkins/WEB-INF/lib/jna-3.3.0-jenkins-3.jar", 0xFFFFFFFF468F6D68) = 0

28912/71:       stat("/opt/jenkins/server/7.0.57.42/webapps/jenkins/WEB-INF/lib/jna-3.3.0-jenkins-3.jar", 0xFFFFFFFF468F6D68) = 0

28912/71:       lseek(119, 13985, SEEK_SET)                     = 13985

28912/71:       read(119, 0xFFFFFFFF468F4AF0, 30)               = 30

28912/71:          P K0304\n\0\0\b\b\0 jAA - ?CEFBF195 D01\0\0 +02\0\0 '\0\0\0

28912/71:       lseek(119, 14054, SEEK_SET)                     = 14054

28912/71:       read(119, 0xFFFFFFFF468F4D00, 324)              = 324

28912/71:         8590CF NC2 @10C6BF -85DA R A QD1A01E88978289EB ]E38584 SE1A084FB

28912/71:          R V (96 6 ) [E2 c iA2 1F1E003F8 PC6 iA9C4837FB2D9CDCC 7BFF9 fB2

28912/71:         EF1FAF o\0CE q ` AC78E8902 v\rEC19A8 394E6 R MA3 1 CC39D89A5E081

28912/71:         \b ' <96B781F414EF eB5\vA2 .FDD0 W W\fC7CEEF X kC8A0 wA2B1 dA8BA

28912/71:          ~ (FBC9 | $E3811805A4D4DCC813C1 PC4 ~9AE7A2AEA6FE82E1C4F5A2 9 _

28912/71:          $ !9F8582 w D108C84 w w -17 IA0 : QA8E4BDA21D * ^ ^E8E51B9B13A9

28912/71:         BEE2 #A7F5D7FED6 M94C49EECFAE9D4C68F13CED2 v1B06F6 m14 Q b hFEBB

28912/71:         14C3E1 w f10F585F297 r ] - 8AD !9AF4DB :FD >8396FA RA4D1 5B0 A8A

28912/71:          I Q97F2 TB1DA / `EDD3 g h8F19 cD1 [A1 ^10A913 i92 RA6ACBEA2 a c

28912/71:         13C8A2\nB9 2 : U lE59E <CB81 bFB\tDAC3DAAE9489E5CCC6 ^01B9\rC3 v

28912/71:          FD5 >01

28912/71:       write(1, " [ L o a d i n g   c o m".., 42)      = 42

28912/71:       write(1, "   f r o m   f i l e : /".., 82)      = 82

28912/71:       write(1, " ]\n", 2)                             = 2

28912/71:       write(1, " [ L o a d e d   c o m .".., 125)     = 125

28912/71:       lwp_cond_signal(0x101CF4548)                    = 0

28912/62:       lwp_cond_wait(0x101CF4548, 0x101CF4530, 0xFFFFFFFF47DFB200, 0) = 0

28912/62:       stat("/opt/jenkins/server/webapps/jenkins/WEB-INF/classes/org/bouncycastle/asn1/bc/BCObjectIdentifiers.class", 0xFFFFFFFF47DFA308) Err#2 ENOENT

28912/71:       mmap(0xFFFFFFFF3F8C0000, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFFFFFFFF3F8C0000

28912/62:       stat("/opt/jenkins/server/webapps/jenkins/WEB-INF/lib/bcprov-jdk15on-1.47.jar", 0xFFFFFFFF47DFA3F8) = 0

28912/62:       stat("/opt/jenkins/server/7.0.57.42/webapps/jenkins/WEB-INF/lib/bcprov-jdk15on-1.47.jar", 0xFFFFFFFF47DFA3F8) = 0

28912/71:       mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFFFFFFFF3C500000

28912/62:       lseek(73, 0x000304C8, SEEK_SET)                 = 197832

28912/62:       read(73, 0xFFFFFFFF47DF7FC0, 30)                = 30

28912/62:          P K030414\0\b\0\b\0BD 4 { @\0\0\0\0\0\0\0\0\0\0\0\0 2\0\0\0

28912/62:       lseek(73, 0x00030518, SEEK_SET)                 = 197912

28912/62:       read(73, 0xFFFFFFFF47DF81D0, 666)               = 666

28912/62:         8D94 k SD3 @1486DF94 K0314A505 DE4 ~ o111A93 61504 QC01B8A -88 r

28912/62:         F1 V93 4D6 b 'EDD0E08C ? K9D811999F107F8A31CF7B4\v $ t qEC87E7DD

28912/62:          =FB dF7A4 MF3FBCFCF _\0 TD8 2 B 2DA e \91 q U F878CB08C888C N19

28912/62:          ] 2BA % 4 D cDB1202 q U B c \8D k129A ( x & 5 ZD3 h MAB )DAE9 ,

28912/62:          Q8BA484B01A OC4 S d WAFD4 u V9317AD bC1 )B8 K129A17 O07 FB9 l ;

28912/62:          9B6A3 iB199 i eCBA6 -A1AD 6C8 V >19EC8488 g96 -7FB6 * * ; oA0BE

28912/62:         98 5 lC6B9AC E { ]B6 >AFFD s ]D3 SB5F5F0C5 u ] BFB y MD392FEB99E

28912/62:         92D0E59B9FB5 : $ *FB9ABDDC 8 mF7 RE3ACE1CE zC3DF r bCEDFB2 NBD 5

28912/62:         E5 m w8D }FF1DFBC617 C )1A N ^C998FBB6E5 JE8 9 / mB90705 'BF r X

28912/62:          (E6EC03\tD3A583BC b960E1DEBAB e TDCA2AD1815 G U96B7D2 jEDE2B59C

28912/62:         EDB8858F0592 gC4B2 i ) +AB17ED\n { JDC REDB4   n06 113C4 l10 qB6

28912/62:         CBFAFF9FB9C0 ~89 h lFD bFBACDC1DADAFD2B3 > +AAD795F8FD /84D08B d

28912/62:         \b 7\b }84 ~C2\0 a90 0 D18 &8C10 F\t c84 qC204 a92 0 E8812 b841E

28912/62:         B485 0\r =0485D083 T\bD7 qBB05\tCC11E6\t w\b\v84 EC2 ]C212E11EE1

28912/62:          > a99B0 B X % <   < $ < " < & < i eEF81 5C2 3C2F3 V h x J X gFF

28912/62:         E0D5 RCE n9698 u8BBD .C2EC ^D9DDB2 Q /DD p 5FB x8EF28CF28CF11CE1

28912/62:          9CCB39FE714CF 19E03 <07 y0EF11CE7 9C1 sB29AECD1C0 5B4B1CC0181 y

28912/62:          4B21A02B3 'D0F6 "ADC7B0 ~   ME313A8 {C7C8 ~ GFA\bE6118C88C4 p84

28912/62:         0F U #E3 3 202 c C h 4 {8C1DA111F418EF8486EC 1DE\v8D168FB1ED 3 6

28912/62:         049DBE121A M1E cD3 gEC\bF6 x + 4BC {ECFA8C M n04 <C6\vA1D1E0 1B6

28912/62:         84 FA3C7 xE9 3 v05A7EC\t\rEF )AF8586F794 7DF PFDFC05

28912/71:       stat("/dev/zfs", 0xFFFFFFFF468F5EF8)            = 0

28912/71:       fstat(291, 0xFFFFFFFF468F5EF8)                  = 0

28912/71:       sigaction(SIGABRT, 0x00000000, 0xFFFFFFFF468F5B30) = 0

28912/62:       write(1, " [ L o a d i n g   o r g".., 53)      = 53

28912/71:       uucopy(0xFFFFFFFF468F5B40, 0xFFFFFFFF468F5A50, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5A40, 0xFFFFFFFF468F5B40, 16) = 0

28912/62:       write(1, "   f r o m   f i l e : /".., 82)      = 82

28912/71:       uucopy(0xFFFFFFFF468F5B30, 0xFFFFFFFF468F5EB0, 32) = 0

28912/62:       write(1, " ]\n", 2)                             = 2

28912/71:       uucopy(0xFFFFFFFF468F5CC0, 0xFFFFFFFF468F5A60, 32) = 0

28912/71:       uucopy(0xFFFFFFFF468F5A70, 0xFFFFFFFF468F5960, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5950, 0xFFFFFFFF468F5A70, 16) = 0

28912/62:       write(1, " [ L o a d e d   o r g .".., 136)     = 136

28912/71:       sigaction(SIGABRT, 0xFFFFFFFF468F5A60, 0xFFFFFFFF468F5A40) = 0

28912/71:       uucopy(0xFFFFFFFF468F5A50, 0xFFFFFFFF468F5960, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5950, 0xFFFFFFFF468F5A50, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5A40, 0xFFFFFFFF468F5DB0, 32) = 0

28912/71:       sigfillset(0xFFFFFFFF468F5A30)                  = 0

28912/71:       uucopy(0xFFFFFFFF468F5A30, 0xFFFFFFFF468F5970, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5960, 0xFFFFFFFF7E94EC64, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5960, 0xFFFFFFFF468F5880, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5870, 0xFFFFFFFF468F5950, 16) = 0

28912/71:       lwp_sigmask(SIG_SETMASK, 0x00000004, 0x00000000, 0x00000000, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]

28912/71:       uucopy(0xFFFFFFFF468F5950, 0xFFFFFFFF468F5880, 16) = 0

28912/71:       uucopy(0xFFFFFFFF468F5870, 0xFFFFFFFF468F5960, 16) = 0

28912/71:       lwp_kill(71, SIGABRT)                           = 0

28912/71:           Received signal #6, SIGABRT [default]

28912/71:             siginfo: SIGABRT pid=28912 uid=9220 code=-1

28912/34:       lwp_cond_wait(0x1003D6548, 0x1003D6530, 0xFFFFFFFF56CFF7A0, 0) Err#62 ETIME

I've compared with truss on the working environment, and the line 

stat("/dev/zfs", ...) 

in the non working environment is replaced by

ioctl(9, ZFS_IOC_POOL_CONFIGS, 0xA152ABF8)      = 0 

in the working.

I have tried starting TC without Jenkins and then deploying Jenkins - effectively the same outcome - everything dies.

Directory structure

/opt/jenkins/server/

+ ---- jenkins

+ ---- webapps

+ ---- current --> 7.0.57/

+ ---- 7.0.57/

+ ---- + ---- bin

+ ---- + ---- conf

+ ---- + ---- lib

+ ---- + ---- logs

+ ---- + ---- temp

+ ---- + ---- webapps --> /opt/jenkins/server/webapps/

with JENKINS_HOME set to /opt/jenkins/server/jenkins

We do it like that to preserve our jenkins and tomcat configurations between upgrades; however in this case it is moot as it is a vanilla, brand new installation (I have started from scratch again [several times]).

Thanks...Andrew

[Julien Verger]

Hello Andrew,

did you hav any update since this post? I've a same problem to solve.

Regards

[Andrew]

Julien

No, no fix, no update. Ended up retaining on physical kit. :-(