Slow startup issue in 5.3.8.4

[Xander Steinmann]

Hi all,

I'm upgrading a single dotCMS 5.2.6 to 5.3.8.4 (LTS) and I thought that when dotCMS was starting up that it failed somehow but in fact is was just very slow. See the following loglines:

[15/04/21 08:42:08:649 CEST]  INFO util.AsciiArt:

[15/04/21 08:42:08:649 CEST]  INFO util.AsciiArt:

[15/04/21 08:43:35:058 CEST]  INFO business.CacheLocator: loading cache administrator: com.dotmarketing.business.ChainableCacheAdministratorImpl

[15/04/21 08:43:35:152 CEST]  INFO business.CacheLocator: loading cache administrator: com.dotmarketing.business.ChainableCacheAdministratorImpl

As you see there are 1,5 minutes missing between line 2 and 3. And later these:

[15/04/21 08:54:27:800 CEST]  INFO caffine.CaffineCache: ***     Building Cache : indiciescache, size:8,Concurrency:32

[15/04/21 08:54:27:874 CEST]  INFO startup.StartupTasksExecutor: Finishing upgrade tasks.

[15/04/21 08:55:27:853 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)

[15/04/21 09:05:28:066 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)

[15/04/21 09:15:28:268 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)

[15/04/21 09:25:28:455 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)

[15/04/21 09:34:26:001 CEST]  INFO cluster.ClusterFactory: Cluster topography has changed. This might force a rewire

[15/04/21 09:34:26:001 CEST]  INFO cluster.ClusterFactory: Expecting Servers:[]

(and now dotCMS is starting up)

So dotCMS finally started up but it took more than 45 minutes... When I tested this locally I had no issues. Any ideas why this could happen? In this environment everything is in AWS instead of locally on my PC but then still I wouldn't expect these kind of wait times.

Kind regards,

Xander

[Will Ezell]

Xander:

Was anything happening in the catalina.out while this went on?  If not, you might want to spit a threaddump out into your logs and see what is happening inside the jvm - killall -3 java

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3c053598-e478-4855-b676-446966e223abn%40googlegroups.com.

--

382 NE 191st St #92150

Miami, Florida 33179-3899

Main: 305-900-2001 | Direct: 978.294.9429

wi...@dotcms.com | dotcms.com

   

[Xander Steinmann]

Hi Will,

I have a threaddump of 15000 lines. Is there anything specific that I should look for?

Of the threads there are:

* 124 blocked

* 216 timed_waiting

* 205 waiting

* 26 runnable

Kind regards,

Xander

Op donderdag 15 april 2021 om 14:58:03 UTC+2 schreef Will Ezell:

[Will Ezell]

The ones that are blocked, what are they blocked on?  Try adding the thread dump to an online thread dump analyzer and see which thread is blocking the other threads. 

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/b49339d4-7c44-49fc-a913-a7e7acd49927n%40googlegroups.com.

[Xander Steinmann]

They're all blocked on the getClusterData (I have no cluster by the way)

 java.lang.Thread.State: BLOCKED (on object monitor)
at io.vavr.Lazy.computeValue(Lazy.java:159)
- waiting to lock <0x00000006c602e1f0> (a io.vavr.Lazy)
at io.vavr.Lazy.get(Lazy.java:155)
at com.dotcms.enterprise.cluster.ClusterFactory.getClusterData(SourceFile:147)

Kind regards,

Xander  

Op maandag 19 april 2021 om 15:02:40 UTC+2 schreef Will Ezell:

[Will Ezell]

Shooting in the dark here but is this on a clean box/container?  The system might not have enough entropy to generate a secure key.  Maybe install havgred or something that generates entropy before starting dotcms.

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/dfcd59e4-8f90-4da6-9504-7f518b206e51n%40googlegroups.com.

[Nathan Keiter]

Do you have clustering explicitly turned off in your cluster? I'm wondering if the default "auto-cluster" state is getting stuck.

Nathan I. Keiter | Lead Network Applications Programmer | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>
________________________________
From: dot...@googlegroups.com <dot...@googlegroups.com> on behalf of Xander Steinmann <xan...@isaac.nl>
Sent: Monday, April 19, 2021 9:17 AM
To: dotCMS User Group
Subject: Re: [dotcms] Slow startup issue in 5.3.8.4

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
________________________________

They're all blocked on the getClusterData (I have no cluster by the way)

java.lang.Thread.State: BLOCKED (on object monitor)

at io.vavr.Lazy.computeValue(Lazy.java:159<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazy.java%3a159&c=E,1,yFyIaufYol7NGlJGEfVY4p5BPn7zkHl-LxA0YshzJfSbiVpEAUbkcmvGBrjx92K0lN3RYjXMPKZBLPkVooIhkUwqcNDxtarx2qRMV08hz0dw3MTXbA,,&typo=1&ancr_add=1>)

- waiting to lock <0x00000006c602e1f0> (a io.vavr.Lazy)

at io.vavr.Lazy.get(Lazy.java:155<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazy.java%3a155&c=E,1,uda-KubCrjwBGldt7JWAG8nvpCgoWeJ_uKpKrKpKrPiJDw9Lb5L1xPcweFPeGTJXSE5jpS4c5olaWkXSELZzmCJKWr4-KXYAhbnjG02bRwXfPPfq22h_6K0,&typo=1&ancr_add=1>)

http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,cqraeMV5UcWgHWCf1zTtPWtsda-8EJZojBBf0OET-uFO13Pot7imToLQrPp8GS_MgBtpJsJCa7ft8kbc12uIzUwuQJ82V_9s1DdJ7m8a0ix_mBRvQ5wm&typo=1> - Open Source Java Content Management

---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3c053598-e478-4855-b676-446966e223abn%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2f3c053598-e478-4855-b676-446966e223abn%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,GrgCNHtfT6zEZs_AzZgDUwc089CRLjZOennabUP0Bq7tVAhpezOg8ZRpiioefQCTQW8SYyvVwJ3mWI0V9zvHXpAZu9k4gt3KRUqIUsLwDaCqwvhFcx2GYZY,&typo=1>.


--
[https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci4.googleusercontent.com%2fproxy%2fX9lAIhwzFcyac0u6VckJIJOVKHsVGrb2-WVPneybOshP-_9Qtuowal1-gnnA4giqPrNqM9Bub2QCowjKdInrHRRV7-BqHQ%3ds0-d-e1-ft%23https%3a%2f%2fdotcms.com%2fdA%2f99fe3769-d649%2f256w%2fdotcms.png&c=E,1,aPCf9mGUX22SZG3s9GKKklVCPkSc_58R6U-GPNgbJ81WkSRidZc8C6kdeYUuPkbsJzfaFlGSIL0daJDTAOgJTx2-YfrDEnRbEd-UF36rM5p0UXU,&typo=1]

382 NE 191st St #92150
Miami, Florida 33179-3899
Main: 305-900-2001 | Direct: 978.294.9429

wi...@dotcms.com | dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.dotcms.com%2f&c=E,1,6YEL-A_BH9t6y2z0p1wDkv6cByjbYykE03iWnCzeJzkquTBVilqsibcQxwB6qgw0zQlwMiAkDQNLb-4wJmZPE7MVQ85fFjMau7cIMOOd3DyFLW0v&typo=1>

[https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci6.googleusercontent.com%2fproxy%2fcmkOvj-Lmnc1m6I0I4JByU6E_tqmEgGg8eG-L2qMp1J37Kb_3dGsHgupQ8AFJiIDQcTSaASsKimZecPq9OvckCG7yYFVLipY8Lx0gAM_QZ-fNiRJa9vy%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f11.png&c=E,1,BJBxumsYeUnaaf9tgIuvoElzNjz8ljkkJ4kr421Iyu0w0T77mLujEnrL7MtWrf6dXKPPIdEsaKc0hwW1SD3P_2AtHVPxOOXmeZzIXHjWzNyhS90hMMM,&typo=1]<https://twitter.com/dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci3.googleusercontent.com%2fproxy%2fYUK5J6Z5acoGtktGsQpEFlUYRzYYwV-Ahk4l-P8v91F4lSntXcTs5ufL6v2WS_Jozl7n5VP8WaUN7XlfmKRfpv1siTBGOgAHfYnIASkil6cFkJ7rr0PL%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f14.png&c=E,1,1vXaBw9Gl3_NZtpvxzCQESUOCGSGBceDGfZo1PmBYULv7NR9ACwL62svVqgDzGKDliFgBAUwD6bhDz1jEJ2CsibMGSkWbE6TYwCP5xRj4XMcVvff&typo=1] <http://www.linkedin.com/company/dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci5.googleusercontent.com%2fproxy%2feG1MZ6nIkN1cgtm80hkWS9FS3sv95G3hZ231Mbd5w_MdUOoAyZc8RrY592Sv52YEbcts3Yd4JbFSRYbA-JLvh5dqggSRgefLQzHgLdoBaJwv5wOxmAza%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f15.png&c=E,1,P5ZjzpzA9vO9PeXM5gC4S1xaJVmEEBX23UXQrThBtYSvjdYvRyCwnuZSkvLGBsCBFVdfT6Pv7_zMyb-mz7Zocvf9VzfMx5ngWosSmL0_k4tW&typo=1] <https://plus.google.com/+Dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci6.googleusercontent.com%2fproxy%2f-Mo3gBU-3HQjvZWkduSazsWE0q2xmHIDrMV46crWeKixn16Ov0JCmDK_ch4hdHIQ3OIP3tEQFNd6ZizUPn4jK6E8w-3U9sNxLyFKFBg78SxlDQFJr-ht%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f13.png&c=E,1,Kwt1c6Ddmijep-LiP0vIQOr_qRGUPIAKMiWX9QYcYcjm-Ez74wBOer-2aCUuhoghtvVdndkx5YBBrl4mDOc58uhRh4nB35EFKmHScxAuy4j-7jX3fZsBNXvmAyc,&typo=1] <https://www.facebook.com/dotCMS>

--
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,yohmuFhzMpgPni-WwbZxh8xwDgNZXkOL472_o9HpU3W4-tGPTNXmThkUL24b5ioBnPgCPL0Q1AkG_pgb1hvIqT_gNWPyhxNxMy5d98j83wIZYMhIsk-joMhL&typo=1> - Open Source Java Content Management

---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/b49339d4-7c44-49fc-a913-a7e7acd49927n%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2fb49339d4-7c44-49fc-a913-a7e7acd49927n%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,2975bjWTJvXuG0Pl2cC7dTzJ1HixZuF4GEkar_LJ0zjR0yZDdI-geyFIA-3m1HOLV5cAUc7RQf9rEnu53q6u-KxAZgef9SFoqJVaBV3H_IYv30DBDQ,,&typo=1>.

--
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,c_b3_k6j7HQc1z0dXCD5424wrbmV-w5kUeooJVxJR50nFv15E63HSRd6siRgzbDCxSuqSyWiUlZjK00l4Xdelt1bW4R8o46Q-iOlkztbHyZfFtUz&typo=1> - Open Source Java Content Management

---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com<mailto:dotcms+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/dfcd59e4-8f90-4da6-9504-7f518b206e51n%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2fdfcd59e4-8f90-4da6-9504-7f518b206e51n%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,c5vwfUSZoaW26_GC3hMRLDPM-iDJnC7NeoLz34qfeKiQ40gGFJZUefJ3audlWCLeoe3JR0V2Qxj8Zhp5jD05R_rC2xaN66Wzhd4-geqmao_78sqE8HuabINH7S9Q&typo=1>.

[beautiful sky]

Actually this is the CLS(Cumulative Layout Shift) issue, I think.

How can I fix CLS issue on dotcms?

--

http://dotcms.com - Open Source Java Content Management

---

You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/1618838430366.31291%40gettysburg.edu.

 

[Chris Ling]

Hi Xander,

I'm having the same issue with dotCMS hanging while installing dotCMS. It's actually hanging for 2 hours at the exact same spot as you. Did you ever figure out what the cause was?

Thanks,

Chris

[Will Ezell]

One thing I have seen hang dotCMS on start up is if the host/virtual machine dotCMS is running on does not have enough entropy/randomness.  This basically kills java's attempts to hash or use cryptography.  One way around this is to install something like haveged on the host or virtual machine before starting dotCMS, which ensures there is enough randomness to get java working properly.

Try 

apt-get install haveged 

or

yum install haveged

and see if you get further?

[Chris Ling]

Will, thanks so much for the advice about not enough entropy. The install doesn't hang anymore after installing haveged.

We've been banging our heads for the past weeks trying to figure it out going back and forth with support.

Thanks so much!