Slow startup issue in 5.3.8.4

23 views
Skip to first unread message

Xander Steinmann

unread,
Apr 15, 2021, 4:00:30 AMApr 15
to dotCMS User Group
Hi all,

I'm upgrading a single dotCMS 5.2.6 to 5.3.8.4 (LTS) and I thought that when dotCMS was starting up that it failed somehow but in fact is was just very slow. See the following loglines:

[15/04/21 08:42:08:649 CEST]  INFO util.AsciiArt:
[15/04/21 08:42:08:649 CEST]  INFO util.AsciiArt:
[15/04/21 08:43:35:058 CEST]  INFO business.CacheLocator: loading cache administrator: com.dotmarketing.business.ChainableCacheAdministratorImpl
[15/04/21 08:43:35:152 CEST]  INFO business.CacheLocator: loading cache administrator: com.dotmarketing.business.ChainableCacheAdministratorImpl

As you see there are 1,5 minutes missing between line 2 and 3. And later these:

[15/04/21 08:54:27:800 CEST]  INFO caffine.CaffineCache: ***     Building Cache : indiciescache, size:8,Concurrency:32
[15/04/21 08:54:27:874 CEST]  INFO startup.StartupTasksExecutor: Finishing upgrade tasks.
[15/04/21 08:55:27:853 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)
[15/04/21 09:05:28:066 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)
[15/04/21 09:15:28:268 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)
[15/04/21 09:25:28:455 CEST]  INFO reindex.ReindexThread: --- ReindexThread Paused (every 3600000 millis)
[15/04/21 09:34:26:001 CEST]  INFO cluster.ClusterFactory: Cluster topography has changed. This might force a rewire
[15/04/21 09:34:26:001 CEST]  INFO cluster.ClusterFactory: Expecting Servers:[]
(and now dotCMS is starting up)

So dotCMS finally started up but it took more than 45 minutes... When I tested this locally I had no issues. Any ideas why this could happen? In this environment everything is in AWS instead of locally on my PC but then still I wouldn't expect these kind of wait times.

Kind regards,

Xander


Will Ezell

unread,
Apr 15, 2021, 8:58:03 AMApr 15
to dot...@googlegroups.com
Xander:

Was anything happening in the catalina.out while this went on?  If not, you might want to spit a threaddump out into your logs and see what is happening inside the jvm - killall -3 java

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3c053598-e478-4855-b676-446966e223abn%40googlegroups.com.


--



382 NE 191st St #92150
Miami, Florida 33179-3899
Main: 
305-900-2001 | Direct: 978.294.9429

   

Xander Steinmann

unread,
Apr 19, 2021, 7:53:12 AMApr 19
to dotCMS User Group
Hi Will,

I have a threaddump of 15000 lines. Is there anything specific that I should look for?
Of the threads there are:
* 124 blocked
* 216 timed_waiting
* 205 waiting
* 26 runnable

Kind regards,

Xander
Op donderdag 15 april 2021 om 14:58:03 UTC+2 schreef Will Ezell:

Will Ezell

unread,
Apr 19, 2021, 9:02:40 AMApr 19
to dot...@googlegroups.com
The ones that are blocked, what are they blocked on?  Try adding the thread dump to an online thread dump analyzer and see which thread is blocking the other threads. 

Xander Steinmann

unread,
Apr 19, 2021, 9:17:39 AMApr 19
to dotCMS User Group
They're all blocked on the getClusterData (I have no cluster by the way)

 java.lang.Thread.State: BLOCKED (on object monitor)
at io.vavr.Lazy.computeValue(Lazy.java:159)
- waiting to lock <0x00000006c602e1f0> (a io.vavr.Lazy)
at io.vavr.Lazy.get(Lazy.java:155)
at com.dotcms.enterprise.cluster.ClusterFactory.getClusterData(SourceFile:147)

Kind regards,

Xander  
Op maandag 19 april 2021 om 15:02:40 UTC+2 schreef Will Ezell:

Will Ezell

unread,
Apr 19, 2021, 9:34:31 AMApr 19
to dot...@googlegroups.com
Shooting in the dark here but is this on a clean box/container?  The system might not have enough entropy to generate a secure key.  Maybe install havgred or something that generates entropy before starting dotcms.

Nathan Keiter

unread,
Apr 19, 2021, 11:29:34 PMApr 19
to dot...@googlegroups.com
Do you have clustering explicitly turned off in your cluster? I'm wondering if the default "auto-cluster" state is getting stuck.

Nathan I. Keiter | Lead Network Applications Programmer | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>
________________________________
From: dot...@googlegroups.com <dot...@googlegroups.com> on behalf of Xander Steinmann <xan...@isaac.nl>
Sent: Monday, April 19, 2021 9:17 AM
To: dotCMS User Group
Subject: Re: [dotcms] Slow startup issue in 5.3.8.4

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
________________________________
They're all blocked on the getClusterData (I have no cluster by the way)

java.lang.Thread.State: BLOCKED (on object monitor)
at io.vavr.Lazy.computeValue(Lazy.java:159<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazy.java%3a159&c=E,1,yFyIaufYol7NGlJGEfVY4p5BPn7zkHl-LxA0YshzJfSbiVpEAUbkcmvGBrjx92K0lN3RYjXMPKZBLPkVooIhkUwqcNDxtarx2qRMV08hz0dw3MTXbA,,&typo=1&ancr_add=1>)
- waiting to lock <0x00000006c602e1f0> (a io.vavr.Lazy)
at io.vavr.Lazy.get(Lazy.java:155<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazy.java%3a155&c=E,1,uda-KubCrjwBGldt7JWAG8nvpCgoWeJ_uKpKrKpKrPiJDw9Lb5L1xPcweFPeGTJXSE5jpS4c5olaWkXSELZzmCJKWr4-KXYAhbnjG02bRwXfPPfq22h_6K0,&typo=1&ancr_add=1>)
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,cqraeMV5UcWgHWCf1zTtPWtsda-8EJZojBBf0OET-uFO13Pot7imToLQrPp8GS_MgBtpJsJCa7ft8kbc12uIzUwuQJ82V_9s1DdJ7m8a0ix_mBRvQ5wm&typo=1> - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3c053598-e478-4855-b676-446966e223abn%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2f3c053598-e478-4855-b676-446966e223abn%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,GrgCNHtfT6zEZs_AzZgDUwc089CRLjZOennabUP0Bq7tVAhpezOg8ZRpiioefQCTQW8SYyvVwJ3mWI0V9zvHXpAZu9k4gt3KRUqIUsLwDaCqwvhFcx2GYZY,&typo=1>.


--
[https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci4.googleusercontent.com%2fproxy%2fX9lAIhwzFcyac0u6VckJIJOVKHsVGrb2-WVPneybOshP-_9Qtuowal1-gnnA4giqPrNqM9Bub2QCowjKdInrHRRV7-BqHQ%3ds0-d-e1-ft%23https%3a%2f%2fdotcms.com%2fdA%2f99fe3769-d649%2f256w%2fdotcms.png&c=E,1,aPCf9mGUX22SZG3s9GKKklVCPkSc_58R6U-GPNgbJ81WkSRidZc8C6kdeYUuPkbsJzfaFlGSIL0daJDTAOgJTx2-YfrDEnRbEd-UF36rM5p0UXU,&typo=1]


382 NE 191st St #92150
Miami, Florida 33179-3899
Main: 305-900-2001 | Direct: 978.294.9429

wi...@dotcms.com | dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.dotcms.com%2f&c=E,1,6YEL-A_BH9t6y2z0p1wDkv6cByjbYykE03iWnCzeJzkquTBVilqsibcQxwB6qgw0zQlwMiAkDQNLb-4wJmZPE7MVQ85fFjMau7cIMOOd3DyFLW0v&typo=1>

[https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci6.googleusercontent.com%2fproxy%2fcmkOvj-Lmnc1m6I0I4JByU6E_tqmEgGg8eG-L2qMp1J37Kb_3dGsHgupQ8AFJiIDQcTSaASsKimZecPq9OvckCG7yYFVLipY8Lx0gAM_QZ-fNiRJa9vy%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f11.png&c=E,1,BJBxumsYeUnaaf9tgIuvoElzNjz8ljkkJ4kr421Iyu0w0T77mLujEnrL7MtWrf6dXKPPIdEsaKc0hwW1SD3P_2AtHVPxOOXmeZzIXHjWzNyhS90hMMM,&typo=1]<https://twitter.com/dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci3.googleusercontent.com%2fproxy%2fYUK5J6Z5acoGtktGsQpEFlUYRzYYwV-Ahk4l-P8v91F4lSntXcTs5ufL6v2WS_Jozl7n5VP8WaUN7XlfmKRfpv1siTBGOgAHfYnIASkil6cFkJ7rr0PL%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f14.png&c=E,1,1vXaBw9Gl3_NZtpvxzCQESUOCGSGBceDGfZo1PmBYULv7NR9ACwL62svVqgDzGKDliFgBAUwD6bhDz1jEJ2CsibMGSkWbE6TYwCP5xRj4XMcVvff&typo=1] <http://www.linkedin.com/company/dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci5.googleusercontent.com%2fproxy%2feG1MZ6nIkN1cgtm80hkWS9FS3sv95G3hZ231Mbd5w_MdUOoAyZc8RrY592Sv52YEbcts3Yd4JbFSRYbA-JLvh5dqggSRgefLQzHgLdoBaJwv5wOxmAza%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f15.png&c=E,1,P5ZjzpzA9vO9PeXM5gC4S1xaJVmEEBX23UXQrThBtYSvjdYvRyCwnuZSkvLGBsCBFVdfT6Pv7_zMyb-mz7Zocvf9VzfMx5ngWosSmL0_k4tW&typo=1] <https://plus.google.com/+Dotcms> [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fci6.googleusercontent.com%2fproxy%2f-Mo3gBU-3HQjvZWkduSazsWE0q2xmHIDrMV46crWeKixn16Ov0JCmDK_ch4hdHIQ3OIP3tEQFNd6ZizUPn4jK6E8w-3U9sNxLyFKFBg78SxlDQFJr-ht%3ds0-d-e1-ft%23https%3a%2f%2forigin.ih.constantcontact.com%2ffs172%2f1109987089110%2fimg%2f13.png&c=E,1,Kwt1c6Ddmijep-LiP0vIQOr_qRGUPIAKMiWX9QYcYcjm-Ez74wBOer-2aCUuhoghtvVdndkx5YBBrl4mDOc58uhRh4nB35EFKmHScxAuy4j-7jX3fZsBNXvmAyc,&typo=1] <https://www.facebook.com/dotCMS>

--
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,yohmuFhzMpgPni-WwbZxh8xwDgNZXkOL472_o9HpU3W4-tGPTNXmThkUL24b5ioBnPgCPL0Q1AkG_pgb1hvIqT_gNWPyhxNxMy5d98j83wIZYMhIsk-joMhL&typo=1> - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/b49339d4-7c44-49fc-a913-a7e7acd49927n%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2fb49339d4-7c44-49fc-a913-a7e7acd49927n%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,2975bjWTJvXuG0Pl2cC7dTzJ1HixZuF4GEkar_LJ0zjR0yZDdI-geyFIA-3m1HOLV5cAUc7RQf9rEnu53q6u-KxAZgef9SFoqJVaBV3H_IYv30DBDQ,,&typo=1>.

--
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,c_b3_k6j7HQc1z0dXCD5424wrbmV-w5kUeooJVxJR50nFv15E63HSRd6siRgzbDCxSuqSyWiUlZjK00l4Xdelt1bW4R8o46Q-iOlkztbHyZfFtUz&typo=1> - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com<mailto:dotcms+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/dfcd59e4-8f90-4da6-9504-7f518b206e51n%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2fdfcd59e4-8f90-4da6-9504-7f518b206e51n%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,c5vwfUSZoaW26_GC3hMRLDPM-iDJnC7NeoLz34qfeKiQ40gGFJZUefJ3audlWCLeoe3JR0V2Qxj8Zhp5jD05R_rC2xaN66Wzhd4-geqmao_78sqE8HuabINH7S9Q&typo=1>.

beautiful sky

unread,
Apr 25, 2021, 3:46:48 PMApr 25
to dot...@googlegroups.com

Actually this is the CLS(Cumulative Layout Shift) issue, I think.

How can I fix CLS issue on dotcms?

--

http://dotcms.com - Open Source Java Content Management

---

You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

Reply all
Reply to author
Forward
0 new messages