deploy / unndeploy OOM

45 views
Skip to first unread message

Eduard Drenth

unread,
Dec 22, 2020, 3:57:20 PM12/22/20
to Payara Forum
Dear all,

When I undeploy/deploy an application several times (say 30) payara goes out of memory.

For me not realy a problem, I use this only for testing and dock stack nicely restarts payara.

Does anyone recognize this behaviour?

Regards, Eduard
signature.asc

Hans Pikkemaat

unread,
Dec 23, 2020, 4:37:35 AM12/23/20
to Eduard Drenth, Payara Forum
Hi,

If you deploy an application classes are loaded, initialized, possibly static data is initialized etc etc.
This all costs memory. When you undeploy object references and classes still may stay in memory
causing memory leaks.

My standard working procedure is that when I deploy applications, especially in production, I always
first restart payara so you have a clean starting point.

During development or testing this may not be a requirement. I think its even possible to hot swap classes
so you can develop/test more easily.

gr. Hans


--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/3b0495e68b97fd56277a95cf1bf37484b5c403e7.camel%40gmail.com.

Piotr Zych

unread,
Dec 23, 2020, 4:41:48 AM12/23/20
to Hans Pikkemaat, Eduard Drenth, Payara Forum
We have the same situation in our test env. After tenths of redeployments sometime the OOM will occur. Payara restart each redeployment is not the best scenario for prod, because it causes more unavailability time. Simple redeployment is a lot quicker but there are multiple cases, when after redepoloyment there are some errors, not even OOM: RejectedExecutionException etc...

To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/FE39D49C-88D9-4C67-B8FC-B04B84E09139%40gmail.com.

Vincent Lee

unread,
Dec 23, 2020, 4:44:49 AM12/23/20
to Piotr Zych, Eduard Drenth, Hans Pikkemaat, Payara Forum
Why doesn’t GC take care of the unused references after an undeployment?

Hans Pikkemaat

unread,
Dec 23, 2020, 4:45:47 AM12/23/20
to Piotr Zych, Eduard Drenth, Payara Forum
Yes that’s true it takes more time. In my production environment I have two load balanced payara servers so when I need to deploy a new version I start a new server while the old two keep running.
This way it is not a big problem if starting up takes some extra time. Plus stability is always more important than speed IMHO

Hans Pikkemaat

unread,
Dec 23, 2020, 4:55:03 AM12/23/20
to Vincent Lee, Piotr Zych, Eduard Drenth, Payara Forum
An object is usually only GC-ed when it has no references anymore. If you e.g. have some singleton that initializes some static objects and it doesn’t have a shutdown hook or something like that which will cleanup (remove the references) those objects then they will stay in memory. Any library you use could have such issues.

But class loading is also an issue IIRC, a class is only GC-ed when the classloader has no reference to the class. 
But if there is an object for another class not GC-ed and the class for this object was loaded by a classloader then all other classes
loaded by this classloader are still referenced by the classloader and as such will not be gc-ed.

So one little bug can cause a lot of classed not to be GC-ed.

Piotr Zych

unread,
Dec 23, 2020, 5:03:12 AM12/23/20
to Hans Pikkemaat, Eduard Drenth, Payara Forum
If not all data is replicated through all servers (we for example are not replicating user sessions) then when we switch off single server users are redirected to the other one but they can see that something happened because they have to start from scratch on the application.

Eduard Drenth

unread,
Dec 23, 2020, 5:05:00 AM12/23/20
to Payara Forum
Saw the below was private and not to the list...

-------- Forwarded Message --------
From: Eduard Drenth <eduard...@gmail.com>
To: Steve Millidge (Payara) <steve.m...@payara.fish>
Subject: Re: [payara-forum] deploy / unndeploy OOM
Date: Tue, 22 Dec 2020 23:15:11 +0100

Thanks, I'll have a look some time. The effect I describe I see with all apps I deploy/undeploy, not all of them have the same architecture/libs. If I find a cause that's worth mentioning in general I'll post it.

Regards

-----Original Message-----
From: Steve Millidge (Payara) <steve.m...@payara.fish>
To: Eduard Drenth <eduard...@gmail.com>
Subject: RE: [payara-forum] deploy / unndeploy OOM
Date: Tue, 22 Dec 2020 22:08:26 +0000

Normally this indicates memory leaks due to ClassLoaders not being released. Now we do test Payara for Classloader memory leaks on redeployment regularly but it is also possible for some to slip through. However it could also be caused by the application you are deploying if you keep references to application class instances in thread locals or global caches etc.

 

Simplest way to debug this is deploy the application. Do a Heap Dump and use a memory analyser tool like Eclipse MAT to look for instances of WebAppClassLoader in the dump. Then undeploy you application force a few GCs and then do a Heap Dump again and (assuming your application is a war) you should see the number of instances of WebAppClassLoader decrease by 1. Rinse and repeat. If the number of WebAppClassLoader instances keeps increasing by one on every cycle you have a ClassLoader leak. In that case you need to find the GC Roots that are holding onto these WebAppClassLoader instances. That’s normally not too difficult using MAT but can be fiddly.

 

That’s how we track them down in our testing

 

Steve

signature.asc

Ruslan Synytsky

unread,
Dec 24, 2020, 11:03:12 AM12/24/20
to Payara Forum
Hi Eduard, what is garbage collector do you use? If it's G1, can you try to call Full GC explicitly after the each redeployment and check if it helps? 
Regards 
Ruslan 

Eduard Drenth

unread,
Dec 25, 2020, 2:42:35 AM12/25/20
to Ruslan Synytsky, Payara Forum
G1 is being used, how can I call gc?
signature.asc

Ruslan Synytsky

unread,
Dec 25, 2020, 4:49:51 AM12/25/20
to Eduard Drenth, Payara Forum
--
Ruslan Synytsky
CEO @ Jelastic Multi-Cloud PaaS



Eduard Drenth

unread,
Dec 26, 2020, 5:41:23 AM12/26/20
to r...@jelastic.com, Payara Forum
Calling GC does not help
signature.asc

AleksNo

unread,
Jan 4, 2021, 2:07:48 PM1/4/21
to Payara Forum
Hi, this is a general JavaEE/JakartaEE-problem and not a specific Glassfish/Payara one. Other application servers have this problem too. I guess there is no real fix for this. You could increase the perm space and this error will still apear but not so often.


Cheers

Eduard Drenth

unread,
Jan 5, 2021, 3:08:23 AM1/5/21
to AleksNo, Payara Forum
Good to know this, it is not a big problem for me
signature.asc
Reply all
Reply to author
Forward
0 new messages