pax-exam failing to shut down karaf

Benson Margulies

unread,

Oct 17, 2016, 3:39:40 PM10/17/16

to OPS4J

Using pax-exam 4.9.1, we suffer from unpredictable hangs on shutdown. We've seen this with the exam-maven-plugin and when using

@ClassRule
public static PaxExamServer exam = new PaxExamServer();

to launch karaf 4.0.6. Most of the time, everything works as expected, but when it goes wrong, it just gets stuck, sitting there. We're on the verge of implementing a thread waiting for a connection that will just stop the OSGi framework forcibly. That's pretty ugly. Can anyone suggest a debugging strategy? The non-reproducibility of this is, of course, a barrier.

Niclas Hedhman

unread,

Oct 17, 2016, 7:59:06 PM10/17/16

to OPS4J

It doesn't even timeout after 10, 20, 60 minutes?

--
--
------------------
OPS4J - http://www.ops4j.org - op...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "OPS4J" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ops4j+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java

Niclas Hedhman

unread,

Oct 17, 2016, 8:02:33 PM10/17/16

to OPS4J

My guess is that the shutdown is happening before the startup is completed, or something to that effect. Try to capture the relative timing of getting there and the signal to shutdown.

Right now, no suggestion how to solve this, if it is he case.

Benson Margulies

unread,

Oct 17, 2016, 8:43:47 PM10/17/16

to op...@googlegroups.com

On Mon, Oct 17, 2016 at 7:58 PM, Niclas Hedhman <nic...@hedhman.org> wrote:
> It doesn't even timeout after 10, 20, 60 minutes?

Nope. And it's not failing to start: in the exam plugin case, it
starts, the tests run that talk to the Karaf instance, and then it
hangs in stop. And even if I use the Linux kill command to kill the
karaf process, the exam-maven-plugin remains stuck waiting for
something on the shutdown side.

>
> On Tue, Oct 18, 2016 at 3:39 AM, Benson Margulies <ben...@basistech.com>
> wrote:
>>
>> Using pax-exam 4.9.1, we suffer from unpredictable hangs on shutdown.
>> We've seen this with the exam-maven-plugin and when using
>>
>> @ClassRule
>> public static PaxExamServer exam = new PaxExamServer();
>>
>>
>> to launch karaf 4.0.6. Most of the time, everything works as expected, but
>> when it goes wrong, it just gets stuck, sitting there. We're on the verge of
>> implementing a thread waiting for a connection that will just stop the OSGi
>> framework forcibly. That's pretty ugly. Can anyone suggest a debugging
>> strategy? The non-reproducibility of this is, of course, a barrier.
>>
>> --
>> --
>> ------------------
>> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "OPS4J" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to ops4j+un...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>

> --
> --
> ------------------
> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>
> ---

> You received this message because you are subscribed to a topic in the
> Google Groups "OPS4J" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> ops4j+un...@googlegroups.com.

Niclas Hedhman

unread,

Oct 17, 2016, 11:10:19 PM10/17/16

to OPS4J

Then I leave it to the experts... :-(

>> email to ops4j+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
> --
> --
> ------------------
> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>
> ---

> You received this message because you are subscribed to a topic in the
> Google Groups "OPS4J" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> ops4j+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

--
------------------
OPS4J - http://www.ops4j.org - op...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "OPS4J" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ops4j+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Guillaume Nodet

unread,

Oct 18, 2016, 3:59:24 AM10/18/16

to op...@googlegroups.com

Would you be able to get a thread dump of both JVMs when they hang ?

That may be able to give a pointer ...

>> email to ops4j+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
> --
> --
> ------------------
> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>
> ---

> You received this message because you are subscribed to a topic in the
> Google Groups "OPS4J" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> ops4j+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

--
------------------
OPS4J - http://www.ops4j.org - op...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "OPS4J" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ops4j+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------
Guillaume Nodet
------------------------

Red Hat, Open Source Integration

Email: gno...@redhat.com
Web: http://fusesource.com
Blog: http://gnodet.blogspot.com/

Toni Menzel

unread,

Oct 18, 2016, 4:14:19 AM10/18/16

to OPS4J Mailing Lists

I just had this behaviour when the forked process (guessing that you are using a forked container) got stuck loading a dependency (network, proxy issue).

Please send in a trace log and dump.

Toni Menzel

unread,

Oct 18, 2016, 4:16:15 AM10/18/16

to OPS4J Mailing Lists

yuck, just read that its not happening at startup. So your tests already finished running and "just" the shutdown got stuck?

Benson Margulies

unread,

Oct 18, 2016, 6:42:17 AM10/18/16

to op...@googlegroups.com

On Tue, Oct 18, 2016 at 4:16 AM, Toni Menzel <toni....@rebaze.com> wrote:
> yuck, just read that its not happening at startup. So your tests already
> finished running and "just" the shutdown got stuck?

Yes. I will try to get some thread dumps.

>>>> >> email to ops4j+un...@googlegroups.com.

>>>> >> For more options, visit https://groups.google.com/d/optout.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Niclas Hedhman, Software Developer
>>>> > http://zest.apache.org - New Energy for Java
>>>> >
>>>> > --
>>>> > --
>>>> > ------------------
>>>> > OPS4J - http://www.ops4j.org - op...@googlegroups.com
>>>> >
>>>> > ---
>>>> > You received this message because you are subscribed to a topic in the
>>>> > Google Groups "OPS4J" group.
>>>> > To unsubscribe from this topic, visit
>>>> > https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.
>>>> > To unsubscribe from this group and all its topics, send an email to

>>>> > ops4j+un...@googlegroups.com.

>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>> --
>>>> ------------------
>>>> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OPS4J" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to ops4j+un...@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> ------------------------
>>> Guillaume Nodet
>>> ------------------------
>>> Red Hat, Open Source Integration
>>>
>>> Email: gno...@redhat.com
>>> Web: http://fusesource.com
>>> Blog: http://gnodet.blogspot.com/
>>>
>>> --
>>> --
>>> ------------------
>>> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "OPS4J" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to ops4j+un...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> --
> ------------------
> OPS4J - http://www.ops4j.org - op...@googlegroups.com
>
> ---

> You received this message because you are subscribed to a topic in the
> Google Groups "OPS4J" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> ops4j+un...@googlegroups.com.

Benson Margulies

unread,

Oct 20, 2016, 11:26:51 AM10/20/16

to OPS4J

Here are some thread dumps:

karaf:

https://gist.github.com/benson-basis/195078c7b489daffc93b9ceb390d5b44

surefire ForkedBooter:

https://gist.github.com/benson-basis/d4ec93288077e6fcb733f5d08d019409

maven:

https://gist.github.com/benson-basis/9701c735c7e70067db0473714007464d

David Karr

unread,

Oct 24, 2016, 11:26:44 AM10/24/16

to OPS4J

I don't have a lot of data, but we're seeing similar behavior, but integrating with Karaf 4.0.5. If I run a suite of tests, many of them fail with "Address in use" errors, and when the tests are all completed, I see several leftover karaf instances that I have to manually kill. Some of the tests are also failing for other reasons (dependencies, other details). I don't know if it's possible for those errors to cause the shutdown to fail.

Benson Margulies

unread,

Oct 24, 2016, 11:48:15 AM10/24/16

to op...@googlegroups.com

On Mon, Oct 24, 2016 at 11:26 AM, David Karr <davidmic...@gmail.com> wrote:

I don't have a lot of data, but we're seeing similar behavior, but integrating with Karaf 4.0.5. If I run a suite of tests, many of them fail with "Address in use" errors, and when the tests are all completed, I see several leftover karaf instances that I have to manually kill. Some of the tests are also failing for other reasons (dependencies, other details). I don't know if it's possible for those errors to cause the shutdown to fail.

I was hoping that posting the thread dumps would shed light. Our team has given up, and plans to replace pax-exam with launching the app with Docker for testing.

On Monday, October 17, 2016 at 12:39:40 PM UTC-7, Benson Margulies wrote:
Using pax-exam 4.9.1, we suffer from unpredictable hangs on shutdown. We've seen this with the exam-maven-plugin and when using
@ClassRule
public static PaxExamServer exam = new PaxExamServer();
to launch karaf 4.0.6. Most of the time, everything works as expected, but when it goes wrong, it just gets stuck, sitting there. We're on the verge of implementing a thread waiting for a connection that will just stop the OSGi framework forcibly. That's pretty ugly. Can anyone suggest a debugging strategy? The non-reproducibility of this is, of course, a barrier.

--
--
------------------
OPS4J - http://www.ops4j.org - op...@googlegroups.com

---
You received this message because you are subscribed to a topic in the Google Groups "OPS4J" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ops4j/GlyXQeXcwmY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ops4j+unsubscribe@googlegroups.com.

Toni Menzel

unread,

Oct 24, 2016, 12:47:49 PM10/24/16

to op...@googlegroups.com

Cool! Please have a look to provide a pax exam docker container impl. Should not be too hard when starting with a remote test container that is alread there!

You received this message because you are subscribed to the Google Groups "OPS4J" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ops4j+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Toni Menzel

Developer Advocates - The Rebaze Way

www.rebaze.de | www.rebaze.com | @rebazeio

David Karr

unread,

Oct 27, 2016, 12:50:53 PM10/27/16

to OPS4J

On Monday, October 24, 2016 at 8:48:15 AM UTC-7, Benson Margulies wrote:

On Mon, Oct 24, 2016 at 11:26 AM, David Karr <davidmic...@gmail.com> wrote:
I don't have a lot of data, but we're seeing similar behavior, but integrating with Karaf 4.0.5. If I run a suite of tests, many of them fail with "Address in use" errors, and when the tests are all completed, I see several leftover karaf instances that I have to manually kill. Some of the tests are also failing for other reasons (dependencies, other details). I don't know if it's possible for those errors to cause the shutdown to fail.

I was hoping that posting the thread dumps would shed light. Our team has given up, and plans to replace pax-exam with launching the app with Docker for testing.

That isn't a practical option for us. Our desktops are Windows7. The CI server is Linux, but our developers need to be able to run the suite locally.

Benson Margulies

unread,

Oct 27, 2016, 1:19:47 PM10/27/16

to op...@googlegroups.com

Docker works on Windows.

David Karr

unread,

Oct 27, 2016, 1:38:18 PM10/27/16

to OPS4J

On Thursday, October 27, 2016 at 10:19:47 AM UTC-7, Benson Margulies wrote:

On Thu, Oct 27, 2016 at 12:50 PM, David Karr <davidmic...@gmail.com> wrote:

On Monday, October 24, 2016 at 8:48:15 AM UTC-7, Benson Margulies wrote:

On Mon, Oct 24, 2016 at 11:26 AM, David Karr <davidmic...@gmail.com> wrote:
I don't have a lot of data, but we're seeing similar behavior, but integrating with Karaf 4.0.5. If I run a suite of tests, many of them fail with "Address in use" errors, and when the tests are all completed, I see several leftover karaf instances that I have to manually kill. Some of the tests are also failing for other reasons (dependencies, other details). I don't know if it's possible for those errors to cause the shutdown to fail.

I was hoping that posting the thread dumps would shed light. Our team has given up, and plans to replace pax-exam with launching the app with Docker for testing.

That isn't a practical option for us. Our desktops are Windows7. The CI server is Linux, but our developers need to be able to run the suite locally.

Docker works on Windows.

"Docker for Windows requires 64bit Windows 10 Pro, Enterprise and Education"

Reply all

Reply to author

Forward