APR pools and thread safety

[Arsen Chaloyan]

I want to pay your attention to issue-29, where it has been turned out that APR pools are actually not thread safe.
This is a major issue, which affects project usability and may cause client and server stacks to crash.

There could be several workarounds, but the only universal one is to make APR pools thread safe themselves.
The patch introduced in comment-18 fixed the issue and allowed the reporter to reach a milestone of 2 million sessions under stress tests he made.

Several patches over the latest release of Sofia-SIP and now yet another one for APR pools bring me to conclusion to provide and mandate the use of custom, patched packages of the libraries UniMRCP uses. So the question is what would be the most convenient way.

1. Provide one dependency pack, which contains all the required and already patched libraries. So the user will obtain UniMRCP source and an appropriate dependency pack for it.
2. Commit all the dependencies with patches into UniMRCP tree and ship them with UniMRCP.
3. Leave everything as is and just warn the users to apply the patches.

Also I'm thinking about the version of APR to use and recommend. Actually this patch with minor modifications can be applied over any version of APR including 1.2.x and 1.3.x.
I used APR-1.2.12 before, but recently have migrated to APR-1.3.3 and APR-Util-13.4. However the latest ones are APR-1.3.5 and APR-1.3.7.

Please feel free to share you thoughts over the issue and the topics above.

--
Arsen Chaloyan
The author of UniMRCP
http://www.unimrcp.org

[Kamil Shakirov]

Hi Arsen,

On Thu, 2009-07-16 at 13:11 +0500, Arsen Chaloyan wrote:
> I want to pay your attention to issue-29, where it has been turned out
> that APR pools are actually not thread safe.
> This is a major issue, which affects project usability and may cause
> client and server stacks to crash.
>
> There could be several workarounds, but the only universal one is to
> make APR pools thread safe themselves.
> The patch introduced in comment-18 fixed the issue and allowed the
> reporter to reach a milestone of 2 million sessions under stress tests
> he made.

Thanks a lot for fixing this issue. My IVR runs not more than 1000 ports
(APR is used as the foundation framework) and hasn't been affected by
this issue yet. I am going to try those patches out.

> Several patches over the latest release of Sofia-SIP and now yet
> another one for APR pools bring me to conclusion to provide and
> mandate the use of custom, patched packages of the libraries UniMRCP
> uses. So the question is what would be the most convenient way.
>
> 1. Provide one dependency pack, which contains all the required and
> already patched libraries. So the user will obtain UniMRCP source and
> an appropriate dependency pack for it.
> 2. Commit all the dependencies with patches into UniMRCP tree and ship
> them with UniMRCP.
> 3. Leave everything as is and just warn the users to apply the
> patches.

Vote for 3. But all changes to UniMRCP itself should be commited.

> Also I'm thinking about the version of APR to use and recommend.
> Actually this patch with minor modifications can be applied over any
> version of APR including 1.2.x and 1.3.x.
> I used APR-1.2.12 before, but recently have migrated to APR-1.3.3 and
> APR-Util-13.4. However the latest ones are APR-1.3.5 and APR-1.3.7.

I am using the latest APR libraries without problems.

> Please feel free to share you thoughts over the issue and the topics
> above.
>
> --
> Arsen Chaloyan
> The author of UniMRCP
> http://www.unimrcp.org
>
> >

--
--wbr.

[Arsen Chaloyan]

Hi Kamil,
Thanks for the response.

Actually this issue is very hard to reproduce. It caused me a lot of troubles till I finally understood what was going on.
I've got only a few sporadic crashes under my stress tests on Windows, while it has never happened on Linux.
Fortunately cpsoares (the reporter) didn't only report the issue, but also helped in further analyzes and testing.
Anyway, if you look into apr_palloc(), it'll be clear memory allocation is not thread safe.

[Vlad Socaciu]

Hi Arsen,

If my perception is correct, you made changes to Sofia-SIP and APR which were not included in those projects by the people who maintain them.

Is this true?

If this is the case, then the ideal solution would be to determine those people to include those patches in the source code for the respective projects.

If this is not possible, then certainly we are facing a tedious but not uncommon situation.

My guess is that it would be very convenient for the UniMRCP users to have the already patched dependencies available. Of course, this puts all the burden on you again, but you would have to make the effort of patching them anyway for your own purpose :-(. So I would definitely favor options 1 or 2 and let you decide which one is more convenient to you.

Sorry in case I misunderstood the issue.

On a side note, for new users, we should probably mention somewhere that the APR suit should also include APR-iconv, and also that the various apr_dbd_xxx... projects will not build unless one also has the corresponding database software package.

[Arsen Chaloyan]

Hi Vlad,
Thanks for the response and see below.

On Fri, Jul 17, 2009 at 5:08 AM, Vlad Socaciu <cura...@gmail.com> wrote:

Hi Arsen,

If my perception is correct, you made changes to Sofia-SIP and APR which were not included in those projects by the people who maintain them.

Is this true?

Partially correct. However there are two typical cases here:
1. We identified a typical bug in Sofia-SIP, reported it to the authors, this bug has been fixed and included in Sofia-SIP's repository. However there has been no official release yet. Meantime we should use the latest release plus the mentioned patch. Still not so convenient as I believe.
2. There is a patch submitted to Sofia-SIP, which makes SIP TRYING configurable (optional) from user space. I think it's helpful, taking into account there are still user agents (MRCP clients), which don't support SIP TRYING. However as far as I know this patch has not been accepted/included yet, probably the author don't meet the same problem and we can do nothing about.

If this is the case, then the ideal solution would be to determine those people to include those patches in the source code for the respective projects.

There is no guarantee it always works.

If this is not possible, then certainly we are facing a tedious but not uncommon situation.

I think, there is some probability we'll meet similar issues as we go along. The probability is not high, but exists.

My guess is that it would be very convenient for the UniMRCP users to have the already patched dependencies available. Of course, this puts all the burden on you again, but you would have to make the effort of patching them anyway for your own purpose :-(. So I would definitely favor options 1 or 2 and let you decide which one is more convenient to you.

I haven't got enough responses yet, but I assume, the majority of users would be primary interested in viable MRCP solution, they don't care much what is behind. From other hand, there are projects, where APR is used internally and it would not be convinient for them if I mandate to use only the version provided by UniMRCP.
BTW, FreeSWITCH project seems to meet the same problem a few months ago and they have already patched bundled with FreeSWITCH APR library, while I have just applied the same patch to the newer version of APR we use.

So most probably I'll end up with some sort of mix of option 1 and 3.
I'll provide already patched and ready to use with UniMRCP dependency pack. This will be the recommended way to go for new users, while for the advanced users the exact patches over the oficially released versions of libraries will be separately available to apply.
 

Sorry in case I misunderstood the issue.

On a side note, for new users, we should probably mention somewhere that the APR suit should also include APR-iconv, and also that the various apr_dbd_xxx... projects will not build unless one also has the corresponding database software package.

The dependency pack should contain only the stuff UniMRCP uses, those options (APR-iconv, apr_dbd) should not be included for now.

On Thu, Jul 16, 2009 at 1:11 AM, Arsen Chaloyan <acha...@gmail.com> wrote:

I want to pay your attention to issue-29, where it has been turned out that APR pools are actually not thread safe.
This is a major issue, which affects project usability and may cause client and server stacks to crash.

There could be several workarounds, but the only universal one is to make APR pools thread safe themselves.
The patch introduced in comment-18 fixed the issue and allowed the reporter to reach a milestone of 2 million sessions under stress tests he made.

Several patches over the latest release of Sofia-SIP and now yet another one for APR pools bring me to conclusion to provide and mandate the use of custom, patched packages of the libraries UniMRCP uses. So the question is what would be the most convenient way.

1. Provide one dependency pack, which contains all the required and already patched libraries. So the user will obtain UniMRCP source and an appropriate dependency pack for it.
2. Commit all the dependencies with patches into UniMRCP tree and ship them with UniMRCP.
3. Leave everything as is and just warn the users to apply the patches.

Also I'm thinking about the version of APR to use and recommend. Actually this patch with minor modifications can be applied over any version of APR including 1.2.x and 1.3.x.
I used APR-1.2.12 before, but recently have migrated to APR-1.3.3 and APR-Util-13.4. However the latest ones are APR-1.3.5 and APR-1.3.7.

Please feel free to share you thoughts over the issue and the topics above.

--
Arsen Chaloyan
The author of UniMRCP
http://www.unimrcp.org

[Anthony Masse]

Hi Arsen,

1 or 3 for me (I prefer point 3)

Anthony

2009/7/17 Arsen Chaloyan <acha...@gmail.com>

[Arsen Chaloyan]

Well, I have done both options 1 and 3 in order to help new users making first steps with UniMRCP (#1) and also allow experienced users to continue using the libraries they have already installed (#3).

In other words,
1. Packages, which include patched APR and Sofia-SIP libraries, are available from download area.
http://unimrcp.googlecode.com/files/unimrcp-deps.tar.gz (Unix)
http://unimrcp.googlecode.com/files/unimrcp-deps.zip (Win)
This is the default and recommended way to install libraries UniMRCP uses.

3. Original non-patched libraries as well as patches themselves are available from
http://sites.google.com/a/unimrcp.org/unimrcp/dependencies
If you have to use other versions of libraries and you are confident of what you are doing, go ahead, review and apply available patches.


Thank you all for the responses and let me know if I'm missing anything over the topic.