res_speech_unimrcp.c Warnings and astobj2.c Errors

[JackWalt]

Hi 

I am periodically seeing these messages in the asterisk console:

[Feb 28 14:05:56] WARNING[25409][C-00000011]: res_speech_unimrcp.c:822 uni_recog_get: (RSU-17) Recognition completed abnormally cause: 001 reason: none

After a number of calls and a few of these messages I start seeing hundreds of asterisk errors in the console and asterisk log:

[Feb 28 14:06:00] ERROR[26492][C-00000014]: astobj2.c:129 INTERNAL_OBJ: bad magic number for object 0x1dfc370. Object is likely destroyed

Which seems like something is either getting stepped on or out of sync. 

These error messages continue to be generated during speech recognition mode calls until asterisk is restarted. Until the restart of asterisk calls don't seem to be affected as they continue, although I do notice that asterisk's data segment continues to grow in size which seems like memory is being allocated and not freed. Once asterisk is restarted I can recreate this error situation by making calls to the system, usually somewhere between 30 to 100 calls will recreate the error. I haven't been able to identify anything specific about the call that causes the error.

I am currently running Asterisk 13.10.0 with UniMRCP version 1.4.0 and Lumenvox 15.1.200, this was recommended version combination from Lumenvox. I have tried a number of other combinations of versions and posted the same question to the asterisk community forum with different versions: Asterisk 13.19.0, UniMRCP version 1.5 and Lumenvox 15.1.200: Asterisk Community Forum question. I also tried UniMRCP version 1.3. 

Any ideas about what I can do to figure out why this is happening or what I can do to stop it from happening? 

Thanks for any attention to this, suggestions and help is appreciated. 

[Arsen Chaloyan]

Hi Jack,

Let's move straight to the problem.

[Feb 28 14:06:00] ERROR[26492][C-00000014]: astobj2.c:129 INTERNAL_OBJ: bad magic number for object 0x1dfc370. Object is likely destroyed

Some facts. There have been numerous similar reports. To the best of my knowledge, this issue applies to Asterisk 13 and above, and only the module res_speech_unimrcp is affected, but not app_unimrcp. If you try to compile the same version of res_speech_unimrcp against a former version of Asterisk, such as 1.8 for example, everything would work as expected.

Based on what I have heard, it does not matter how many simultaneous calls are placed, it looks like the problem exhibits after a certain number of calls even being processed one at a time. I last discussed this issue on a phone call with one of the users and offered to investigate the problem in the scope of commercial support. However, the user decided to switch to app_unimrcp instead. Now, having no spare time at all, I will try to assist you with analysis, when I can, if you want to further investigate the issue.

The problem must relate to astobjects. Although I do not know too much about the technique used behind, a wrong assumption could be made with regards objects allocated from memory pools, used intensively by UniMRCP. So, the first thing I would do is tracking the pointer which causes the error "Object is likely destroyed". Hmm...

Hope this helps.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Arsen Chaloyan
Author of UniMRCP
http://www.unimrcp.org

[JackWalt]

Hi 

Yes that helps a lot, thank you. Using app_unimrcp is not an option but temporarily going to an older version of asterisk is an option. We are interested in spending some time further investigating the problem.

To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arsen Chaloyan]

Asterisk users:

I finally found the time to replicate the issue with Asterisk segfaulting in res_speech with "Object is likely destroyed" message.

Scenario

The problem was actually no that hard to reproduce and the behavior is very consistent. You need to use a call generator placing one call at a time hitting dialplan which makes use of Generic Speech API of Asterisk. After about hundred calls being placed (the number may differ and do not actually matter), Asterisk would crash in the same manner.

Root Cause

The backtrace clearly points to the function ast_speech_new() in res_speech.c. However, the problem is coming from the function which invokes ast_speech_new() such as the function speech_create() in app_speech_utils.c.

It looks like the second parameter to asp_speech_new() is supposed to be allocated using ast_format_cap_alloc, whereas an existing object is referenced by ast_channel_nativeformats(chan) instead.

This is purely a problem of Asterisk, external modules using res_speech API cannot have any influence or impact on the exhibition of the problem.

Affected Versions

This issue was first introduced in Asterisk 13, where ast_format_cap came into the place, and all the upstream versions seem to be affected.

How To Address

The workaround is simple enough. You may use the attached patch, which is made for Asterisk 13.11.2. Since the file ast_speech_utils.c has not been changed significantly, it should be straightforward to apply the same patch to all the upstream versions. Here is the diff to consider

+#include "asterisk/format_cache.h"

+    struct ast_format_cap *cap;
 
+    if (!(cap = ast_format_cap_alloc(AST_FORMAT_CAP_FLAG_DEFAULT))) {
+        pbx_builtin_setvar_helper(chan, "ERROR", "1");
+        return 0;
+    }
+    ast_format_cap_append(cap, ast_format_slin, 0);
+
     /* Request a speech object */
-    speech = ast_speech_new(data, ast_channel_nativeformats(chan));
+    speech = ast_speech_new(data, cap);

Summary

The patched version has been running for a while having processed thousands of calls without any issues. Those of you experiencing the same problem: please apply the patch and post back the results. This will need to be reported to Asterisk as well. Someone please take care of that. It is not clear when I'll have the time to get back to this topic next time.

To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[JackWalt]

That is excellent news. Thanks for your time. We will be testing the patch asap and I will post the results. 

I replied to a message on the Asterisk Community Support forum in response to my original question there, I'm not sure if there's a more appropriate place to post it: Asterisk Community

[JackWalt]

Hi.

I was able to recreate the segfault in asterisk with only 26 calls while running asterisk 13.10.0. I applied your patch to app_speech_utils.c and I'm on call 400-ish and no seg faults. I plan to leave the calls running for the rest of the day.

I posted in the asterisk community forum about your patch and linked to this thread, however I received feedback that either you or your employer "have legal authority" to submit it to asterisk: Asterisk Community thread.

On Wednesday, March 21, 2018 at 9:51:02 PM UTC-5, Arsen Chaloyan wrote:

[Arsen Chaloyan]

Glad to hear the patch resolves the problem. Thanks for your note. I'll try to submit this upstream when have some spare time.

To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.