G729 Illegal Instruction Error (Asterisk 1.6 Xeon Single Core)

[kster...@gmail.com]

Hi All,

I'm running Asterisk 1.6.2 and downloaded "codec_g729-ast16-gcc4-glibc-pentium4-sse3.so" but i keep getting illegal instruction. I have tried other variants of g729 like pentium4, pentium3 etc but couldn't get it right. Appreciate if there's anybody that had encountered similar issues & know found the working solution!

[root@ELASTIX-203 modules]# /usr/sbin/safe_asterisk: line 145: 12383 Illegal instruction     (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 132
Asterisk exited on signal 4.
Automatically restarting Asterisk.
/usr/sbin/safe_asterisk: line 145: 12424 Illegal instruction     (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 132
Asterisk exited on signal 4.
Automatically restarting Asterisk.


[root@ELASTIX-203 modules]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 9
cpu MHz         : 2777.214
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips        : 5554.42

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 9
cpu MHz         : 2777.214
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips        : 5553.42

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 9
cpu MHz         : 2777.214
cache size      : 512 KB
physical id     : 3
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 6
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips        : 5553.31

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 9
cpu MHz         : 2777.214
cache size      : 512 KB
physical id     : 3
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 7
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xt

[Arkadi Shishlov]

Please post the error from plain pentium4 binary as the CPU does not have SSE3, so SIGILL is an expected outcome.

[kster...@gmail.com]

Dear Arkadi,

Thanks for your reply, I get the following result using pentium4 binary:

[root@ELASTIX-203 modules]# /etc/init.d/asterisk restart
Stopping safe_asterisk:                                    [  OK  ]
Shutting down asterisk:                                    [  OK  ]
Starting asterisk:                                         [  OK  ]
[root@ELASTIX-203 modules]# /usr/sbin/safe_asterisk: line 145:  3184 Segmentation fault      (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.

Regards,
Kevin

[kster...@gmail.com]

Additional Info that might be relevant:

[root@ELASTIX-203 modules]# tail /var/log/asterisk/full
[Nov 10 11:47:49] VERBOSE[3297] pbx.c:   == Registered application 'PlayTones'
[Nov 10 11:47:49] VERBOSE[3297] pbx.c:   == Registered application 'StopPlayTones'
[Nov 10 11:47:49] VERBOSE[3297] loader.c:  app_playtones.so => (Playtones Application)
[Nov 10 11:47:49] VERBOSE[3297] translate.c:   == Registered translator 'alawtolin' from format alaw to slin, cost 1
[Nov 10 11:47:49] VERBOSE[3297] translate.c:   == Registered translator 'lintoalaw' from format slin to alaw, cost 1
[Nov 10 11:47:49] VERBOSE[3297] loader.c:  codec_alaw.so => (A-law Coder/Decoder)
[Nov 10 11:47:49] VERBOSE[3297] codec_dahdi.c:   == No hardware transcoders found.
[Nov 10 11:47:49] VERBOSE[3297] loader.c:  codec_dahdi.so => (Generic DAHDI Transcoder Codec Translator)
[Nov 10 11:47:49] VERBOSE[3297] translate.c:   == Registered translator 'g729tolin' from format g729 to slin, cost 2000
[Nov 10 11:47:49] VERBOSE[3297] translate.c:   == Registered translator 'lintog729' from format slin to g729, cost 4999

[root@ELASTIX-203 modules]# cat /etc/asterisk/modules.conf
;
; Asterisk Module Loader configuration file
;
;

[modules]
autoload=yes
;
; Any modules that need to be loaded before the Asterisk core has been
; initialized (just after the logger has been initialized) can be loaded
; using 'preload'. This will frequently be needed if you wish to map all
; module configuration files into Realtime storage, since the Realtime
; driver will need to be loaded before the modules using those configuration
; files are initialized.
;
; An example of loading ODBC support would be:
;preload => res_odbc.so
;preload => res_config_odbc.so
;
; As FreePBX is using Local as the channel for queue members we need to make sure
; that pbx_config.so and chan_local.so are preloaded. If not, queue members
; will be marked as invalid until app_queue is reloaded.
preload => pbx_config.so
preload => chan_local.so
;
; Uncomment the following if you wish to use the Speech Recognition API
;preload => res_speech.so
;
; If you want, load the GTK console right away.
; KDE console is obsolete and was removed from Asterisk 2008-01-10
;
noload => pbx_gtkconsole.so
;load => pbx_gtkconsole.so
noload => pbx_kdeconsole.so
;
; Intercom application is obsoleted by
; chan_oss.  Don't load it.
;
noload => app_intercom.so
;
; DON'T load the chan_modem.so, as they are obsolete in * 1.2

noload => chan_modem.so
noload => chan_modem_aopen.so
noload => chan_modem_bestdata.so
noload => chan_modem_i4l.so

; don't load the following channels in Elastix by default
noload => chan_skinny.so
noload => chan_unicall.so

; Trunkisavail is a broken module supplied by Trixbox
noload => app_trunkisavail.so

; Ensure that format_* modules are loaded before res_musiconhold
;load => format_ogg_vorbis.so
load => format_wav.so
load => format_pcm.so

; format_au.so is removed from Asterisk 1.4 and later, remove ; to enable
;load => format_au.so

; This isn't part of 'Asterisk' iteslf, it's part of asterisk-addons. If this isn't
; installed, asterisk will fail to start. But it does need to go here for native MOH
; to work using mp3's.
;       Note that on a system with a high number of calls, using a compressed audio format for
;       musiconhold takes CPU resources. Converting these files to ulaw/alaw makes the job
;       much easier for your CPU.
load => format_mp3.so
load => res_musiconhold.so
;
; Load either OSS or ALSA, not both
; By default, load no console driver
;
noload => chan_alsa.so
noload => chan_oss.so

noload => res_config_pgsql.so
noload => cdr_pgsql.so

noload => chan_unicall.so

;
noload => app_directory_odbcstorage.so
noload => app_voicemail_odbcstorage.so

[root@ELASTIX-203 modules]# ls
app_addon_sql_mysql.so  app_url.so                  func_blacklist.so
app_adsiprog.so         app_userevent.so            func_callerid.so
app_alarmreceiver.so    app_verbose.so              func_cdr.so
app_amd.so              app_voicemail.so            func_channel.so
app_authenticate.so     app_waitforring.so          func_config.so
app_cbmysql.so          app_waitforsilence.so       func_curl.so
app_cdr.so              app_waituntil.so            func_cut.so
app_chanisavail.so      app_while.so                func_db.so
app_channelredirect.so  app_zapateller.so           func_devstate.so
app_chanspy.so          bridge_builtin_features.so  func_dialgroup.so
app_confbridge.so       bridge_multiplexed.so       func_dialplan.so
app_controlplayback.so  bridge_simple.so            func_enum.so
app_dahdibarge.so       bridge_softmix.so           func_env.so
app_dahdiras.so         cdr_addon_mysql.so          func_extstate.so
app_dahdiscan.so        cdr_csv.so                  func_global.so
app_db.so               cdr_custom.so               func_groupcount.so
app_dial.so             cdr_manager.so              func_iconv.so
app_dictate.so          cdr_pgsql.so                func_lock.so
app_directed_pickup.so  cdr_sqlite3_custom.so       func_logic.so
app_directory.so        chan_agent.so               func_math.so
app_disa.so             chan_alsa.so                func_md5.so
app_dumpchan.so         chan_bridge.so              func_module.so
app_echo.so             chan_dahdi.so               func_rand.so
app_exec.so             chan_iax2.so                func_realtime.so
app_externalivr.so      chan_local.so               func_sha1.so
app_fax.so              chan_mgcp.so                func_shell.so
app_festival.so         chan_ooh323.so              func_speex.so
app_flash.so            chan_oss.so                 func_sprintf.so
app_followme.so         chan_phone.so               func_strings.so
app_forkcdr.so          chan_sip.so                 func_sysinfo.so
app_getcpeid.so         chan_skinny.so              func_timeout.so
app_ices.so             chan_unistim.so             func_uri.so
app_image.so            chan_woomera.so             func_version.so
app_macro.so            codec_adpcm.so              func_vmcount.so
app_meetme.so           codec_alaw.so               func_volume.so
app_milliwatt.so        codec_a_mu.so               pbx_ael.so
app_minivm.so           codec_dahdi.so              pbx_config.so
app_mixmonitor.so       codec_g722.so               pbx_dundi.so
app_morsecode.so        codec_g726.so               pbx_loopback.so
app_mp3.so              codec_g729.so               pbx_realtime.so
app_nbscat.so           codec_gsm.so                pbx_spool.so
app_originate.so        codec_lpc10.so              res_adsi.so
app_page.so             codec_speex.so              res_ael_share.so
app_parkandannounce.so  codec_ulaw.so               res_agi.so
app_playback.so         format_g723.so              res_clialiases.so
app_playtones.so        format_g726.so              res_clioriginate.so
app_privacy.so          format_g729.so              res_config_curl.so
app_queue.so            format_gsm.so               res_config_ldap.so
app_readexten.so        format_h263.so              res_config_mysql.so
app_readfile.so         format_h264.so              res_config_pgsql.so
app_read.so             format_ilbc.so              res_convert.so
app_record.so           format_jpeg.so              res_crypto.so
app_saycountpl.so       format_mp3.so               res_curl.so
app_sayunixtime.so      format_ogg_vorbis.so        res_limit.so
app_senddtmf.so         format_pcm.so               res_monitor.so
app_sendtext.so         format_siren14.so           res_musiconhold.so
app_setcallerid.so      format_siren7.so            res_phoneprov.so
app_sms.so              format_sln16.so             res_realtime.so
app_softhangup.so       format_sln.so               res_smdi.so
app_speech_utils.so     format_vox.so               res_snmp.so
app_stack.so            format_wav_gsm.so           res_speech.so
app_system.so           format_wav.so               res_timing_dahdi.so
app_talkdetect.so       func_aes.so                 res_timing_pthread.so
app_test.so             func_audiohookinherit.so
app_transfer.so         func_base64.so

[Arkadi Shishlov]

On Sun, Nov 10, 2013 at 6:03 AM, <kster...@gmail.com> wrote:
>
> Thanks for your reply, I get the following result using pentium4 binary:
>
> [root@ELASTIX-203 modules]# /etc/init.d/asterisk restart
> Stopping safe_asterisk: [ OK ]
> Shutting down asterisk: [ OK ]
> Starting asterisk: [ OK ]
> [root@ELASTIX-203 modules]# /usr/sbin/safe_asterisk: line 145: 3184 Segmentation fault (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
> Asterisk ended with exit status 139
> Asterisk exited on signal 11.
> Automatically restarting Asterisk.

1. Which binary exactly have you downloaded?
2. There may be something in dmesg, probably some security module kicks in.
3. Try build without SSE and plain Pentium builds also.
4.Then you need to follow GDB instructions from Troubleshooting section.

[kster...@gmail.com]

Dear Arkadi,

Thanks for your reply. My response as follows:

1. Which binary exactly have you downloaded?

    I was using "codec_g729-ast16-gcc4-glibc-pentium4.so"

2. There may be something in dmesg, probably some security module kicks in.

    Most Probably

3. Try build without SSE and plain Pentium builds also.

    I tried with following result

No-SSE

[root@ELASTIX-203 modules]# /usr/sbin/safe_asterisk: line 145:  4994 Segmentation fault      (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}

Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.

PENTIUM

[root@ELASTIX-203 modules]# /usr/sbin/safe_asterisk: line 145:  5126 Segmentation fault      (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}

Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.

exit/usr/sbin/safe_asterisk: line 145:  5184 Segmentation fault      (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}

Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.

4. Troubleshooting using GDB
attached the result

Regards,
Kevin

[Arkadi Shishlov]

Thanks, that was quite informative!

Please, double check that you're using this file specifically:

http://asterisk.hosting.lv/bin162/codec_g729-ast16-gcc4-glibc-pentium4.so

Due to the way I've built 1.6.x codecs they are all tagged "-ast16-", but pay attention to the download path that contains "bin162/". Asterisk 1.6.0 / 1.6.1 / 1.6.2 are not ABI compatible and thus module crashes.

The stack trace indicates that EIP register is corrupted and that is most likely by module expecting in-memory data structures being of different sizes than they really are.

#0  0x00000000 in ?? ()

#1  0x08098e90 in ast_cli_register ()

#2  0x03b3e5d9 in load_module () at codec_g72x.c:581

There are no explicit protection in the binaries to discover version (in)compatibility. So if symbols links in run-time, then we're good to go.

If you're absolutely sure you have the correct binary, then it's probably the Elastix build of Asterisk that is compiled with a different set of features or compiler setting that does not match my binaries at ABI level. In that case you need to recompile the codec against Elastix headers and with Elastix compiler/settings that was used to build Asterisk.

Get IPP 7.1 from Intel website as current version of ./configure is not yet updated&tested against IPP 8.0.

For details see the README and ./configure --help.

[kster...@gmail.com]

Dear Arkadi,

Thanks a million for pointing out the binary path issue :), It seems that I was downloading the wrong version (didn't know that the binary of 1.6.0, 1.6.1 and 1.6.2 was of different version albeit the same naming). Appreciate your great assistance during the troubleshooting!

ELASTIX-203*CLI> core show translation
         Translation times between formats (in microseconds) for one second of data
          Source Format (Rows) Destination Format (Columns)

           g723   gsm  ulaw  alaw g726aal2 adpcm  slin lpc10  g729 speex  ilbc  g726  g722 siren7 siren14 slin16
     g723     -     -     -     -        -     -     -     -     -     -     -     -     -      -       -      -
      gsm     -     -  1001  1001     3999  2000  1000  4000  6000 16997     -  4999  1999      -       -   3999
     ulaw     -  3001     -     1     3000  1001     1  3001  5001 15998     -  4000  1000      -       -   3000
     alaw     -  3001     1     -     3000  1001     1  3001  5001 15998     -  4000  1000      -       -   3000
 g726aal2     -  4000  1001  1001        -  2000  1000  4000  6000 16997     -  4999  1999      -       -   3999
    adpcm     -  3001     2     2     3000     -     1  3001  5001 15998     -  4000  1000      -       -   3000
     slin     -  3000     1     1     2999  1000     -  3000  5000 15997     -  3999   999      -       -   2999
    lpc10     -  4999  2000  2000     4998  2999  1999     -  6999 17996     -  5998  2998      -       -   4998
     g729     -  4000  1001  1001     3999  2000  1000  4000     - 16997     -  4999  1999      -       -   3999
    speex     -  5000  2001  2001     4999  3000  2000  5000  7000     -     -  5999  2999      -       -   4999
     ilbc     -     -     -     -        -     -     -     -     -     -     -     -     -      -       -      -
     g726     -  4000  1001  1001     3999  2000  1000  4000  6000 16997     -     -  1999      -       -   3999
     g722     -  4000  1001  1001     3999  2000  1000  4000  6000 16997     -  4999     -      -       -   2000
   siren7     -     -     -     -        -     -     -     -     -     -     -     -     -      -       -      -
  siren14     -     -     -     -        -     -     -     -     -     -     -     -     -      -       -      -
   slin16     -  6999  4000  4000     6998  4999  3999  6999  8999 19996     -  7998  2999      -       -      -

Regards,
Kevin