Freeswitch dies itself without making any crash logs.

[Tajamul Maqbool]

Hi Team ,

We are using Freeswitch with Sangoma ftmd cards connected to it. Freeswitch is getting killed itself without any crash logs and we are not able to find the why of it. Firstly we were suspecting DB queries that we use for callcenter to be the reason behind but we have some other instances where FS died and we do not have any callcenter DB query running at that time or may be these are two different scenarios.

FS logs just at that time are ::  

2018-05-03 11:22:22.031199 [DEBUG] mod_freetdm.c:645 [4:12] FreeTDM/4:12/09972766570 CHANNEL HANGUP EXIT

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:60 FreeTDM/4:12/09972766570 Standard HANGUP, cause: SUBSCRIBER_ABSENT

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:741 (FreeTDM/4:12/09972766570) State HANGUP going to sleep

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:508 (FreeTDM/4:12/09972766570) State Change CS_HANGUP -> CS_REPORTING

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:473 (FreeTDM/4:12/09972766570) Running State Change CS_REPORTING

02997df9-ea93-45d1-bb4b-c87cd67ca07d 2018-05-03 11:22:22.031199 [DEBUG] mod_callcenter.c:2972 Member  <9908633335> abandoned waiting in queue eb52d7b4-780e-45b5-89c2-9390643488e2

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:827 (FreeTDM/4:12/09972766570) State REPORTING

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:104 FreeTDM/4:12/09972766570 Standard REPORTING, cause: SUBSCRIBER_ABSENT

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:827 (FreeTDM/4:12/09972766570) State REPORTING going to sleep

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_state_machine.c:499 (FreeTDM/4:12/09972766570) State Change CS_REPORTING -> CS_DESTROY

e4fee377-132f-492c-b504-3e8d7f6353ee_0 2018-05-03 11:22:22.031199 [DEBUG] switch_core_session.c:1646 Session 60701 (FreeTDM/4:12/09972766570) Locked, Waiting on external entities

2018-05-03 11:23:13.921696 [CONSOLE] switch_loadable_module.c:1538 Successfully Loaded [mod_logfile]

2018-05-03 11:23:13.922009 [CONSOLE] switch_loadable_module.c:1538 Successfully Loaded [mod_enum]

2018-05-03 11:23:13.922015 [NOTICE] switch_loadable_module.c:250 Adding Dialplan 'enum'

We will be thankful if somebody is able to help us out here.

[ros tel]

try running:

sysctl -w kernel.core_uses_pid=1

sysctl -w kernel.core_pattern=/tmp/core-%e-%s-%u-%g-%p-%t

sysctl -w fs.suid_dumpable=2

четверг, 3 мая 2018 г., 13:41:28 UTC+5 пользователь Tajamul Maqbool написал:

[virendra singh]

HI Check your syslogs and reinstall lib PRI driver.  If possible give me system. access. I can help you out end to end/

--
Вы получили это сообщение, поскольку подписаны на группу "freeswitch-ru".
Чтобы отменить подписку на эту группу и больше не получать от нее сообщения, отправьте письмо на электронный адрес freeswitch-ru+unsubscribe@googlegroups.com.
Чтобы отправлять сообщения в эту группу, отправьте письмо на электронный адрес freesw...@googlegroups.com.
Чтобы зайти в группу, перейдите по ссылке https://groups.google.com/group/freeswitch-ru.
Чтобы настроить другие параметры, перейдите по ссылке https://groups.google.com/d/optout.

--

VIRENDRA SINGH
Software Engineer

[Tajamul Maqbool]

Got something in syslogs but couldn't find much help for this error while googling it.

May  3 11:22:22 BlrFs121Out kernel: [1845157.917303] freeswitch[11934] general protection ip:7f34bdd23b59 sp:7f33a7f83650 error:0 in libfreeswitch.so.1.0.0[7f34bdcdc000+1b7000]

May  3 11:58:26 BlrFs121Out kernel: [1847320.030995] freeswitch[34973] general protection ip:7f6d2c389773 sp:7f6c14686648 error:0 in libpthread-2.15.so[7f6d2c37e000+18000] 

[Tajamul Maqbool]

Also we already enabled coredump but nothing is coming there. 

Now we have also enabled strace on it .

[Tajamul Maqbool]

Finally got the core dump . Getting this error:

""  <error reading variable state (Cannot access memory at address 0xfff3fff7fff90042)>   ""

The GDB of core is given below.

root@FS:/var/crash# gdb /usr/local/freeswitch/bin/freeswitch core.freeswitch.29553.FS.1526015835 

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.launchpad.net/gdb-linaro/>...

Reading symbols from /usr/local/freeswitch/bin/freeswitch...done.

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff6f2f7000

Core was generated by `/usr/local/freeswitch/bin/freeswitch -nc -core -rp'.

Program terminated with signal 11, Segmentation fault.

#0  switch_channel_get_state (channel=0xfff3fff7fff8fff2) at src/switch_channel.c:2081

2081            return state;

(gdb) bt full

#0  switch_channel_get_state (channel=0xfff3fff7fff8fff2) at src/switch_channel.c:2081

        state = <error reading variable state (Cannot access memory at address 0xfff3fff7fff90042)>

#1  0x00007f88f8080489 in switch_core_session_read_lock (session=0x7f88d3ca40c8) at src/switch_core_rwlock.c:88

        status = SWITCH_STATUS_FALSE

#2  0x00007f88f8071fbb in switch_core_session_hupall_matching_var_ans (var_name=0x7f87fd655943 "cc_member_pre_answer_uuid", 

    var_val=0x7f87e505ff30 "29bfe1e5-44a7-45c7-b4b3-0d4d4afe6ad8", cause=SWITCH_CAUSE_LOSE_RACE, type=<optimized out>) at src/switch_core_session.c:230

        hi = 0x7f87dcc403c0

        val = 0x7f88d3ca40c8

        session = 0x7f88d3ca40c8

        pool = 0x7f88d0b35578

        head = <optimized out>

        np = <optimized out>

        r = 0

        __func__ = "switch_core_session_hupall_matching_var_ans"

#3  0x00007f87fd64beb7 in outbound_agent_thread_run (thread=<optimized out>, obj=0x7f87e505fe88) at mod_callcenter.c:1774

        res = "1", '\000' <repeats 254 times>

        member_channel = 0x7f88701ad030

        agent_channel = 0x7f88d32aae00

        other_loopback_leg_uuid = <optimized out>

        agent_uuid = 0x7f88d32438b4 "daf3befe-8af4-41a1-929e-a0a2b6635020"

        o_announce = 0x0

        agent_session = 0x7f88d32430b8

        cause = SWITCH_CAUSE_SUCCESS

        status = <optimized out>

        sql = <optimized out>

        dialstr = "{ivr_refnum=1000028741,ivr_disp_num=+912233485852,origination_caller_id_number=+911244948676}freetdm/outgoing/r/09130034868\000\207\177\000\000\350\003\000\000\270\00

6\000\000O\352C?\347k\005", '\000' <repeats 29 times>, "\001", '\000' <repeats 12 times>, " \031\217\060g\361#0\241\022\370\210\177\000\000h\004\024<\210\177\000\000\002\000\000\000\000

\000\000\000\001\000\000\000\000\000\000\000pB\027\370\210\177\000\000.+"...

        tiers_state = CC_TIER_STATE_READY

        member_session = 0x7f88701aef58

        __func__ = "outbound_agent_thread_run"

        ovars = 0x0

        t_agent_called = <optimized out>

        t_agent_answered = 0

        t_member_called = <optimized out>

        event = 0x0

        _cycle_on = 1

        xfer_uuid = 0x0

        _h = 0x0

        agent_bridged = 0

        wrt = 0

        agent_timeout = <optimized out>

#4  0x00007f88f8157a56 in dummy_worker (opaque=0x7f87e5060258) at threadproc/unix/thread.c:151

        thread = 0x7f87e5060258

---Type <return> to continue, or q <return> to quit---

#5  0x00007f88f7dece9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

No symbol table info available.

#6  0x00007f88f7b1a2ed in clone () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#7  0x0000000000000000 in ?? ()

No symbol table info available.

(gdb) p/a 0xfff3fff7fff90042

$1 = 0xfff3fff7fff90042

(gdb) p/a 0xfff3fff7fff8fff2

$2 = 0xfff3fff7fff8fff2

(gdb) 41

Undefined command: "41".  Try "help".

(gdb) $1

Undefined command: "$1".  Try "help".

(gdb) info $1

Undefined info command: "$1".  Try "help info".

(gdb) p 0xfff3fff7fff90042

$3 = 18443366339628826690

(gdb) a 0xfff3fff7fff90042

Ambiguous command "a 0xfff3fff7fff90042": actions, add-inferior, add-symbol-file, add-symbol-file-from-memory, advance, alias, aliases...

(gdb) x/s 0xfff3fff7fff90042

0xfff3fff7fff90042:      <Address 0xfff3fff7fff90042 out of bounds>

(gdb) x/s 0xfff3fff7fff8fff2

0xfff3fff7fff8fff2:      <Address 0xfff3fff7fff8fff2 out of bounds>

(gdb) x/s 0x7f88d0b35578

0x7f88d0b35578:  "8\241r"

(gdb) x/s 0x7f87dcc403c0

0x7f87dcc403c0:  "N"

(gdb) x/s 0x7f88d3ca40c8

0x7f88d3ca40c8:  "\002"

(gdb) x/s 0x7f87e505ff30

0x7f87e505ff30:  "29bfe1e5-44a7-45c7-b4b3-0d4d4afe6ad8"

(gdb) x/s 0x7f87fd655943

0x7f87fd655943:  "cc_member_pre_answer_uuid"

(gdb) x/s 0xfff3fff7fff8fff2

0xfff3fff7fff8fff2:      <Address 0xfff3fff7fff8fff2 out of bounds>

(gdb) x/s 0xfff3fff7fff90042

0xfff3fff7fff90042:      <Address 0xfff3fff7fff90042 out of bounds>

(gdb) x/s 0x00007f88f8080489

0x7f88f8080489 <switch_core_session_read_lock+57>:       "\203\370\tH\213\273\350\003"

(gdb) quit

You have new mail in /var/mail/root

root@MumFs20:/var/crash# 

root@MumFs20:/var/crash# 

root@MumFs20:/var/crash# ulimit -c

0

root@MumFs20:/var/crash# ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 126891

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 126891

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited