QoS call stats stopped working, but RTCP is there

[Matthew Ogden]

Hi all

I'm getting this as a result in my QoS tab: 

Yet if I query the DB:

select * from rtcp_capture WHERE correlation_id like '%2a75d64b23dd426c17a46caa41a23805%' ORDER BY id desc;

I get 30 records: 

such as:

| 18087182 | 2016-05-11 16:54:24 | 1462978464471387 | 2a75d64b23dd426c17a46caa41a23805@xxxxxx       | yyyyy |       41261 | zzzzzzzz   |    7045 |     1 |      2 |    0 | homer01:2001 | { "sender_information":{"ntp_timestamp_sec":566681050,"ntp_timestamp_usec":4076596292,"octets":40000,"rtp_timestamp":2430284401, "packets":250},"ssrc":1665124203,"type":200, "report_blocks                                                                      ":[{"source_ssrc":61944884,"highest_seq_no":57429,"fraction_lost":0,"ia_jitter":                                                                      160,"packets_lost":0,"lsr":0,"dlsr":4294127715}],"report_count":1,"sdes_ssrc":16                                                                      65124203,"sdes_chunk_ssrc":16777216,"sdes_information": [],"sdes_report_count":0                                                                      }        |

+----------+---------------------+------------------+---------------------------                                                                      -------------------------+---------------+-------------+----------------+-------                                                                      -----------+-------+--------+------+--------------+-----------------------------                                                                      --------------------------------------------------------------------------------                                                                      --------------------------------------------------------------------------------                                                                      --------------------------------------------------------------------------------                                                                      --------------------------------------------------------------------------------                                                                      --------------------------------------------------------------------------------                                                                      ---------+

30 rows in set (2 min 13.42 sec)

I see it takes long to query, though I dont know what SQL you guys use to do the query

Any ideas why it might break?

[Matthew Ogden]

Any ideas from anyone? 

I can also confirm the post to /api/v1/report/qos is definately coming back blank as in the screenshot below:

[Lorenzo Mangani]

Hi Matthew,

Empty array = no RTCP packets/statistics received in most cases. The remainder of the values are determined from SDP.

Kind Regards,

Lorenzo Mangani

Managing Director and Core Dev

QXIP BV - Capture Engineering

Amsterdam, The Netherlands

Desk: +31 20 8080468 / +1 202-632-7227

Mobile: +31 6 4603-2730

Email: in...@qxip.net

Find our more about HOMER, PCAPTURE, CAPTAGENT and all our Services and Solutions at http://qxip.net 

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or legally privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of this original message. 

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Ogden]

Hi Lorenzo

Not sure if you see in my first post. But if I query the table from mysql for these calls callid /correlation-id there is plenty of rtcp. It's been working pretty well and now all calls show no stats even when there are stats.

Kind Regards

[Lorenzo Mangani]

Matthew, 

Are the report timestamps matching the sessions for the results you can find on those Call-IDs?

Kind Regards,

Lorenzo Mangani

Managing Director and Core Dev

QXIP BV - Capture Engineering

Amsterdam, The Netherlands

Desk: +31 20 8080468 / +1 202-632-7227

Mobile: +31 6 4603-2730

Email: in...@qxip.net

Find our more about HOMER, PCAPTURE, CAPTAGENT and all our Services and Solutions at http://qxip.net 

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or legally privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of this original message. 

[Matthew Ogden]

Hi Lorenzo

Are you referring to the datetime stamps or the integer timestamps?

[Matthew Ogden]

Hi Lorenzo

The timestamps for date look fine to me?

Kind Regards

On Saturday, 14 May 2016 18:02:01 UTC+2, Matthew Ogden wrote:

Hi Lorenzo

Are you referring to the datetime stamps or the integer timestamps?

On 14 May 2016 5:51 PM, "Lorenzo Mangani" <lorenzo...@gmail.com> wrote:

Matthew, 

Are the report timestamps matching the sessions for the results you can find on those Call-IDs?

Kind Regards,

Lorenzo Mangani

Managing Director and Core Dev

QXIP BV - Capture Engineering

Amsterdam, The Netherlands

Desk: +31 20 8080468 / +1 202-632-7227

Mobile: +31 6 4603-2730

Email: in...@qxip.net

Find our more about HOMER, PCAPTURE, CAPTAGENT and all our Services and Solutions at http://qxip.net 

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or legally privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of this original message. 

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discuss+unsubscribe@googlegroups.com.

[Matthew Ogden]

Hi

I'm getting pretty desperate on this. 

Really not sure what else I can look at. The data is there, but something about it the query just not returning any reports, if you can point me where to look, I'll dig deeper.

Kind Regards

[Alexandr Dubovikov]

Matthew, can you please open web development console (F12 in chrome) and click again on a callid. In the console you should see some API calls and one has name "qos" (the full path api/v1/report/qos)

Please check that and see the result.

After, please go to RestApi/Report.php, find function: doRTCPServerReport (line 398)

in the "foreach ($nodes as $node)" after $db->loadObjectArray, add: line 450)

 openlog("myhomerapilog", LOG_PID | LOG_PERROR, LOG_LOCAL0);
 $access = date("Y/m/d H:i:s");
 syslog(LOG_WARNING, "RTCP Query: $access : Query: $query");

closelog();

save and execute the query again (right button on call and "Replay XHR)

after check your query in the syslog and check why it doesn't return any data.

Wbr,

Alexandr

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.

[Matthew Ogden]

Hi

Ok, so you have definitely found the place of the bug!

The correlation ID's seem to but a INVITE packet data instead of correlation ID.

SELECT *       ,

       'single' as dbnode

FROM rtcp_capture

WHERE ('date' BETWEEN FROM_UNIXTIME(1463499734) AND FROM_UNIXTIME(1463544462)) AND ( ( correlation_id

= 'INVITE sip:09876...@1.1.2.235 SIP/2.0\r\nRecord-Route: <sip:1.1.1.94' OR correlation_id = 'lr' OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\n

Via: SIP/2.0/UDP 1.1.1.84:5070' OR correlation_id = 'received=1.1.1.84' 

OR correlation_id = 'branch=z9hG4bK378e2373' OR correlation_id = 'rport=5070\r\nFrom:'  OR correlation_id = 'INVITE sip:09876...@1.1.2.235 SIP/2.0\r\nRecor

d-Route: <sip:1.1.1.94' OR correlation_id = 'lr' OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\nVia: SIP/2.0/UDP 1.1.1.84:5070' OR correlation_id = 'received=1.1.1.84' OR correlation_id = 'branch=z9hG4bK378e2373' 

OR correlation_id = 'rport=5070\r\nFrom' OR correlation_id = 'SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'rport=5060' OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\nVia: SIP/2.0/UDP 1.1.1.84:5070' 

OR correlation_id = 'received=1.1.1.84' OR correlation_id = 'branch=z9hG4bK378e2373' 

OR correlation_id = 'rport=5070\r\nFrom: \"+27123456789\" <sip:+27123...@1.1.1.84:5070>' 

OR correlation_id = 'tag=as5eaf33b' OR correlation_id = 'SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'rport=5060' OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\nVia: SIP/2.0/UDP 1.1.1.84:5070' 

OR correlation_id = 'received=1.1.1.84' OR correlation_id = 'branch=z9hG4bK378e2373' 

OR correlation_id = 'rport=5070\r\nFrom: \"+27123456789\" <sip:+27123...@1.1.1.84:5070>' 

OR correlation_id = 'tag=as5eaf33' OR correlation_id = 'SIP/2.0 200 OK\r\nRecord-Route: <sip:127.0.0.1:5062' OR correlation_id = 'lr=on' 

OR correlation_id = 'ftag=as5eaf33be' OR correlation_id = 'did=7aa.8a52' OR correlation_id = 'mpd=ii' OR correlation_id = 'ice_caller=strip' 

OR correlation_id = 'ice_callee=strip' OR correlation_id = 'rtpprx=yes' OR correlation_id = 'vsf=N052NHtNe0ZdRVJQSldZEFtpKUFxGSl7Mw84VgMaEmF8>\r\nRecord-Route: <sip:127.0.0.1' 

OR correlation_id = 'r2=on' OR correlation_id = 'lr=on' OR correlation_id = 'ftag=as5eaf33be' OR correlation_id = 'ngcplb=yes' OR correlation_id = '' 

OR correlation_id = 'SIP/2.0 200 OK\r\nRecord-Route: <sip:127.0.0.1:5062' OR correlation_id = 'lr=on' OR correlation_id = 'ftag=as5eaf33be' 

OR correlation_id = 'did=7aa.8a52' OR correlation_id = 'mpd=ii' OR correlation_id = 'ice_caller=strip' OR correlation_id = 'ice_callee=strip' O

R correlation_id = 'rtpprx=yes' OR correlation_id = 'vsf=N052NHtNe0ZdRVJQSldZEFtpKUFxGSl7Mw84VgMaEmF8>\r\nRecord-Route: <sip:127.0.0.1' OR correlation_id = 'r2=on' 

OR correlation_id = 'lr=on' OR correlation_id = 'ftag=as5eaf33be' OR correlation_id = 'ngcplb=yes' OR correlation_id = 'ACK sip:ngc...@1.1.2.235:5060' 

OR correlation_id = 'ngcpct=c2lwOjEyNy4wLjAuMTo1MDgw SIP/2.0\r\nRecord-Route: <sip:1.1.1.94' OR correlation_id = 'lr' 

OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.2\r\nVia: SIP/2.0/UDP 1.1.1.84:5070' 

OR correlation_id = 'received=1.1.1.84' OR correlation_id = 'branch=z9' OR correlation_id = 'ACK sip:ngc...@1.1.2.235:5060' 

OR correlation_id = 'ngcpct=c2lwOjEyNy4wLjAuMTo1MDgw SIP/2.0\r\nRecord-Route: <sip:1.1.1.94' OR correlation_id = 'lr' 

OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.2\r\nVia: SIP/2.0/UDP 1.1.1.84:5070' 

OR correlation_id = 'received=1.1.1.84' OR correlation_id = 'branch=z' OR correlation_id = 'INVITE sip:+27123...@1.1.1.84:5070 SIP/2.0\r\nRecord-Route: <sip:1.1.2.235' 

OR correlation_id = 'r2=on' OR correlation_id = 'lr=on' OR correlation_id = 'ftag=36688386-573B3E2F00065DD8-4A675700' OR correlation_id = 'ngcplb=yes' 

OR correlation_id = 'socket=udp:1.1.2.235:5060>\r\nRecord-Route: <sip:127.0.0.1' OR correlation_id = 'r2=on' OR correlation_id = 'lr=on' 

OR correlation_id = 'ftag=36688386-573B3E2F00065DD8-4A675700' OR correlation_id = 'INVITE sip:+27123...@1.1.1.84:5070 SIP/2.0\r\nRecord-Route: <sip:1.1.2.235' 

OR correlation_id = 'r2=on' OR correlation_id = 'lr=on' OR correlation_id = 'ftag=36688386-573B3E2F00065DD8-4A675700' OR correlation_id = 'ngcplb=yes' 

OR correlation_id = 'socket=udp:1.1.2.235:5060>\r\nRecord-Route: <sip:127.0.0.1' OR correlation_id = 'r2=on' OR correlation_id = 'lr=on' OR correlation_id =

---- it goes on and on, but you can see its obviously not right

To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Homer Capture Server. sipcapture.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to homer-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexandr Dubovikov]

Matthew,

I am very sorry, but this bug was fixed maybe a half year ago. Please update your kamailio to 4.4.x

Wbr,
Alexandr

[Matthew Ogden]

HI Alexander,

Interesting: It was working fine, but more importantly, this installation was done beginning of April. I checked out the kamailio and homer from git. and make && installed them. (I'm  sure I had to get 4.4.x any way for RTCP reports to work the first time )

I get this if I check my version. (it was 4.4.x latest from repo during April)

[root@localhost /]# kamailio -h

version: kamailio 5.0.0-dev0 (x86_64/linux) 6a362f

The fact that it was working fine is also odd though (Lorenzo can also testament to that too as I shared some QoS reports with him

Kind Regards

On Wednesday, 18 May 2016 08:42:08 UTC+2, Alexandr Dubovikov wrote:

Matthew,

I am very sorry, but this bug was fixed maybe a half year ago. Please update your kamailio to 4.4.x

Wbr,
Alexandr

On May 18, 2016 06:34, "'Matthew Ogden' via Homer Capture Server. sipcapture.org" <homer-...@googlegroups.com> wrote:

Hi

Ok, so you have definitely found the place of the bug!

The correlation ID's seem to but a INVITE packet data instead of correlation ID.

SELECT *       ,

       'single' as dbnode

FROM rtcp_capture

WHERE ('date' BETWEEN FROM_UNIXTIME(1463499734) AND FROM_UNIXTIME(1463544462)) AND ( ( correlation_id

= 'INVITE sip:098...@1.1.2.235 SIP/2.0\r\nRecord-Route: <sip:1.1.1.94' OR correlation_id = 'lr' OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\n

Via: SIP/2.0/UDP 1.1.1.84:5070' OR correlation_id = 'received=1.1.1.84' 

OR correlation_id = 'branch=z9hG4bK378e2373' OR correlation_id = 'rport=5070\r\nFrom:'  OR correlation_id = 'INVITE sip:098...@1.1.2.235 SIP/2.0\r\nRecor

[Alexandr Dubovikov]

Do u use HEPv2 or 3?

[Andy Thompson]

Just want to say I see the exact same behavior with Homer5 and Kamailio 4.4 without RTCP (only SIP).
I believe this issue was introduced sometime early April but I have not bothered with it as I use same callid/correlation (update .. set correlation = callid..) so easy to sort the dirty way until the field is fixed.

When is your earliest entry in your database with "borked correlation_ids" ?

On Wednesday, 18 May 2016 06:34:02 UTC+2, Matthew Ogden wrote:

Hi

Ok, so you have definitely found the place of the bug!

The correlation ID's seem to but a INVITE packet data instead of correlation ID.

SELECT *       ,

       'single' as dbnode

FROM rtcp_capture

WHERE ('date' BETWEEN FROM_UNIXTIME(1463499734) AND FROM_UNIXTIME(1463544462)) AND ( ( correlation_id

= 'INVITE sip:098...@1.1.2.235 SIP/2.0\r\nRecord-Route: <sip:1.1.1.94' OR correlation_id = 'lr' OR correlation_id = 'ftag=as5eaf33be>\r\nVia: SIP/2.0/UDP 1.1.1.94:5060' 

OR correlation_id = 'branch=z9hG4bK779c.0c9f7ae4.0\r\n

Via: SIP/2.0/UDP 1.1.1.84:5070' OR correlation_id = 'received=1.1.1.84' 

OR correlation_id = 'branch=z9hG4bK378e2373' OR correlation_id = 'rport=5070\r\nFrom:'  OR correlation_id = 'INVITE sip:098...@1.1.2.235 SIP/2.0\r\nRecor

[Matthew Ogden]

Hi

I'm not sure, my captagent 6 uses the default of 3, and the capt agent 4's use their default (assume 4). So possibly a mix then. 

Kind Regards

[Matthew Ogden]

Hi Andy

Interesting... Yup, correlation IDs are borked. But it was working for some time no problem. So its not just a bug, its also enviromental. I did update my GUIs at Lorenzo's request at some point as well.

Kind Regards

[Matthew Ogden]

Hi Andy,

And that is indeed the resolution to make it work again, though wont the _b2b legs be incorrect then?

Kind Regards

Find our more about HOMER, PCAPTURE, CAPTAGENT and all our Services and Solutions at <a href="http://qxip.net" rel="nofollow" target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fqxip.net\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoxIjCZLAgq3doM7sDjVX1XrvAIg';return true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fqxip.net\x26sa

[Matthew Ogden]

Hi Lorenzo & Alex,

Is there some additional info we can provide about the bug (both HEP v2 and 3 are doing), currently on Kamailio 4.4.x.  Updated homer around 6th of April.

Andy, do you have any other environmental info about it?

Kind Regards

[Alexandr Dubovikov]

Can u check the last kamailio's master git? We are now at Kamailio World and could check 4.4 repository.

--

[Matthew Ogden]

Hi Alexander

I'm away now until Monday, I will checkout latest git then and recompile.

(Still confused about it working then breaking without any code changes)

Kind Regards

--

 

 

Matthew Ogden

Management

TenacIT

 

 

Strategic IT Consulting • Advanced Networking • Virtualisation

Custom Development • Hosting • Syspro Support  • MS Licensing

National Tel: 041 10 10 100 | Cell: 084 205 4445 | Email: mat...@tenacit.net 

CT Tel: 021 201 0333 | Skype Name: matthew.ogden | Web: http://www.tenacit.net

    

[Alexandr Dubovikov]

Hello Matthew,

we will check it on this WE, but the fact is, you have "garbage" in the correlation id, and this was the bug in the old kamailio 4.3. We have fixed it for HEPv3, but not for v2, therefore if you receive HEPv2 this can generate same issue as before. For hepv2 was fixed here;

https://github.com/kamailio/kamailio/commit/b400123d272edc98f7b9fa3fde6f7ff650a81f89

Wbr,

Alexandr

[Andy Thompson]

Hello,

I just want to state I had the same issue with Kamailio 4.4.0-pre1, but after recompiling (including the sipcapture module) to 5.0.0-dev4 the correlation is inserted correctly again.