CDR Export for *cdrlog

244 views
Skip to first unread message

Michael Klimenko

unread,
Oct 31, 2017, 3:44:04 AM10/31/17
to CGRateS
Hello Dan,
I try to export CDRs that were produced by execution of action having *cdrlog action type.
No matter which filter I use, they are not exported.
In opposite, CDRs that were produced by ProcessCDR API are exported OK.
Am I missing something ? I have a feeling that "online_cdr_exports" is firing only when API ProcessCDR is called.
Is that right ?

I tried the following "cdr_filter" values :

"cdr_filter": "Source(*cdrlog)", 
"cdr_filter": "RunID(*topup_reset);RunID(*topup)",

Please assist
Thanks,
Michael

DanB

unread,
Oct 31, 2017, 5:06:07 AM10/31/17
to cgr...@googlegroups.com
Hey Michael,

Can you maybe try debugging on db side? Capture the RPC call to export
as well as the query hitting DB.

DanB
> --
> You received this message because you are subscribed to the Google
> Groups "CGRateS" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to cgrates+u...@googlegroups.com.
> To post to this group, send email to cgr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/cgrates/916a404b-f996-48a9-8929-fc15b07a505c%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.

Michael Klimenko

unread,
Oct 31, 2017, 8:16:40 AM10/31/17
to CGRateS
Hello Dan,
I am not sure I understand you.
When I execute action with *cdrlog type, new row is inserted into storDB.
Config is set to auto export, but nothing is getting out.
How can I troubleshoot  "online_cdr_exports" ?

DanB

unread,
Oct 31, 2017, 8:30:12 AM10/31/17
to cgr...@googlegroups.com
Michael,

I get you now.
You are right the *cdrlog does not go through online exports (only
query-able CDRs since the CDR is dumped directly into StorDB).
Please open an issue and will fix it at some point.

Thanks,
DanB
> msgid/cgrates/534c70a3-b7c1-4d6b-bb8e-6f80871836a3%40googlegroups.com

Michael Klimenko

unread,
Oct 31, 2017, 9:40:22 AM10/31/17
to CGRateS

Michael Klimenko

unread,
Nov 1, 2017, 8:21:00 AM11/1/17
to CGRateS
Hello Dan,
I was trying to get export manually, but it fails with the following :

Nov  1 13:17:48 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 920b90ef048691c250e1fe7b973dbebba5fec7a9 and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509535066.*http_json_cdr: http: no Host in request URL

I am not sure I understand this error, as the same config works good with auto-export.

Here is the config:

"cdrlog_exports": {
                "export_format": "*http_json_cdr",     // exported CDRs format <*file_csv|*file_fwv|*http_post|*http_json_cdr|*http_json_map|*amqp_json_cdr|*amqp_json_map>
                "export_path": "http://10.224.248.81:8000",     // path where the exported CDRs will be placed
                "cdr_filter": "Source(*cdrlog)",        // filter CDRs exported by this template

DanB

unread,
Nov 1, 2017, 10:27:35 AM11/1/17
to cgr...@googlegroups.com
Hi Michael,

For some reason I can see 3 backslashes in your url. Is that
intentional?

DanB

On Wed, 2017-11-01 at 05:20 -0700, Michael Klimenko wrote:
> Hello Dan,
> msgid/cgrates/e06b4ee5-4875-460c-a430-0a54abced894%40googlegroups.com

Michael Klimenko

unread,
Nov 1, 2017, 3:08:47 PM11/1/17
to CGRateS
Hello Dan,
No, see the config, it states only double slashes and works with auto-export.
This log is only when I try to export_cdrs manually.

DanB

unread,
Nov 3, 2017, 11:56:03 AM11/3/17
to cgr...@googlegroups.com
Hi Michael,

Can you post a capture of your API call to export? There should be no
difference between online and offline export, except the parameters
hitting export routine.

Thanks,
DanB
> msgid/cgrates/48cabea9-a961-4312-84dd-e3e41e448c83%40googlegroups.com

Michael Klimenko

unread,
Nov 5, 2017, 5:39:29 AM11/5/17
to CGRateS
Hello Dan,

Here is the capture:
=================
#####
T2017/11/0512: 31: 42.234217127.0.0.1: 39304->127.0.0.1: 2012[AP]{
"method": "ApierV1.ExportCDRs",
"params": [{
"ExportTemplate": "cdrlog_exports",
"ExportFormat": "*http_json_cdr",
"ExportPath": "http://10.224.248.81:8000",
"Synchronous": null,
"Attempts": null,
"FieldSeparator": null,
"UsageMultiplyFactor": null,
"CostMultiplyFactor": null,
"ExportID": null,
"ExportFileName": null,
"RoundingDecimals": null,
"Verbose": false,
"CGRIDs": null,
"NotCGRIDs": null,
"RunIDs": null,
"NotRunIDs": null,
"OriginHosts": null,
"NotOriginHosts": null,
"Sources": null,
"NotSources": null,
"ToRs": null,
"NotToRs": null,
"RequestTypes": null,
"NotRequestTypes": null,
"Directions": null,
"NotDirections": null,
"Tenants": null,
"NotTenants": null,
"Categories": null,
"NotCategories": null,
"Accounts": null,
"NotAccounts": null,
"Subjects": null,
"NotSubjects": null,
"DestinationPrefixes": null,
"NotDestinationPrefixes": null,
"Suppliers": null,
"NotSuppliers": null,
"DisconnectCauses": null,
"NotDisconnectCauses": null,
"Costs": null,
"NotCosts": null,
"ExtraFields": null,
"NotExtraFields": null,
"OrderIDStart": null,
"OrderIDEnd": null,
"SetupTimeStart": "",
"SetupTimeEnd": "",
"AnswerTimeStart": "",
"AnswerTimeEnd": "",
"CreatedAtStart": "",
"CreatedAtEnd": "",
"UpdatedAtStart": "",
"UpdatedAtEnd": "",
"MinUsage": "",
"MaxUsage": "",
"MinPDD": "",
"MaxPDD": "",
"MinCost": null,
"MaxCost": null,
"Limit": null,
"Offset": null,
"SearchTerm": ""
}],
"id": 0
}

##
T2017/11/0512: 31: 42.522162127.0.0.1: 2012->127.0.0.1: 39304[AP]{
"id": 0,
"result": {
"ExportedPath": "",
"TotalRecords": 0,
"TotalCost": 0,
"FirstOrderID": 0,
"LastOrderID": 0,
"ExportedCGRIDs": null,
"UnexportedCGRIDs": null
},
"error": null
}
=================


And the log :
================
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 885a0c56a80b5e0e59bc0d2fbb03c7ebe4d6213f and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: f694d2980fef487fa0800e70f52e5e2f11c7e824 and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: d4535542c7450b6632f99c7871594b699e4e513d and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 9c133536776a7a8091738783e89ce0e7062e44fc and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: a167179f9fbd5b42bfaf4d04be81dd135acca738 and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 667ea85bcc181b26820a6b492fe5cacf4bd79933 and runid: *topup, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 2bc36062d76b66399cb6571ac9f77d7640dbd929 and runid: *topup, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 4247f5a5a132fda0705465c5988553cc4f25a4c9 and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: da0289a24c0bf22abc642376ecc8a58329d1c85c and runid: *topup, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 27e7d43c5ffcbbfee5876e71779b0bc5eec4b744 and runid: *topup, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: dce48665b51a02ad34085e2b028f9d75eee4da6c and runid: *topup_reset, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
Nov  5 12:31:43 cgr-ptk-node-02 CGRateS[10262]: <CDRE> Cannot export CDR with CGRID: 6416f78a3da6f4ffa49fc3b231504de5aef4a8d9 and runid: *topup, error: Post http:///10.224.248.81:8000/cdre_1509877902.*http_json_cdr: http: no Host in request URL
================


I have tested different templates, those that works in auto-export , and it is the same log appears.
So, I think it is some param that I miss in manually running the following command:

cgr-console 'cdrs_export ExportTemplate="cdrlog_exports" ExportFormat="*http_json_cdr" ExportPath="http://10.224.248.81:8000"'


Here is the template config :
======================
"cdrlog_exports": {
                "export_format": "*http_json_cdr",     // exported CDRs format <*file_csv|*file_fwv|*http_post|*http_json_cdr|*http_json_map|*amqp_json_cdr|*amqp_json_map>
                "export_path": "http://10.224.248.81:8000",     // path where the exported CDRs will be placed
                "cdr_filter": "Source(*cdrlog)",        // filter CDRs exported by this template
                "usage_multiply_factor": {
                        "*any": 1                                                                       // multiply usage based on ToR field or *any for all
                },
                "content_fields": [                                                             // template of the exported content fields
                        {"tag":"CGRID", "type": "*composed", "value": "CGRID"},
                        {"tag":"RunID", "type": "*composed", "value": "RunID"},
                        {"tag":"TOR", "type": "*composed", "value": "ToR"},
                        {"tag":"OriginID", "type": "*composed", "value": "OriginID"},
                        {"tag":"RequestType", "type": "*composed", "value": "RequestType"},
                        {"tag":"Direction", "type": "*composed", "value": "Direction"},
                        {"tag":"Tenant", "type": "*composed", "value": "Tenant"},
                        {"tag":"Category", "type": "*composed", "value": "Category"},
                        {"tag":"Account", "type": "*composed", "value": "Account"},
                        {"tag":"Subject", "type": "*composed", "value": "Subject"},
                        {"tag":"Destination", "type": "*composed", "value": "Destination"},
                        {"tag":"SetupTime", "type": "*composed", "value": "SetupTime", "layout": "2006-01-02T15:04:05Z07:00"},
                        {"tag":"AnswerTime", "type": "*composed", "value": "AnswerTime", "layout": "2006-01-02T15:04:05Z07:00"},
                        {"tag":"Usage", "type": "*composed", "value": "Usage"},
                        {"tag":"Cost", "type": "*composed", "value": "Cost", "rounding_decimals": 4},
                ],
        }
======================

Thanks,
Michael

DanB

unread,
Nov 6, 2017, 5:44:28 AM11/6/17
to cgr...@googlegroups.com
Michael,

If you use ExportTemplate, try not to populate ExportFormat and
ExportPath. Did not check it yet why the ExportPath is coming in
wrongly but in this particular case is not needed since it overwrites
the settings in ExportTemplate.

DanB

On Sun, 2017-11-05 at 02:39 -0800, Michael Klimenko wrote:
> Hello Dan,
>
> > > > gle.
> > > > > > > com/d/
> > > > > > > > msgid/cgrates/534c70a3-b7c1-4d6b-bb8e-
> > > > > > > 6f80871836a3%40googlegroups.com
> > > > > > > > .
> > > > > > > > For more options, visit https://groups.google.com/d/opt
> > out.
> > > >
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the
> > > > Google
> > > > > Groups "CGRateS" group.
> > > > > To unsubscribe from this group and stop receiving emails
> > from
> > > > it,
> > > > > send an email to cgrates+u...@googlegroups.com.
> msgid/cgrates/4de37fa0-3054-412a-be4b-ec98e1786ae7%40googlegroups.com

Michael Klimenko

unread,
Nov 6, 2017, 8:44:04 AM11/6/17
to CGRateS
Dan,
I tried it both ways, but it is still the same error.

Thanks,
Michael

DanB

unread,
Nov 6, 2017, 8:45:29 AM11/6/17
to cgr...@googlegroups.com
Michael,

OK, please attach me both your API call and export template.

Thanks,
DanB
> > > > > > gle.
> > > > > > > > > com/d/
> > > > > > > > > > msgid/cgrates/534c70a3-b7c1-4d6b-bb8e-
> > > > > > > > > 6f80871836a3%40googlegroups.com
> > > > > > > > > > .
> > > > > > > > > > For more options, visit https://groups.google.com/d
> > /opt
> > > > out.
> > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to
> > the
> > > > > > Google
> > > > > > > Groups "CGRateS" group.
> > > > > > > To unsubscribe from this group and stop receiving emails
> > > > from
> > > > > > it,
> > > > > > > send an email to cgrates+u...@googlegroups.com.
> > > > > > > To post to this group, send email to cgr...@googlegroups.
> > com.
> > > >
> > > > > > > To view this discussion on the web visit https://groups.g
> > oogl
> > > > e.co
> > > > > > m/d/
> > > > > > > msgid/cgrates/e06b4ee5-4875-460c-a430-
> > > > > > 0a54abced894%40googlegroups.com
> > > > > > > .
> > > > > > > For more options, visit https://groups.google.com/d/optou
> > t.
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the
> > > > Google
> > > > > Groups "CGRateS" group.
> > > > > To unsubscribe from this group and stop receiving emails
> > from
> > > > it,
> > > > > send an email to cgrates+u...@googlegroups.com.
> > > > > To post to this group, send email to cgr...@googlegroups.com.
> >
> > > > > To view this discussion on the web visit https://groups.googl
> > e.co
> > > > m/d/
> msgid/cgrates/73764da1-9e71-4d60-897f-8950fe1a0ca9%40googlegroups.com

Michael Klimenko

unread,
Nov 6, 2017, 8:55:14 AM11/6/17
to CGRateS
Hiello Dan,

From Config:

 "cdrlog_exports": {
                "export_format": "*http_json_cdr",     // exported CDRs format <*file_csv|*file_fwv|*http_post|*http_json_cdr|*http_json_map|*amqp_json_cdr|*amqp_json_map>
                "export_path": "http://10.224.248.81:8000",     // path where the exported CDRs will be placed
                "cdr_filter": "Source(*cdrlog)",        // filter CDRs exported by this template
                "usage_multiply_factor": {
                        "*any": 1                                                                       // multiply usage based on ToR field or *any for all
                },
                "content_fields": [                                                             // template of the exported content fields
                        {"tag":"CGRID", "type": "*composed", "value": "CGRID"},
                        {"tag":"RunID", "type": "*composed", "value": "RunID"},
                        {"tag":"TOR", "type": "*composed", "value": "ToR"},
                        {"tag":"OriginID", "type": "*composed", "value": "OriginID"},
                        {"tag":"RequestType", "type": "*composed", "value": "RequestType"},
                        {"tag":"Direction", "type": "*composed", "value": "Direction"},
                        {"tag":"Tenant", "type": "*composed", "value": "Tenant"},
                        {"tag":"Category", "type": "*composed", "value": "Category"},
                        {"tag":"Account", "type": "*composed", "value": "Account"},
                        {"tag":"Subject", "type": "*composed", "value": "Subject"},
                        {"tag":"Destination", "type": "*composed", "value": "Destination"},
                        {"tag":"SetupTime", "type": "*composed", "value": "SetupTime", "layout": "2006-01-02T15:04:05Z07:00"},
                        {"tag":"AnswerTime", "type": "*composed", "value": "AnswerTime", "layout": "2006-01-02T15:04:05Z07:00"},
                        {"tag":"Usage", "type": "*composed", "value": "Usage"},
                        {"tag":"Cost", "type": "*composed", "value": "Cost", "rounding_decimals": 4},
                ],
        }


golan@cgr-ptk-node-02:~$ cgr-console 'cdrs_export ExportTemplate="cdrlog_exports"'
{
 "ExportTemplate": null,
 "ExportFormat": null,
 "ExportPath": null,



NGREP:

####
T 2017/11/06 15:51:33.109917 127.0.0.1:39475 -> 127.0.0.1:2012 [AP]
{"method":"ApierV1.ExportCDRs","params":[{"ExportTemplate":"cdrlog_exports","ExportFormat":null,"ExportPath":null,"Synchronous":null,"Attempts":null,"FieldSeparator":null,"UsageMultiplyFactor":null,"CostMultiplyFactor":null,"ExportID":null,"ExportFileName":null,"RoundingDecimals":null,"Verbose":false,"CGRIDs":null,"NotCGRIDs":null,"RunIDs":null,"NotRunIDs":null,"OriginHosts":null,"NotOriginHosts":null,"Sources":null,"NotSources":null,"ToRs":null,"NotToRs":null,"RequestTypes":null,"NotRequestTypes":null,"Directions":null,"NotDirections":null,"Tenants":null,"NotTenants":null,"Categories":null,"NotCategories":null,"Accounts":null,"NotAccounts":null,"Subjects":null,"NotSubjects":null,"DestinationPrefixes":null,"NotDestinationPrefixes":null,"Suppliers":null,"NotSuppliers":null,"DisconnectCauses":null,"NotDisconnectCauses":null,"Costs":null,"NotCosts":null,"ExtraFields":null,"NotExtraFields":null,"OrderIDStart":null,"OrderIDEnd":null,"SetupTimeStart":"","SetupTimeEnd":"","AnswerTimeStart":"","AnswerTimeEnd":"","CreatedAtStart":"","CreatedAtEnd":"","UpdatedAtStart":"","UpdatedAtEnd":"","MinUsage":"","MaxUsage":"","MinPDD":"","MaxPDD":"","MinCost":null,"MaxCost":null,"Limit":null,"Offset":null,"SearchTerm":""}],"id":0}

##
T 2017/11/06 15:51:33.422010 127.0.0.1:2012 -> 127.0.0.1:39475 [AP]
{"id":0,"result":{"ExportedPath":"","TotalRecords":0,"TotalCost":0,"FirstOrderID":0,"LastOrderID":0,"ExportedCGRIDs":null,"UnexportedCGRIDs":null},"error":null}

####
Message has been deleted

Teofil Voivozeanu

unread,
Nov 7, 2017, 6:48:47 AM11/7/17
to CGRateS
Hello Michael , 
We fixed this issue on master branch . 
Do the test again and tell us your result . 
Thanks,
TeoV

Michael Klimenko

unread,
Nov 7, 2017, 7:45:25 AM11/7/17
to CGRateS
Hello Teofil,
Confirmed working !

Thanks a lot,
Michael

Michael Klimenko

unread,
Nov 21, 2017, 4:05:18 AM11/21/17
to CGRateS
Hello Teofil,
Just to make sure, confirmed CDR exports are OK, when exported manually within cgr-console.
Trying to get cdrlog to auto-export with "online_cdr_exports" is still not good.

However, that can be achieved by running a cronjob (triggering cdr_exports), 
please, let me know if that is correct way to proceed, or is going to be fixed in near future.

Thanks,
Michael

Teofil Voivozeanu

unread,
Nov 22, 2017, 6:35:21 AM11/22/17
to CGRateS
Hi Michael,
Attaching here the answer from the issue answered on github:

Thanks,
TeoV
Reply all
Reply to author
Forward
0 new messages