Reading CDR from external MYSQL DB to CGRATES and push to billing engine

[Rakesh Sahadevan]

HI Dan,

Could you please advise the best way to fetch CDR's from external DB and send to CGRATES for rating purpose.

 

Summary of our question is

---------------------------------------- 

1) How CGRates can read CDR's from external(vendor) mysql DB and apply rates and push to our billing engine DB?.

2) What is the best feasible way in CGRates ?.

We are doing a POC to replace our current rating engine with CGRATES.

Thanks And Regards

Rakesh

[DanB]

Hi Rakesh,

We don't read but you should create a script which should "push" CDRs
from your old DB into CGRateS (eg: using this API call:
https://godoc.org/github.com/cgrates/cgrates/apier/v1#CdrsV1.ProcessExternalCDR).
Once the CDR hits the CDRS of CGRateS (via ProcessExternalCDR) we will
rate and store(optionally) or/and "export" it in real-time to your CDR
server (see as sample this configuration for online exports -
https://github.com/cgrates/cgrates/blob/master/data/conf/samples/cdrsonexpmaster/cdrsreplicationmaster.json#L24).

Feel free to come up with any additional questions.

DanB

[Rakesh Sahadevan]

Hi Dan,

Thank you Dan for the quick reply. Will test the performance after exporting and update back(We need to insert again to a DB).

Regards

Rakesh 

[DanB]

Rakesh,

If you are looking to just update your CDR, I suggest checking *http_json_map transport which gives you the opportunity to customize the exported CDR to you (so you can just add the fields you are interested about). Should be a bit faster than the *http_json.

DanB

--
You received this message because you are subscribed to the Google Groups "CGRateS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cgrates+u...@googlegroups.com.
To post to this group, send email to cgr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/d7dbe198-d3a3-4bcf-af79-bca1b21c1274%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rakesh Sahadevan]

HI Dan,

Please let me know any sample config file you have for reference to export as CSV  from CGRATES after inputting ProcessExternalCDR 

Below are my questions

1) Could I use "cdrsreplicationmaster.json" by changing the format  to CSV format  

2) Please let me know the changes required below.

"cdre": {

"http_localhost": {

"export_format": "*http_post",

"export_path": "http://127.0.0.1:12080/cdr_http",

"cdr_filter": "RunID(*default);OriginID(httpjsonrpc1)",

Regards

Rakesh

[DanB]

Hi Rakesh,

Answers inline ...

On 04.03.2017 16:28, Rakesh Sahadevan wrote:

HI Dan,

Please let me know any sample config file you have for reference to export as CSV  from CGRATES after inputting ProcessExternalCDR

Before all, make sure you understand the difference between:
    * online exports - CDRs are exported in real-time, you get the calculated CDR one by one (correlated with the input). For this to work all you need to do is configure the "online_cdr_exports" section in cdrs.
    * offline exports - CDRs are exported on demand (by calling an API). This works only when you store the CDRs in CGRateS and it is more like the old style exports where you expect your billing system to give you the CDRs at certain intervals (eg: end of the month billing).

I personally suggested you the first option, online exports since you get performance as well as possibility to define your own storage strategy.

Below are my questions

1) Could I use "cdrsreplicationmaster.json" by changing the format  to CSV format 

Not really. If you use online cdrs replication with .csv file, it will generate a file for each CDR you feed in (I find it quite unoptimized solution).
If you are still looking for .csv exports, all you need to use this API (which will use default config, pointing towards .csv export):
https://godoc.org/github.com/cgrates/cgrates/apier/v1#ApierV1.ExportCDRs

2) Please let me know the changes required below.

"cdre": {

"http_localhost": {

"export_format": "*http_post",

"export_path": "http://127.0.0.1:12080/cdr_http",

"cdr_filter": "RunID(*default);OriginID(httpjsonrpc1)",

I need to understand what are you trying to export but please see above.

DanB

To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/0b8ef0c5-ad20-494f-85b8-ab9835627885%40googlegroups.com.

[Rakesh Sahadevan]

HI Dan,

Thanks for the response. We are currently have online Rating. Immediately CDR's are processing and reduce the balance in customer account. Customer can opt for pre-paid OR post-paid. In both cases we are rating Online. 

Currently we are reading 1000 records in each chunk and pushing to rating engine(Java based) and will internally insert to billing Database. Every month we are processing approximate 50 million calls .

New planning approach with CgRates(Please advise. Below may be completely wrong. Sorry. I am really new to CGRates and questions are bad.)

1) We will be reading 1000 records in each chunk and pushing to CGR rating engine through the API ProcessExternalCDR 

2) What is the best option for export from CgRates. 

            a) Can we  explicitly call ApierV1.ExportCsvCdrs after each chunk of  ProcessExternalCDR 

                         OR

            b)  Can we configure online cdr_exports in cdrs. ?.

Regards

Rakesh

Regards

Rakesh

[DanB]

Hi Rakesh,

I would go with online_cdr_exports. The CDR should be sent back to you
and not longer needing to be stored in CGRateS since you run anyway an
external database (you can disable storing if you like here:
https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L146).


All you need to do after is use the online export data to update your
database with cost.

On the other hand I would recommend you to use *http_json_map as
transport for your online cdr exports (or *amqp_json_map if you prefer
to get the exports via RabbitMQ) so you can customize the data exported
(and not to process useless CDR fields if you only need the cost).

DanB

[Rakesh Sahadevan]

HI Dan,

My question is 

a) Do CGRates default have a amqp server. ?. If I am using amqp_json_map, I just need to create a RabbitMQ client and consume online processed CDRs'.

"cdre": {

"amqp_localhost": {

"export_format": "*amqp_json_map",

"export_path": "amqp://guest:guest@localhost:5672/?queue_id=cgrates_cdrs",

Regards

Rakesh

[DanB]

Hi Rakesh,

Answers inline ...

On 28.03.2017 04:49, Rakesh Sahadevan wrote:

HI Dan,

My question is 

a) Do CGRates default have a amqp server. ?

Nope, you should setup RabbitMQ one.

. If I am using amqp_json_map, I just need to create a RabbitMQ client and consume online processed CDRs'.

Yes, but from RabbitMQ. CGRateS will post them there (based on your config bellow)

DanB

"cdre": {

"amqp_localhost": {

"export_format": "*amqp_json_map",

"export_path": "amqp://guest:guest@localhost:5672/?queue_id=cgrates_cdrs",

Regards

Rakesh

On Sunday, March 5, 2017 at 10:23:44 PM UTC+4, DanB wrote:

Hi Rakesh,

I would go with online_cdr_exports. The CDR should be sent back to you
and not longer needing to be stored in CGRateS since you run anyway an
external database (you can disable storing if you like here:
https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L146).


All you need to do after is use the online export data to update your
database with cost.

On the other hand I would recommend you to use *http_json_map as
transport for your online cdr exports (or *amqp_json_map if you prefer
to get the exports via RabbitMQ) so you can customize the data exported
(and not to process useless CDR fields if you only need the cost).

DanB

--
You received this message because you are subscribed to the Google Groups "CGRateS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cgrates+u...@googlegroups.com.
To post to this group, send email to cgr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/b66cd202-0b42-4ba9-a1f1-fb5a1dbea973%40googlegroups.com.

[వెంకటేష్ ఎన్నల]

Hi Dan,

How does cgrates behave in case of errors.

For Example: for any reason if the connection for rabbitmq is down a min .. does cgrates try to send them again ?

For Example: if cgrates crashes .. how does it deal with the cdrs which are not already exported.

I am looking to implement this in our current system ( currently we are using http ) to achieve high throughput  and I was thinking about these failure scenarios as we have some very strict guidelines about not loosing any cdrs.

Thanks.

[Dan Christian Bogos]

Hi enna...@gmail.com,

There are various scenarios of failover(and stages where you can loose
the CDR):

1. If the remote RabbitMQ does not reply, we try a number of times
(configurable here:
https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L38
), then we dump the CDR to filesystem (configurable here:
https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L39
). You can then "replay" the failed attempts via this API:
https://godoc.org/github.com/cgrates/cgrates/apier/v1#ApierV1.ReplayFailedPosts

2. If CGRateS goes down (restart or panic), you can restart the export
process via API (the CDR is stored to StorDB before exporting). One API
you can use is this one:
https://godoc.org/github.com/cgrates/cgrates/apier/v1#ApierV1.ExportCDRs

3. If the CGRateS panics during receiving the CDR from you, you will
not get the OK from CGRateS and in that case you are responsible about
failover yourself.

4. In case of CDRc sending the CDR to CDR server and the last one is
not able to process the CDR, CDRc will try the next connection
configured (you can configure there a list for failover):
https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L234

I hope this helps.

DanB

[వెంకటేష్ ఎన్నల]

Awesome .. thanks.