Can Stats periodically dump the data into StoreDB for later usage?

[ryan embgrets]

Greetings, 

I was wondering can Stats component periodically dump the data into StoreDB, 

ie, i would like to see the *24 hours, but if Stats can dump that stats upon some time into StoreDB and only keep the 24 hours data into cache, i would be available to query the stats for longer(from last week) period, to show clients some graphs

Ryan. 

[Alexandru Tripon]

Hi Ryan,

You can configure how often stats dumps the data in StorDB using strore_interval(https://github.com/cgrates/cgrates/blob/master/config/config_defaults.go#L602).

Here you can find the documentation for the StatS: https://cgrates.readthedocs.io/en/v0.10/stats.html.

Thanks,

Trial97

--
You received this message because you are subscribed to the Google Groups "CGRateS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cgrates+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/78fe728a-5d79-40ca-8c01-1172d40f08e9%40googlegroups.com.

[ryan embgrets]

Thanks, for the reply, 

But does not above settings save the stats into dataDB and only for backup purpose? 

"store_interval": "", // dump cache regularly to dataDB, 0 - dump at start/shutdown: <""|$dur>

I wanted to have it stored into storeDB, so we can query them later. 

Ryan.

To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/CAAkCrRP8nOTeZHUWdM%3Di76iWg4EOCoksTSUKbfzHZtg8CZoTNQ%40mail.gmail.com.

[Alexandru Tripon]

Hi Ryan,

In this case we recommend an external component and querying regularly stats and saving in your own environment.

On the other hand, the new subsystem we are working on, EEs, will be able to fix automatic exports too.

You can open an issue on Github and we will let you know when is implemented.

Thanks,

Trial97 

To view this discussion on the web visit https://groups.google.com/d/msgid/cgrates/CAJ_X8k5Epj8mfh_E6uyF8U%2BKw72BT02YE2inHLW87bEc7qBy0w%40mail.gmail.com.

[Sami Ali]

Hello Cgrates Team,

Kindly let me know if this is already done, or not as it's an important feature ..

Thank you in advance

Sami

[Armir Veliaj]

Hi Sami,

The EEs is already implemented and functional since a long time ago, you can check about it in our docs:
https://cgrates.readthedocs.io/en/latest/ees.html#ees.

Thanks,
Armir

[Sami Ali]

Thank you Armir,

I checked the EES and tested some use cases with the below :

"stats": {
"enabled": true,
"ees_conns": ["*localhost"],
"ees_exporter_ids": ["exporter1"]
},

"ees": {
"enabled": true,
"exporters": [{
"id": "exporter1",
"type": "*file_csv",
"export_path": "/tmp/testCSV",
"attempts": 1,
"synchronous": true,
"field_separator": ",",
"fields": [{
"tag": "StatID","path": "*exp.StatID",
"type": "*variable","value": "~*req.StatID"
},
{
"tag": "TotalCallDuration","path": "*exp.TotalCallDuration",
"type": "*variable","value": "~*req.Metrics.*tcd"
},
{
"tag": "TotalCallCost","path": "*exp.TotalCallCost",
"type": "*variable","value": "~*req.Metrics.*tcc"
},
{
"tag": "NumberOfCDRs","path": "*exp.NumberOfCDRs",
"type": "*variable","value": "~*req.Metrics.*sum#1"
}
]
}]
},

and my stats as below :

{

"method": "ApierV1.SetStatQueueProfile",

"params": [

{

"Tenant": "XXXXX",

"ID": "STAT_Fraud",

"FilterIDs": null,

"ActivationInterval": {

"ActivationTime": "2024-01-31T15:00:00Z",

"ExpiryTime": "0001-01-01T00:00:00Z"

},

"QueueLength": 10000000,

"TTL": -1,

"MinItems": 0,

"Metrics": [

{

"FilterIDs": null,

"MetricID": "*tcd"

},

{

"FilterIDs": null,

"MetricID": "*tcc"

},

{

"FilterIDs": null,

"MetricID": "*asr"

},

{

"FilterIDs": null,

"MetricID": "*sum#1"

},

{

"FilterIDs": null,

"MetricID": "*acd"

}

],

"Stored": true,

"Blocker": true,

"Weight": 100,

"ThresholdIDs": ["THProfile_FraudDetection"]

}

]

}

So i can get all stats as below :

{

"method": "StatSv1.GetQueueStringMetrics",

"params": [

{

"Tenant": "sabafon.com",

"id": "STAT_Fraud"

}

]

}

response:

{

"id": null,

"result": {

"*acd": "14.14706s",

"*asr": "80%",

"*sum#1": "28",

"*tcc": "N/A",

"*tcd": "8m1s"

},

"error": null

}

and i found the export is partially executed by creating the files ;) in the specific path {{put all are empty}} , Could you please kindly guide what is missing in my config?
root@9d88f068bb51:/tmp/testCSV# ls -ltrh
total 84K
-rw-r--r-- 1 root root 1 Mar 12 07:08 exporter1_92b84d2.csv
-rw-r--r-- 1 root root 1 Mar 12 07:09 exporter1_5b1e14a.csv
-rw-r--r-- 1 root root 1 Mar 12 07:55 exporter1_509ab14.csv
-rw-r--r-- 1 root root 1 Mar 12 07:59 exporter1_ded2fdd.csv
-rw-r--r-- 1 root root 1 Mar 12 08:00 exporter1_b2bf428.csv
-rw-r--r-- 1 root root 1 Mar 12 08:00 exporter1_59d668d.csv
-rw-r--r-- 1 root root 1 Mar 12 08:01 exporter1_3b7b28c.csv
-rw-r--r-- 1 root root 1 Mar 12 08:01 exporter1_cf2378b.csv
-rw-r--r-- 1 root root 1 Mar 12 08:01 exporter1_698b66c.csv
-rw-r--r-- 1 root root 1 Mar 12 08:10 exporter1_127d834.csv
-rw-r--r-- 1 root root 1 Mar 12 08:10 exporter1_d6d2c9d.csv
-rw-r--r-- 1 root root 1 Mar 12 08:11 exporter1_490dd97.csv
-rw-r--r-- 1 root root 1 Mar 12 08:11 exporter1_95b78f0.csv
-rw-r--r-- 1 root root 1 Mar 12 08:13 exporter1_afedf6c.csv
-rw-r--r-- 1 root root 1 Mar 12 08:15 exporter1_80f7344.csv
-rw-r--r-- 1 root root 1 Mar 12 08:21 exporter1_bc4cafb.csv
-rw-r--r-- 1 root root 1 Mar 12 08:21 exporter1_a950b15.csv
-rw-r--r-- 1 root root 1 Mar 12 08:22 exporter1_823ab5b.csv
-rw-r--r-- 1 root root 1 Mar 12 08:23 exporter1_f133329.csv
-rw-r--r-- 1 root root 1 Mar 12 08:23 exporter1_2927db8.csv
-rw-r--r-- 1 root root 1 Mar 12 08:26 exporter1_1b86021.csv

[Armir Veliaj]

Hi Sami,

Please open the issue on our GitHub, and we can update from there for any fixes that may be needed,

Thanks,
Armir

[selim jabou]

Hello, i am currently working on this as well, i had the same problem, but i fixed it. the first step is to make this commands : 

  chown -R cgrates:cgrates   /tmp/testCSV 
  sudo chmod -R 755   /tmp/testCSV 

there is another problem, if the files are empty, except the headers, the problem is that the exports stats in the cache, you can verify it by reloading cgrates, if it is the case the next export would work. if this is your case i got it fixed by adding these in config file of cgrates : 

"cache": {
            "*file_csv": {
                "limit": -1,
                "ttl": "1s",
                "static_ttl": false
            }
        },

so it would look like this : 

"ees": {
        "enabled": true,
        "cache": {
            "*file_csv": {
                "limit": -1,
                "ttl": "1s",
                "static_ttl": false
            }
        },
        "exporters": [
            {....}]

[Armir Veliaj]

Hi Selim,

There was a GitHub issue opened for that case: https://github.com/cgrates/cgrates/issues/4689. After the implementation, there is no need to reload CGRateS, and it should work fine now as the changes have been applied to the default config.

Thanks,
Armir