Blueflood TS DB Opinion

[Learner]

Hi All,

We are in the process of evaluating various Open source, highly available  time series databases like InfluxDB, KairosDB, Bluefood, Cassandra etc. for storing system events which will have many attributes(including texts and numeric values). The events will be stored for years together.

Need the expert advise on Blueflood to see if Blueflood fits the requirement in hand. Anyone is using Blueflood for such use cases. Are there any known limitations which might affect such a use case?

Thanks in advance for your response.

Regards,

Lp

[Chandra Addala]

Blueflood is meant to be used to store metrics, not events. I dont know if other time series databases does events, but you can definitely program Cassandra to store events. Thats my 2 cents.

[Jonathan Creasy]

You can store events in OpenTSDB as annotations and Graphite as events. 

Have you considered just using Elasticsearch? Tools like that might be a closer fit than timeseries databases.

Depending on the structure of your events, and perhaps you can proivde a few examples, you might just be talking about tags. In that case, OpenTSDB and InfluxDB are going to be good choices. I don't think Blueflood supports tags on a datapoint. (If this is wrong, I'll be very interested, and I'm sure someone will correct me)

An example of an event in InfluxDb's wire protocol:

event.errors,host=app103,dc=us-east-1,application=mysql sum=15,min=2,max,4 1,497,456,122,000

And OpentTSDB:

event.errors.sum 1497456122000 15 host=app103 dc=us-east-1 application=mysql

event.errors.min 1497456122000 2 host=app103 dc=us-east-1 application=mysql

event.errors.max 1497456122000 4 host=app103 dc=us-east-1 application=mysql

  

--
You received this message because you are subscribed to the Google Groups "Blueflood Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blueflood-discuss+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/blueflood-discuss.
To view this discussion on the web visit https://groups.google.com/d/msgid/blueflood-discuss/5d2daafc-8d86-4dab-9571-b6df54086eb6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Learner]

Many thanks @Jonathan and @Chandra for the responses. Would just like to add more details around the exact requirements.

We have two types of Time Series Data. One is "Metrics" and others is "Event"

Metrics: 

Sample Metrics:

source:sensor1,timestamp:<ts>,param1:val1,param2:val2,param3:val3,...

Operations on metrics:

1> Read raw metrics by sensor and timestamp

2> Aggregate sensor data by computing average, min, max values of the parameters

Events: 

Events look like:

source:source1,timestamp:<ts>,event_type:<string>,event_name:<string>,event_description:<string>,...

Operations on events:

1> Read events by source or any other attribute

2> Filter Events bases on attributes

3> Retrieve most recent events from a source

Would need help in understanding if blueflood would be a good choice for both of the use cases or only for metrics.

Regards,

Lp

On Wednesday, June 14, 2017 at 9:34:58 PM UTC+5:30, Jonathan Creasy wrote:

You can store events in OpenTSDB as annotations and Graphite as events. 

Have you considered just using Elasticsearch? Tools like that might be a closer fit than timeseries databases.

Depending on the structure of your events, and perhaps you can proivde a few examples, you might just be talking about tags. In that case, OpenTSDB and InfluxDB are going to be good choices. I don't think Blueflood supports tags on a datapoint. (If this is wrong, I'll be very interested, and I'm sure someone will correct me)

An example of an event in InfluxDb's wire protocol:

event.errors,host=app103,dc=us-east-1,application=mysql sum=15,min=2,max,4 1,497,456,122,000

And OpentTSDB:

event.errors.sum 1497456122000 15 host=app103 dc=us-east-1 application=mysql

event.errors.min 1497456122000 2 host=app103 dc=us-east-1 application=mysql

event.errors.max 1497456122000 4 host=app103 dc=us-east-1 application=mysql

  

On Wed, Jun 14, 2017 at 9:24 AM, Chandra Addala <chandra...@gmail.com> wrote:

Blueflood is meant to be used to store metrics, not events. I dont know if other time series databases does events, but you can definitely program Cassandra to store events. Thats my 2 cents.

On Wednesday, June 14, 2017 at 1:48:37 AM UTC-5, Learner wrote:

Hi All,

We are in the process of evaluating various Open source, highly available  time series databases like InfluxDB, KairosDB, Bluefood, Cassandra etc. for storing system events which will have many attributes(including texts and numeric values). The events will be stored for years together.

Need the expert advise on Blueflood to see if Blueflood fits the requirement in hand. Anyone is using Blueflood for such use cases. Are there any known limitations which might affect such a use case?

Thanks in advance for your response.

Regards,

Lp

--
You received this message because you are subscribed to the Google Groups "Blueflood Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blueflood-disc...@googlegroups.com.

Visit this group at https://groups.google.com/group/blueflood-discuss.

[Chandrasekhar A]

Using blueflood:

For metrics:

Based on the example you provided, you have to store the sensor data as below. Basically the structure is flat and in the below example sensor1.param1 is the metric name and val1 is the metric value which changes over time.

source:sensor1.param1:val1:,timestamp:<ts>,

source:sensor1.param2:val2:,timestamp:<ts>,

source:sensor1.param3:val3:,timestamp:<ts>,

They do get aggregated into 5min, 20min, 60min, 240min, 1440min resolutions and based on the metric type(counter/guage etc) you will have min, max, avg etc values for the metric. 

For events:

Blueflood does not do events as far as I know.  May be you can hack it, build something on top of it but thats a long shot.

Thanks,

Chandra

[Tutorials Java]

Many Thanks @Chandrasekhar it helps.