real time data

[woomin park]

I try to use real-time example which you provided in the metatron blog.

(with metatron the lastest release version)

generate_sample_data.py

import sys

import json

import math

from datetime import datetime

from time import sleep

from kafka import KafkaProducer

#producer = KafkaProducer(bootstrap_servers='localhost:9092')

producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))

list_category = ['10', '20', '30'];

for i in range(0, 100000) :

        for cur_cate in list_category :

                cur_result = {};

                cur_result['category'] = cur_cate;

                cur_result['value_01'] = int(math.sin(i/float(cur_cate))*100)*1;

                cur_result['value_02'] = int(math.sin(i/float(cur_cate))*100)*2;

                cur_result['value_03'] = int(math.sin(i/float(cur_cate))*100)*3;

                cur_result['timestamp'] = datetime.strftime(datetime.utcnow(), "%Y-%m-%dT%H%M:%SZ");

                print(json.dumps(cur_result) );

                producer.send('realtime_sample', cur_result );

                sys.stdout.flush();

                producer.flush();

        sleep(1)

Json insomnia

{

"name": "metatron_realtime_sample_02",

"description": null,

"dsType": "MASTER",

"connType": "ENGINE",

"srcType": "REALTIME",

"granularity": "SECOND",

"segGranularity": "DAY",

"ingestion": {

"type": "realtime",

"topic": "realtime_sample",

"consumerType": "KAFKA",

"consumerProperties": {

"bootstrap.servers": "localhost:9092"

},

"format": {

"type": "json"

}, 

"rollup": true

},

"fields":[

{

"name": "timestamp",

"alias": "timestamp",

"type": "TIMESTAMP",

"logicalType": "TIMESTAMP",

"role": "TIMESTAMP",

"format": "yyyy-MM0ddTHH:mm:ssZ",

"seq": 0

},

{

"name": "category",

"alias": "category",

"type": "STRING",

"logicalType": "STRING",

"role": "DIMENSION",

"seq": 1

},

{

"name": "value_01",

"alias": "value_01",

"type": "DOUBLE",

"logicalType": "DOUBLE",

"role": "MEASURE",

"seq": 2

},

{

"name": "value_02",

"alias": "value_02",

"type": "DOUBLE",

"logicalType": "DOUBLE",

"role": "MEASURE",

"seq": 3

},

{

"name": "value_03",

"alias": "value_03",

"type": "DOUBLE",

"logicalType": "DOUBLE",

"role": "MEASURE",

"seq": 4

}

]

}

Preview message in the insomnia

{

  "timestamp": "2019-04-25T15:33:27.952+0900",

  "status": 500,

  "error": "Internal Server Error",

  "exception": "javax.validation.ConstraintViolationException",

  "message": "Validation failed for classes [app.metatron.discovery.domain.datasource.Field] during persist time for groups [javax.validation.groups.Default, ]\nList of constraint violations:[\n\tConstraintViolationImpl{interpolatedMessage='Invalid configuration spec.', propertyPath=format, rootBeanClass=class app.metatron.discovery.domain.datasource.Field, messageTemplate='Invalid configuration spec.'}\n]",

  "path": "/api/datasources"

}

Is there something wrong in my code?

Many thanks,

[come...@gmail.com]

Hi.
It is correct that you applied normally.
However, some issues have been added to our time-related specifications.

If you call as below, it will be processed normally.

"format":{
"type":"time_format",
"format":"yyyy-MM-ddTHH:mm:ssZ",
"timeZone":"Asia/Seoul",
"locale":"en"

I will also reflect on the blog.

Thank you for your feedback.

[woomin park]

Thanks for your kind,

I used your json code and it seems that the data is successfully ingested.

the Data is shown at the datasource and I can use at the dashboard. (not shown in the druid: localhost:8081)

But it is not work as the blog.

The line chart is just provide dots(sum value)

The error message is shown when I want to see datasouce detail(at the datasource tap),

QueryTimeExcetpion: [{"error":"Unknown exception","errorMessage":"Illegal pattern component: T","errorClass":"java.lang.IllegalArgumentException","errorStack":["org.joda.time.format.DateTimeFormat.parsePatternTo(DateTimeFormat.java:566)","org.joda.time.format.DateTimeFormat.createFormatterForPattern(DateTimeFormat.java:687)","org.joda.time.format.DateTimeFormat.forPattern(DateTimeFormat.java:177)","io.druid.common.utils.JodaUtils.toTimeFormatter(JodaUtils.java:404)","io.druid.math.expr.DateTimeFunctions$TimeFormatFunc.parameterize(DateTimeFunctions.java:740)","io.druid.math.expr.BuiltinFunctions$ParameterizingNamedParams.toFunction(BuiltinFunctions.java:229)","io.druid.math.expr.BuiltinFunctions$NamedParams.create(BuiltinFunctions.java:178)","io.druid.math.expr.ExprListenerImpl.exitFunctionExpr(ExprListenerImpl.java:316)","io.druid.math.expr.antlr.ExprParser$FunctionExprContext.exitRule(ExprParser.java:213)","org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)","org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)","io.druid.math.expr.Parser.parse(Parser.java:120)","... more"],"host":"{hostname}:8100","serviceName":"druid/middleManager"}]

*I just replace {hostname} 

It might be my test environment is not same as yours.

I use the lasted version of the kafka, and the python code also different.

Is there anything I have to configure environment?

 

below data is created by my python code

{"category": "10", "timestamp": "2019-04-26T0805:02Z", "value_02": 20, "value_03": 30, "value_01": 10}

{"category": "20", "timestamp": "2019-04-26T0805:02Z", "value_02": 198, "value_03": 297, "value_01": 99}

{"category": "30", "timestamp": "2019-04-26T0805:02Z", "value_02": -176, "value_03": -264, "value_01": -88}

Many thanks,

[woomin park]

With above error message, 

The real-time  stream chart is worked well, when I was changed the time format like

"format": "yyyy-MM-ddTHH:mm:ssZ",

I think the error message is shown because the data is not stored.

Is this right?

Additionally, how to make "mydate" dimension at the blog?

Many thanks,

[Mahesh Venkat]

Hi,

Just now I download the latest version from https://metatron.app/download/installation-guide-custom-install/ and deployed it successfully.  https://sktmetatronkrsouthshared.blob.core.windows.net/metatron-public/discovery-dist/latest/metatron-discovery-latest-bin.tar.gz 

It does not have streaming support.