How can I lookup data with multiple keys and with a range of value

Dimitry Khan

unread,

Dec 19, 2017, 12:30:06 AM12/19/17

to vega-js

I am using vega (vega 3.0) to plot a tree graph in Kibana. I am using stratify to transform my data. I have the key field for stratify transformation, but dont have the parentKey field in my data. I want to generate the parentKey from my existing Data based on the following logic.

For each node in the data, it's parent will have step_id=(current step_id)-1.
There is possibility to have multiple nodes which mach step_id=(current step_id)-1. I want to filter the only node which has timestamp most immediate previous to the current node's timestamp.

I have two questions here.

Can I perform a search in my elasticsearch database with the above two conditions, directly from a Vega transformation? I dont see any API call or search posiibility from the documentation. Please let me know if it is possible.
If a direct search is not possible. Then how can I do it using the existing transformation rule?

I have tried with lookup transformation and I can successfully apply #1 rule as mentioned above. But I can not inject #2 rule along with it. I can not see lookup with two different keys and that too for a range of value.

I am trying to look into lookup implementation in the Vega source. But before that want to understand if this is possible with the existing transformation. Any hint is appreciated.

Roy I

unread,

Dec 20, 2017, 10:43:28 AM12/20/17

to vega-js

Vega does not support searching Elasticsearch database directly. But search results using javascript can be passed to Vega as data using Vega "View" API and "signals":

https://github.com/vega/vega-view

There are examples of using Elasticsearch with D3 (d3.js) showing search results with hierarchical data (dendrogram):

https://www.elastic.co/blog/data-visualization-elasticsearch-aggregations

It may be possible to use Vega instead of D3 in the example.

For more specific help using Vega, please provide example data and desired visualization.

Dimitry Khan

unread,

Dec 20, 2017, 3:01:10 PM12/20/17

to vega-js

Thanks Roy! I have seen View API but could not find how to use it with vega specification. I will be seeing your example links more closely.

Let me provide you my example in detail. I have a log file which I read using logstash and put into ElasticSearch. Some sample data stored into ElasticSearch are as below -

note: @timestamp in the below data is the event timestamp when the data was read by logstash.

{

"audit_message":"Operation_ABC",

"step_id":4,

"@timestamp":"2017-12-01T03:15:22.000Z",

"exec_time":2.457235

}

{

"audit_message":"Step_DEF",

"step_id":5,

"@timestamp":"2017-12-01T03:15:27.084Z",

"exec_time":0.020198,

"parent_step_id":4

}

{

"audit_message":"Operation_GHI",

"step_id":6,

"@timestamp":"2017-12-01T03:15:32.175Z",

"exec_time":0.015176,

"parent_step_id":5

}

{

"audit_message":"Function_JKL",

"step_id":5,

"@timestamp":"2017-12-01T03:17:34.182Z",

"exec_time":4.46E-4,

"parent_step_id":4

}

{

"audit_message":"Function_MNO",

"step_id":6,

"@timestamp":"2017-12-01T03:17:39.254Z",

"exec_time":1.56E-4,

"parent_step_id":5

}

{

"audit_message":"Operation_PQR",

"step_id":6,

"@timestamp":"2017-12-01T03:17:44.345Z",

"exec_time":0.003522,

"parent_step_id":5

}

I want to have a visualization as below

As you can understand from the above example that the parent-child set is determined with two conditions.

1) parent's step_id = child's step_id -1

and 2) parent's timestamp is the immediate previous timestamp of the child.

While using stratify I was looking for to generate the parentKey from the transformation itself. I used lookup transformation but, that was only meeting the condition #1.

Here is the entire transformation.

"transform": [

{

"type": "lookup",

"from": "records", <----- records data is the same record set as above.

"key": "_source.step_id", <---- ignore _source here. _source was added to my search result when queried from ElasticSearch

"fields": ["_source.parent_step_id"],

"values": ["_source.@timestamp"],

"as": ["parent_timestamp"],

"default": null

},

{

"type": "stratify",

"key": "_source.@timestamp",

"parentKey": "parent_timestamp"

},

{

"type": "tree",

"method": {"signal": "layout"},

"size": [

{"signal": "height"},

{"signal": "width - 100"}

],

"as": ["y", "x", "depth", "children"]

}

]

I want to search the data with the above two conditions either by API or by transformation.

Dimitry Khan

unread,

Dec 21, 2017, 7:32:26 AM12/21/17

to vega-js

In my previous post, in the last line I said - "I want to search the data with the above two conditions either by API or by transformation."

What I meant is what would be the easiest way to do it from Kibana using Vega? May be my conception is not very clear. But I understand that I can only provide vega spec in the Kibana. And it is the back JS (the vega plugin I installed on Kibana) which works on the spec and create the visualization. I did not find any place in Kibana where I can put my custom JS. So, I was trying to see whether using any other transformations or combination of transformation I can achieve it?