Druid UI/Management Software

[doron gan]

Hi Everyone,

We've been using Druid for a few months now in production and dev and are pretty happy with the results!

One thing that would make the druid experience better is to have a great User Interface out of the box to allow folks to manage their druid clusters visually and to explore their data via the UI. I have a lot of experience with OLAP UI's having done this a few times and wanted solicit folks for some feedback.

1) Are there already tools for managing a druid cluster with a management UI?

2) Are there already UI's to allow one to generically explore and query the data in a druid cluster?

3) If you could have the most awesome management UI for your druid cluster, what would you like to be able to do with it?

4) If you could have an awesome generic data exploration/query UI what would goals would you want to achieve with it?


Thanks,
Doron

[Fangjin Yang]

Hi Doron, one of the biggest requests from the open source Druid community is an open source UI. We'd love to chat about this and even offline if your schedule is accommodating. Some more thoughts inline.

On Friday, November 7, 2014 7:55:02 AM UTC-8, doron gan wrote:

Hi Everyone,

We've been using Druid for a few months now in production and dev and are pretty happy with the results!

One thing that would make the druid experience better is to have a great User Interface out of the box to allow folks to manage their druid clusters visually and to explore their data via the UI. I have a lot of experience with OLAP UI's having done this a few times and wanted solicit folks for some feedback.

Absolutely!! 

1) Are there already tools for managing a druid cluster with a management UI?

We open sourced a new UI in the druid-0.7.x branch for managing cluster usage (https://github.com/druid-io/druid-console). This UI doesn't include any way of visualizing Druid internal metrics though.

2) Are there already UI's to allow one to generically explore and query the data in a druid cluster?

Not any open source ones that I'm aware. I've heard a lot of community request for this. Would love to talk about this more. 

3) If you could have the most awesome management UI for your druid cluster, what would you like to be able to do with it?

Probably a combination that uses some APIs that the coordinator exposes about the state of the cluster and metrics that all Druid nodes emit periodically and on queries. 

4) If you could have an awesome generic data exploration/query UI what would goals would you want to achieve with it?

I think this would be the most beneficial to the community as many users want to just be able to quickly see their data when they first spin up Druid. 

Thanks,
Doron

[Xavier Léauté]

Doron, we would love to get help from the community to improve the Druid management / query interface.

Like Fangjin said, we are revamping the Druid console for 0.7, so it would make sense to build upon that for any other improvements we want to add.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/f5d1e308-1691-4d1c-b58f-39b5af922ead%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jnebrera]

Hi Doron,

If things evolve as expected (and after a few delays), on Nov 17th we will make public the community releases or redBorder framework.

While very specific to security and network operations stuff, the reality is, in essence the system is able to cope with any kind of information it gets as JSON within Kafka messages.

Throught its UI interface you would be able to explore the data very easily.

I have to say at this moment the fields are quite specific but for sure we would love to get some community involvement to make this platform the reference for Druid based Data Analysis in more generalized terms

[Roger Hoover]

Grafana (http://grafana.org/) seems like a nice match for for Druid.  I just started working on extending Grafana to support Druid (https://github.com/theduderog/grafana).  It's still rough with a little hard coding remaining but getting there.

If others are interested as well, let's collaborate on it.

Cheers,

Roger

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/7c4bf71d-d1f8-49ff-94f2-317f3a8ce571%40googlegroups.com.

[Prashant Deva]

We at DripStat (https://dripstat.com) would love to offer full integration with Druid.

If anyone from Druid is interested, we would love to work with you to create a high end UI for Druid

[Eric Tschetter]

If you guys do have a conversation of this sort, can I request that you also make the invitation available to other people from the community who might want to attend?  I know I would like to.

As we move forward, I believe it is important to more deliberately work towards a community-owned and led roadmap and that can only start with conversations like this happening in the open, where anybody from the community could potentially participate.

Also, given the indication that 0.7 will provide a starting point for this, I'm assuming that Metamarkets is planning on calling an RC and stable from 0.7 soon?

--Eric

[Jaime Nebrera]

As already said, we would be more than willing to work in that effort and believe our redBorder framework would be a very nice starting point, even if a bit biased towards security and network operations

It is extremely Druid specific, covers most of its capabilities, etc

The code is AGPL and hopefully soon will be available in GitHub besides the community site (Nov 17th)

Either way, would be an interesting topic in general

You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/pblPGoaYV3c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB8U%2Bh0qE3Rf8hkv4XGbHGvp12i_bCjH9tHUBvwvVrfwj9KXHw%40mail.gmail.com.

[Eric Tschetter]

That sounds awesome Jaime!  Fwiw, my comment was more in response to Fangjin's request to potentially take the conversation offline.

--Eric

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACvsLFYWQ--%2BhH30mHHXBkZmmH_W%2BzV%2BDatsCrf30LQNYm-zsg%40mail.gmail.com.

[Jaime Nebrera]

Yes, no problem

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB8U%2Bh3e4RPbhaTPXhAEe-bAC9gGyfZ1%3Diy%3DQnjLk0MSKhVeMg%40mail.gmail.com.

[Xavier Léauté]

> As we move forward, I believe it is important to more deliberately work towards a community-owned and led roadmap and that can only start with conversations like this happening in the open, where anybody from the community could potentially participate.

Completely agree, that is also why I pointed people to the new druid console work in the hopes that we can use that as a starting point for a discussion to make a better management / query diagnostic tool.

> Also, given the indication that 0.7 will provide a starting point for this, I'm assuming that Metamarkets is planning on calling an RC and stable from 0.7 soon?

0.7 is still in development, but we are close enough to having it be testable that we will create a 0.6 branch and make 0.7 the master branch this week.

Most of the remaining issues to be ironed out for 0.7 should be marked with the 0.7 milestone in GitHub.

From now on we will mainly provide bug fixes for 0.6 and possibly back-port some features and optimizations as needed, however most new work is focused on 0.7.

A beta might be available soon, but I don't expect an RC for another month at least, since we haven't started deploying internally yet.

Xavier

[Fangjin Yang]

Heh, definitely agree the UI should be an open discussion if the community is interested, which I think everyone is.

In addition to just more testing required, I think we should vote on some of the default behavior for Druid (Roaring vs Concise, off-heap vs heap ingestion, etc). I think this decision process will be more informed once we share the testing/benchmarks we've done. I also have a backlog of docs/tutorials/misc things I want to write.

As for feature development, once the initial set of changes are stabilized for 0.7.x, I think we should have a discussion thread about every major change that we as a community want to make to Druid. It'll be an extension of the design email threads we used to do back in the day Eric :)

[doron gan]

Hi

Been waiting to see all the comments and also been heads down the past week. Sounds like redBorder has something, would love to see some screen shots and a list of the functions. Also saw an email about https://github.com/zhenchuan/druidme.

Fangjin, I'm happy to have a real time conversation about this. Feel free to email me/others who may be interested to setup a time. Perhaps, we can do a google hangout.

We should probably start with some requirements gathering. Here are some I can think off the top of my head.

0) Web based UI.
1) See all the Facts available (aka Druid data sources)
2) See the data in a "Fact". View data by each dimensions at the same time. (e.g. see by Time, then also see by Attribute1, then also see by Attribute2 at the same time on the same screen)
3) Be able to do simple filtering easily (hopefully not having to type anything)
4) Be able to generate multi-dimension tables (i.e. group by more than one dimension, e.g. Atr1, Atr2, Atr3)
5) See multi-dimension tables as cross tabs and also create simple charts/graphs from this data.
6) Allow specifying granularity, sort, and limit preferences.
7) Allow saving and restoring the view settings (e.g. bookmark the view into a specific URL, etc)
8) Allow plug-able authentication (single sign on)
9) Allow specifying default time filters (e.g. most of the time we only want to see the last few months. Doing larger time range is just slow.)



Doron

[doron gan]

One more question. You guys mentioned a new druid console. Do you have a feature list and/or screen shots of whats there now. Curious to see what it covers.

Thanks,
Doron

[Fangjin Yang]

Hi Doran, details here: https://github.com/druid-io/druid-console

I believe you can try it out if you can compile the current master.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/ab4a0b79-2568-4a9f-8f10-4a1d46392321%40googlegroups.com.

[Jaime Nebrera]

Hi Doron,

Tomorrow from a proper keyboard I will reply to each point in your email plus provide some specifics on the platform

In advance, redBorder is a JSON over Kafka enterprise bus targeted to security and network operations.

Data is stored in Druid and a Web interface is able to interact with it

The platform is much more complex as includes IPS devices, their management, etc but from Druid perspective we can say as long as you provide JSON / Kafka, redBorder will digest it (with some customization though)

We would love to create a community around it to both improve its general Druid capabilities (for example, dynamically detect the meta data fields) as well as ingestion capabilities (right now barnyard2 and netflow plus some others)

The software will be available under AGPL license and freely available in the website project

Here you have a video teaser of the interface: http://youtu.be/TqQBIRagEl0

Tomorrow more

--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/pblPGoaYV3c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/286f4c03-c830-4777-b4c1-e63ae9b15185%40googlegroups.com.

[Fangjin Yang]

Hi, see inline.

On Thursday, November 13, 2014 11:46:17 AM UTC-8, doron gan wrote:

Hi

Been waiting to see all the comments and also been heads down the past week. Sounds like redBorder has something, would love to see some screen shots and a list of the functions. Also saw an email about https://github.com/zhenchuan/druidme.

Fangjin, I'm happy to have a real time conversation about this. Feel free to email me/others who may be interested to setup a time. Perhaps, we can do a google hangout.

Hangout sounds good. What are some times that work for people? 

We should probably start with some requirements gathering. Here are some I can think off the top of my head.

0) Web based UI.
1) See all the Facts available (aka Druid data sources)
2) See the data in a "Fact". View data by each dimensions at the same time. (e.g. see by Time, then also see by Attribute1, then also see by Attribute2 at the same time on the same screen)
3) Be able to do simple filtering easily (hopefully not having to type anything)
4) Be able to generate multi-dimension tables (i.e. group by more than one dimension, e.g. Atr1, Atr2, Atr3)
5) See multi-dimension tables as cross tabs and also create simple charts/graphs from this data.
6) Allow specifying granularity, sort, and limit preferences.
7) Allow saving and restoring the view settings (e.g. bookmark the view into a specific URL, etc)
8) Allow plug-able authentication (single sign on)
9) Allow specifying default time filters (e.g. most of the time we only want to see the last few months. Doing larger time range is just slow.)

Are these the specifications you are thinking of for an MVP?

-- FJ 

Doron

[Jaime Nebrera]

Hi Doron,

As explained yesterday briefly, redBorder is built around Druid to
manage security and network monitoring data, but in essence, it is
capable of managing any information provided as JSON within Kafka messages.

In general, the platform provides some "enterprise" kind features
that make it interesting in complex scenarios: user roles and auditing,
dashboards, reports, etc

As for data, we create "by default" several views of the same data:

* Top-N tabs for each "variable" or column
* Table views with dynamic addition and removal of columns (what we
call raw view)
* Cardinality specific views

It is extremely easy to apply filters on any dimmension and keep them
for the rest. It is very easy to create a dashboard or report widget
based in any of this views

Data is shown for the last 1h, 3h, 8h, etc

As for a "general" framework, to consider it a more generic Druid UI,
it is clear some stuff needs to be considered or added, for example, the
capability to automatically detect the "columns" or to exploit some
Druid capabilities (like percentile estimation) but in general I believe
it would be a great foundation for analytics on top of Druid

>
> We should probably start with some requirements gathering. Here are
> some I can think off the top of my head.
>
> 0) Web based UI.

Yes, RoR based and available under AGPL license

> 1) See all the Facts available (aka Druid data sources)

Yes

> 2) See the data in a "Fact". View data by each dimensions at the same
> time. (e.g. see by Time, then also see by Attribute1, then also see by
> Attribute2 at the same time on the same screen)

Yes

> 3) Be able to do simple filtering easily (hopefully not having to type
> anything)

Yes, both basic and advanced filtering

> 4) Be able to generate multi-dimension tables (i.e. group by more than
> one dimension, e.g. Atr1, Atr2, Atr3)

Yes

> 5) See multi-dimension tables as cross tabs and also create simple
> charts/graphs from this data.

Please, explain

> 6) Allow specifying granularity, sort, and limit preferences.

Yes

> 7) Allow saving and restoring the view settings (e.g. bookmark the
> view into a specific URL, etc)

You can save views, filters, dashboards and widgets and share them
with different users

> 8) Allow plug-able authentication (single sign on)

For now local and LDAP

> 9) Allow specifying default time filters (e.g. most of the time we
> only want to see the last few months. Doing larger time range is just
> slow.)

We have 1h, 3h, 8h, 1d, 1w, 1m, 1q, 1y

For us would be a great honor to define redBorder as the reference UI
for Druid and anything we can do in that sense

As for a real call, we would be interested in participating. Just
remember we are in Spain :D

--
Jaime Nebrera - jneb...@eneotecnologia.com
CTO - ENEO Tecnologia SL
Plaza de las Naciones s/n
Edif. Aljarafe Center. Planta 12
Mairena del Aljarafe - 41927 - Sevilla
Telf.- 955 60 11 60 / 619 04 55 18

[doron gan]

Hi

Jamie, thanks for sharing about redBorder. I looked at the demo and loved the charts, geo maps, and flow maps. Would like to play with your product when you feel its ready to get a better sense.

> 5) See multi-dimension tables as cross tabs and also create simple
> charts/graphs from this data.

   Please, explain

--> If I generate a table of data by grouping a Fact by several dimensions, I want to see a cross tab.
Example:
Lets say we are looking at advert impressions. Some dimensions are Location, Subject, Campaign, Time Of Day.
I want to query druid and aggregate the number of impressions and price of impressions by Location, Campaign, and Subject.

I want a cross tab where the first row is made of columns that have all the Location values.
The next row down will be columns (under each location) that have "Count" and "Price" columns.

I want rows where the first column of each row would be the Subject, then next column of each row would be the Campaign (I.e. for each Subject, you will see all the Campaigns next to it like a tree)

In each intersection of Subject, Campaign, Location you will see Count and Price.

I should then be able to take this cross tab (Since it specifies and X axis (e.g. Location) and Y Axis (e.g. Subject,Campaign) and graph one or more of the measures (Count, and Price).
The graph can be a line chart, bar chart, etc.

Same kind of thing as above but using time.


Fangling, yeah, I think this would be an MVP that would handle 80% of what I've seen people want in the past in the industries I've been involved in.

[Jaime Nebrera]

Hi Doron,

This is not included at the time, but of course is something to consider and we will be open to contribs too ;)

The current raw view is quite powerful by itself, allowing you to add or remove any aggregation field dynamically, increase the aggregation time (and focus afterwards), etc

--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/pblPGoaYV3c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e57e6c57-55dd-48d0-8148-dbde69212a05%40googlegroups.com.

[Julien Lavigne du Cadet]

Hi Jaime,

Is there any update on the open source release of redborder? I can't find any mention on your website.

Thanks,

Julien

[Jo Witters]

Would be good to have MDX query capability for druid.