Developers interested in continuing as Apache project?

[A. Soroka]

Hi, just putting a feeler out in case any former Kylo devs are reading this list. It's not totally clear to me from the announcement on 12 Feb whether the devs are still mostly around but working on other things or whether they are really scattered and gone.

If anyone is still around, would any of you be potentially interested in restarting Kylo as an Apache project? I think it could fit into the Hadoop ecosystem or could be pursued as an independent effort. I am a committer/project management committee member for an Apache project, which means I know enough to answer some FAQs about becoming an Apache effort and enough to find answers to harder questions, so if there is interest, let's please discuss here! If not, no worries, I hope y'all are all enjoying your new projects.

Adam Soroka

Office of Research Computing / Office of the Chief Information Officer

Smithsonian Institution

[Eric Yuan]

We are a Kylo user and have been doing our own builds since Kylo development discontinued last year. We are also interested to see it continue as an Apache project. I have a few developers who are interested in participating but wouldn't say we can take on a lead role.

Having Kylo as an independent project may be preferable, so it can support other backend data stores in addition to Hive/HDFS.

Dr. Eric Yuan

Principal Scientist

Aerospace Corporation

[임지수]

We are working on a project using kylo and adding some features. 

As NiFi is growing, Kylo will certainly be an amazing project.

We look forward to the Apache Foundation's decision.

Jisu Im

Software Engineer

LG CNS

[Zubair Muhammad]

Hi Team, 

Just wanted to know, did the Kylo got approved as Apache project? 

[A. Soroka]

There seems to have been some confusion-- I know of no such application. In response to my question, I did not hear from even one Kylo dev, so I assume that Teradata has definitively killed Kylo. No devs-- no project.

Adam Soroka

[imku...@gmail.com]

Can someone explain to me why this is dead from a technical standpoint? 

Who are interested in reviving it?  Maybe through cloud-fund or something?

[asks...@gmail.com]

Last year, I was searching for common platform for data ingestion and wrangling.

My initial impression with Kylo was very good as the features they are providing is outstanding.

However I could not make it. Do you still actively using this tool ? Would you mind to share or demo video and is your working build publicly available ?

Also you can provide some insight - what exactly you are doing and how is the performance of this tool :)

~ Smruti

[Zubair Muhammad]

About its current status @Adam has replied. About 2nd point to reinvent it or to bring it back. I am willing to put effort. I previously worked on it a lot, in my previous organization we also customized it a bit. But due to some uncertain reasons Teradata pulled its creators and No devs-- no project happened. 

[imku...@gmail.com]

That sounds great.    I am not a developer but I'm interested in helping however way I can.

I'm partially still trying to understand its market+ technology+business value, especially after 4 years of ceasing development. 

After spending a bit of time researching,  I came back with at least these semi alternatives, though I do still see great value of kylo among them.  Before reviving it, it is perhaps important to position it properly so as to maximize its chance of resurrection.

cloud native solutions:

* https://aws.amazon.com/solutions/implementations/data-lake-solution/.  https://github.com/aws-quickstart?q=DATALAKE&type=&language=

* http://azure.github.io/AzureDataLake/

enterprise solutions (on premise or cloud):

* cloudera CDP :  free trial only

* HPE Ezmeral Data Fabric. (formerly mapr) :  still with community version.

* Qlik Compose/Catalog :  ?

* talend data fabric: https://www.talend.com : free trial only

freemium solutions:

* older version of  HDP:    https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide

* Dremio data-lake.  https://www.dremio.com/, 

* mapr community: https://mapr.com/download/,    mapr+hive+drill+hue

open source:

* WeBankFinTech dataspherestudio DSS + Linkis + Qualitis + Visualis + Azkaban.   https://github.com/WeBankFinTech/DataSphereStudio/releases/tag/0.9.0

* kylo:  ?

related:

* apache hudi https://hudi.apache.org/

[imku...@gmail.com]

If any developer wants to help out, maybe start with providing a fork and release packages? That would really help out many people.   I remember trying to build the latest version, and there are numerous issues with Maven repository SSL version,  missing packages etc.

[Auque Pierre-Arnaud]

If someone is ready to share a built package, so we can start retro-engineering the tool, I am totaly in for helping.

[mnba...@gmail.com]

If project is being revived, i am interested in development front.  

[杨胜文]

https://drive.google.com/file/d/10ICHEmV1M7nG7jRgUqn3HRTRfP6uk3fS/view?usp=sharing
You can try this package, but I forget what I have changed.

[杨胜文]

I hope this project restart too, we use this project to build a service, but it has too many problems, and I don't know how to fix it.

[Zubair Muhammad]

Hi Team, 

I worked on Kylo for 4-5 months and let me share a few things...I will work on building it as I have done it before and will submit a PR. 

Limitations of Kylo: 

• The opensource version of kylo was provided on a VM and you can't run in on large scale/cluster. (I am not sure about their enterprise features though). 
• Kylo was using Ambari and didn't have a mechanism to show the status of HDFS, Hive, and Spark servers on the UI. Though a plugin can be written for this. 
• Security was a major challenge, Kylo didn't have mechanisms for Authentication and Authorization. 
• There wasn't a mechanism to track data lineage. 
• Deploying Nifi on the cluster and making Kylo work with it can be a hassle (I didn't try it though) but to make it able to do some serious work its a must-do thing. 

What we did:

•  We tried to deploy it on AWS and EMR and was successful to this end, but the jobs which use Spark-Processor (in Nifi) only run successfully when the spark job uses Spark-SQL. Scala and python jobs didn't run, further investigation showed that there is no native support of "s3a" out-of-the box. 
• It didn't provide out-of-the box integration with Hive on AWS, though we were able to fetch table names after some modifications from Hive running on EMR. 

Kylo Advantages (Where Kylo Shines):

• Kylo's approach was to put all data in Hive tables and use Nifi for job orchestration. Its mainly build around Hive and Nifi. 
• The concept of feeds makes the processing clean.
• There are also a few more good things which I will post shortly. 

[Aqe Pierre-A]

Thank you for your feedback on Kylo usage.

Kylo seems (and probably have) a kind of complicated architecture (with Nifi, ES...). I guess if the project restarts, the focus have to be on the UI where it leverages Nifi and Spark forces, by abstracting their complexity with some kind of user forms.

The goal if to do simple thing with complex tool.

I really plan to do something similar, where it uses external Nifi and Spark cluster.

[Zubair Muhammad]

Hi Aqe Pierre-A ,  setting Spark on the external cluster is do-able...its explained in setting but Edge Node. However, never tried external Nifi. The approach you suggest is good. Kylo UI is basically an Angular project and a REST API is exposed via Spring-boot project on top of Nifi, Spark, ES, etc.   

[imku...@gmail.com]

 I thought it was architected  as  a service-oriented  extensible system in mind.   Also I thought data-lineage is one of the big features as well?  But other apache projects such as apache atlas have also taken on data lineages?

There are also a few abandoned GitHub projects (search kylo docker)  trying to run it in a docker container while leaving all other parts (nifi, spark, hive) as external components. 

It also needs an easy open/marketplace/plugin system for people to write custom transforms, validators, and/or use data-feeds to transform data-feeds.  Perhaps it also needs to extends its way of exposing data in datalake for user/tool consumption, but that can become out of scope really soon.

[Ahmad Hassan]

Yes, it is an extensible service-oriented system and it can use external Nifi, spark or any other service like ActiveMQ. 

The plugin development is also quite easy. I had developed a custom plugin for JAAS based authentication and authorization for my previous organization to use kylo with their existing security infrastructure.  Custom transformers and validators are all possible too but not through the UI yet. 

Moreover, the front-end code was also being shifted from Angular JS to Angular 5 and was almost done by my team when Teradata decided to pull the plug. Some of the components which are left can be converted easily and Kylo can be easily made to run on latest Angular version for further development. The conversion code is also available on the same github repo.

Regards,

Ahmad Hassan.

[imku...@gmail.com]

Ahmad, 

You seem very well versed in the original project.  What is your take on reviving it?  Thanks. 

[imku...@gmail.com]

Hi, 

Can you provide a brief instruction or maybe a vagrant box for your release?  That would benefit some people here. Thank you!