Tigon Product Feature Roadmap

[Gokul Gunasekaran]

Hi,

For Tigon’s upcoming releases, we are planning for the following features. If you have any questions/comments/suggestions, please send an email to tigo...@googlegroups.com.

General Product Features:

• Capability to pause a Flowlet
• Capability to pause an instance of a Flowlet (including TigonSQL)
• Capability to pause all the instance of a Flowlet (including TigonSQL)
• Capability to kill a Flowlet including a instance or all instances
• Capability to specify placement policy for placing Flowlets on a Cluster
• Capability to run multiple flowlets within a single YARN Container
• Capability to control number of Flowlets instance within a Container at Flowlet level
• Capability to control number of Flowlets instance within a Container at Flow level
• Capability to investigate log of a single Flowlet
• Capability to investigate log of a single Flowlet instance
• Capability to perform a rolling upgrade of a Flow
• Java Implementation of Tigon SQL from top-down and reduce number of moving components
• Capability to locally store the states in a container before persisting to queues
• Capability to add/remove/update Flowlets dynamically to a running Flow
• * Capability to support abstraction that allows a user to specify a Stream, SQL and is able to run a continuous query on the Stream to emit the output to a Dataset
• Capability to dynamically auto-scale instances of Flowlet

CDAP & Developer Experience

• Capability to run Tigon Flow within CDAP
• Capability to run Tigon seamlessly in Singlenode CDAP
• Capability to run Tigon within CDAP on Mac, Linux & Windows environment
• Capability to run Tigon within a Unit Testing Framework provided by CDAP

Windowing

• Capability to pass the windowed events from TigonSQL into rest of the Tigon Flow
• Capability to run TigonSQL on a Stream with Windowing defined on the Stream

Dataset & Stream Integration

• Capability to use Datasets within Tigon
• Capability to specify a dataset as the output of a TigonSQL
• E.g. TigonSQL output to a Timeseries Dataset

Context & Configuration

• Capability to change ticking within a Flowlet at run time or being configured at startup time rather than at coding time

Easing Data Ingestion into TigonSQL

• Capability to ingest data from Stream directly into TigonSQL
• Capability to ingest data from Kafka into TigonSQL
• Capability to ingest data from JMS into TigonSQL
• Capability of Tigon to expose a open TCP port that is easily discoverable and the source can push information directly into TigonSQL
• Capability to process data as it’s being ingest into TigonSQL through Flume
• Capability to easily deploy only TigonSQL Flowlet for processing data as it arrives and write to a Dataset
• Capability to directly ingest data from network interfaces into TigonSQL

Partitioning of Data

• Capability to process partitioned data in Tigon SQL


Performance

• Reduce overhead with SerDe to/from TigonSQL
• Capability to ingest into TigonSQL directly without serializing objects

Security

• Capability to run TigonSQL as the user authorized to run
• Securing all communications between different components of TigonSQL
• Capability to auto-renew tokens required for accessing Datasets

Metrics & Logging

• Capability to emit counter and gauges at Flowlet level
• Capability to aggregate metrics across Flowlet instance level
• Capability to perform aggregations (SUM, MIN, MAX, AVG, STDDEV, HISTOGRAM) on metrics


Console & CLI

• Capability to create operational dashboard for application with a single click
• Capability to define metrics and emit metrics from Tigon and ability to visualize them as operational Dashboard
• Capability to look at logs for given flowlet or flowlet instance from console
• Capability to visualize the DAG
• Capability to investigate metrics and logs for each flowlet including TigonSQL aspect
• Capability to manage lifecycle of Flow
• Capability to manage lifecycle of Flowlet

SQL

• Capability to store the schemas in HCatalog or HIVE Metadata store (Better integration with ecosystem components)
• Capability to run multiple instance of TigonSQL
• Capability to distributed instances of TigonSQL on cluster
• Capability to create rollups on multiple dimensions defined

Coopr

• Capability to deploy Tigon application using a Coopr template


Thanks,
Gokul