[ISSUE] (TIGON-154) InputFlowlet (SQLFlowlet) should be able to read data from Queues (data emitted from preceding Flowlets)

[Gokul Gunasekaran (JIRA)]

Gokul Gunasekaran created an issue

Tigon / TIGON-154 InputFlowlet (SQLFlowlet) should be able to read data from Queues (data emitted from preceding Flowlets) Issue Type: New Feature Assignee: Gokul Gunasekaran Created: 12/Nov/15 2:53 PM Priority: Major Reporter: Gokul Gunasekaran Currently InputFlowlet can receive data only on TCP/HTTP ports. It should be possible to read data from previous flowlets' queues so that it can used midway in a Flow. InputFlowlet currently uses in-memory queue and doesn't provide persistence and so in case of failure of the flowlet, data held in-memory will be lost. This is one of the reasons why InputFlowlet is used as (optionally) first flowlet of a Tigon Flow so that it can improve the throughput of input events ingested by trading off persistence (so that uninteresting events can be dropped) and subsequent Flowlets can process events that are of interest and there we provide full persistence using Queues implemented in HBase. Add Comment

This message was sent by Atlassian JIRA (v6.1.5#6160-sha1:a61a0fc)

[Gokul Gunasekaran (JIRA)]

Gokul Gunasekaran updated an issue

Tigon / TIGON-154 InputFlowlet (SQLFlowlet) should be able to read data from Queues (data emitted from preceding Flowlets)

Change By: Gokul Gunasekaran

Currently InputFlowlet can receive data only on TCP/HTTP ports. It should be possible to read data from previous flowlets' queues so that it can used midway in a Flow. InputFlowlet currently uses in-memory queue and doesn't provide persistence and so in case of failure of the flowlet, data held in-memory will be lost. This is one of the reasons why InputFlowlet is used as (optionally) first flowlet of a Tigon Flow so that it can improve the throughput of input events ingested by trading off persistence (so that uninteresting events can be dropped) and subsequent Flowlets can process events that are of interest and there we provide full persistence using Queues implemented in HBase.

Implementation: In the AbstractFlowlet class, add a ProcessEvent method and convert the StreamEventData into GDATRecord and add it to the  GDATRecordQueue. Also, the AbstractFlowlet can take in a parameter to choose between consuming data from HTTP/TCP ports vs previous Flowlets.

Add Comment