What connects Morphline to an SQL database?

[Alexandre Rafalovitch]

A newbie question. 

If I want to run SQL against relational database and then push that through Morphline pipeline, what components do I need? Morphline documentation seems to say that it can load data from relational databases, but I don't see any specific source to run SQL command and consume the output. Ideally, it would be something that can be run multiple times to do incremental indexing.

What basic, obvious thing am I missing?

Regards,

   Alex.

[Wolfgang Hoschek]

There's currently no morphline command that would run SQL against a relational db, but you could write such a command yourself. Or you could write a program that feeds your SQL result set through a morphline.

Wolfgang.

--
You received this message because you are subscribed to the Google Groups "CDK Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdk-dev+u...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

[Alexandre Rafalovitch]

Ah! Thank you for quick update. I guess it is a good news/bad news. The good news I did not miss anything....

What about other Apache/Cloudera/BigData components? Is there one more piece that integrates with Morphline that does allow to query SQL database? I saw Sqoop, but that's more for batch transfers, as opposed to copy and update kind of approach. Flume also does not seem to have and JDBC/SQL source.

The high-level goal is to have something as an alternative to Solr DataImportHandler, which is quite popular for getting data from the database and then doing mapping/transformation. I think Morphline transformation component is much superior, but I can't figure out the the 'database-connection' use case (as per the question).

Regards,
   Alex.

[Wolfgang Hoschek]

Typically people use sqoop followed my mapreduceindexertool or crunchindexertool. 

Sent from my iPhone