Sink to database in 2.0-WIP

[Ivan Zelensky]

Hi,

I'm trying to write Tap and Sheme classes for database-output using 2.0-WIP using way like used in hadoop-example 'DbCountPageView':

org.apache.hadoop.mapred.lib.db.DBConfiguration.configureDB(jobConf, driverClassName, connectionUrl);

org.apache.hadoop.mapred.lib.db.DBOutputFormat.setOutput(jobConf, databaseTableName, dbTableColumnNames);

jobConf.setOutputKeyClass(MyDBWritable.class);

jobConf.setOutputValueClass(NullWritable.class);

Where class MyDBWritable implements org.apache.hadoop.mapred.lib.db.DBWritable, org.apache.hadoop.io.Writable;

If I understand cascading architecture correctly this code must be executed in method MyDBSinkScheme.sinkConfInit, and in method MyDBSinkScheme.sink() I must to write:

Tuple outputTuple = sinkCall.getOutgoingEntry().getTuple();

sinkCall.getOutput().collect(new MyDBWritable(outputTuple.getString(0),...), NullWritable.get());

Actually, these code works, but it doesn't write nothing. Debugging shows that outputCollector doesn't call no one write() method of class MyDBWritable: nor write(DataOutput out), nor write(PreparedStatement ps).

Also, in job.xml for hadoop-example 'DbCountPageView' job I see value, that option mapred.output.key.class=org.apache.hadoop.examples.DBCountPageView$PageviewRecord and mapred.output.format.class=org.apache.hadoop.mapred.lib.db.DBOutputFormat

but for my cascading job mapred.output.key.class=cascading.tuple.Tuple and mapred.output.format.class=org.apache.hadoop.mapred.lib.NullOutputFormat

I suspect that these options from job.xml are root of my problem. So, question:

How I can to set these options in cascading program?

Thanks,

Ivan Zelenskyy

[Chris K Wensel]

JobConf options are set by a Scheme and/or Tap by the #sinkConfInit() and #sourceConfInit() methods. Make sure if you override the method in Tap, it continues to call the appropriate super#xConfInit methods.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com