Pass Hadoop parameters from command line

[Fei Dong]

Hi all,

In cascading program, I know the parameters can be passed via property
such as

Properties properties = new Properties();
properties.setProperty("io.sort.mb", "300");
FlowConnector.setApplicationJarClass(properties, Main.class);

Now I hope to set some Hadoop parameters in command line such as

>hadoop jar loganalysis.jar -Dio.sort.mb=300 data/apache.200.txt output

The error shows:
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input
path does not exist: file:/home/dongfei/projects/starfish/-
Dio.sort.mb=300
Obviously, it treat "-D" as the first parameter in program (String
inputPath = args[ 0 ];)

Then I make some change about loganalysis main class which implements
Hadoop Tool interface.

Some code sample as following
###############
public class Main extends Configured implements Tool {
public int run(String[] args) throws Exception {
// set the current job jar
System.out.println("io.sort.mb:" +
System.getProperty("io.sort.mb"));
Properties properties = new Properties();
FlowConnector.setApplicationJarClass(properties, Main.class);
.......

But Cascading does not get the "io.sort.mb" settings.

Anyone has ideas? Thanks.

[Chris K Wensel]

Hi

You need to take a look at the bin/hadoop shell script to see how to pass parameters to the child jvm.

loganalyis.jar is currently getting your -D args and it has no idea what to do with them. another option is to tell it how to recognize them in the #main function.

cheers,
chris

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
>

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support for Cascading

[Fei Dong]

Hi Chris,

I solve it. I use the Tool's getConf() then iterate over all of its
entries and add them into the properties.

> > For more options, visit this group athttp://groups.google.com/group/cascading-user?hl=en.
>
> --
> Chris K Wensel
> ch...@concurrentinc.comhttp://www.concurrentinc.com