[scalding] Tool, ToolRunner, Args and Hadoop properties

[Chris Heller]

I'm invoking a scalding Job via the Tool interface, which I kick off using ToolRunner.

I am trying to pass in a hadoop property using a '-D' argument and it doesn't seem to make it into the configuration, instead it ends up in the Args object, and I see it passed out in the 'scalding.job.args' job conf property.

Reading through the scalding source for Tool it seems that GenericOptionParser is invoked on the passed arguments, so I would expect that my property would get parsed out here, but that seems to not be the case.

Does this problem sound familiar to anyone?

-Chris

[Jonathan Coveney]

The order is kind of sensitive (which is lame and a warning would be nice, feel free to file an issue or pr about that)

That said, have you tried the order here?

https://github.com/twitter/scalding/wiki/Frequently-asked-questions#how-do-i-pass-parameters-to-my-hadoop-job-number-of-reducers--memory-options--etc-

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/50a064bd-1fe1-4c0d-a3ed-eebc8515003e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Heller]

Yes that seems to be the issue. If I move the argument after the class name but before the job arguments, it now shows in the job conf. The flag still appears in the Job args as well, but perhaps that is expected? 

Thanks,

Chris

[Jonathan Coveney]

As long as it's in the conf you should be good, yeah. I do think it exposes it to the user as well just in case they want to do something more.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2090c4ac-f155-4cc0-90ed-b3b2a5549efb%40googlegroups.com.

[karthik]

The -D works for me if it is provided after Tool but before the target class name.

[Koert Kuipers]

from hadoop optionparser perspective options parsed by hadoop (-D x=x, libjars, etc) need to come before non-hadoop options. Since options start after the classname which is scalding's Tool they need to be between Tool and the name of your Job class

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/f0532a84-1aaa-4cb3-bb5d-56a1f775b1c6%40googlegroups.com.