How to make num-instances required

[Jeffrey]

Is it possible to write a jobdef that requires --num-instances to be set as a command line arg? I tried the following

(catch-args
[:num-instances "How many MapReduce instances to launch." nil])
(add-validators
(val-opts :required :numeric [:num-instances]))

but it fails regardless of whether I actually pass --num-instances:

Exception in thread "main" java.lang.IllegalStateException: Command spec contains duplicates: ([snip] "num-instances" "num-instances" [snip]) at lemur.command_line$add_command_spec_STAR_$fn__912$fn__913.invoke(command_line.clj:237)

I also tried to only use val-ops without catch-args, but this lets every job pass, since :num-instances is defined by default to "1".

Thank you!
Jeffrey

[Marc Limotte]

Don't put it in catch-args, because it is a standard argument for lemur... it is automatically included in catch-args.  Hence the error that it's a duplicate.

val-opts alone won't work, though because the standard lemur num-instances arg has a default.

What should work is:

1.  put it in val-opts as you have it, and 

2.  in you defcluster remove the default value:

      (defcluster xyz

          :num-instances nil 

          ...)

Let me know how that works.

Marc 

Jeffrey

--
You received this message because you are subscribed to the Google Groups "Lemur User" group.
To post to this group, send email to lemur...@googlegroups.com.
To unsubscribe from this group, send email to lemur-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/lemur-user/-/juQ1cdcATSsJ.
For more options, visit https://groups.google.com/groups/opt_out.

[Jeffrey]

Thanks for your reply Marc.

If I set :num-instances nil, Lemur seems to coerce that to 0.  From the METAJOB:

Jobflow Options:

  slave-type: c1.xlarge

  num-instances: 0

  [snip]

Lemur doesn't throw an error and spawns the cluster; interestingly somewhere along the line EMR overrides the 0 to be 1.

If there's no way around this, I'm not too worried; it's not a major requirement on our side.

Jeffrey

On Tuesday, July 31, 2012 1:25:59 PM UTC-7, Marc Limotte wrote:

Don't put it in catch-args, because it is a standard argument for lemur... it is automatically included in catch-args.  Hence the error that it's a duplicate.

val-opts alone won't work, though because the standard lemur num-instances arg has a default.

What should work is:

1.  put it in val-opts as you have it, and 

2.  in you defcluster remove the default value:

      (defcluster xyz

          :num-instances nil 

          ...)

Let me know how that works.

Marc 

[Marc Limotte]

Jeffrey,

There is code to convert num-instances to an int (which includes the case, nil => 0).  But that code is run AFTER the command line validations.  Did you try adding the validator that you wanted?

If you still have trouble, you can add arbitrary functions as validators.  E.g.:

 

(lfn [num-instances] 

  (when-not (and num-instances (pos? num-instances)) 

     "--num-instances is required and must be > 0."))

Marc

--
You received this message because you are subscribed to the Google Groups "Lemur User" group.
To post to this group, send email to lemur...@googlegroups.com.
To unsubscribe from this group, send email to lemur-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/lemur-user/-/GmrKpsOmZggJ.

[Jeffrey]

Using the :numeric validator, setting :num-instances to nil, and not catching the arg works perfectly.  Thanks for the help!