Job Parallelism with different args per pod

[Eduardo Franco]

Hello,

I am trying to figure out how I can use Keueu to submit large amounts of containers where each container gets a different set of args (without submitting thousands of jobs). Run A Job indicates that the .spec.parallelism  parameter in the manifest enables submitting multiple pods at once, but this seems limited to each parallel pod running the exact same thing, rather than each one taking in a separate set of args.  Run A JobSet seems like a possible solution, but the example indicates that the parallel jobs would all get the same args as well.

In essence, I'm trying to achieve the same behavior as the array job in AWS Batch. I want to launch the same container thousands of times, but each one with a different set of args. Any tips would be greatly appreciated, thanks.

Best,
Eduardo

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.

[Abdullah Gharaibeh]

How do you want to set the different args?

For Job, you can use IndexedJob which associates each pod with a unique index that you can pass as an env var to customize your args. Samething with JobSet, each Job in a ReplicatedJob can gets a unique index that you can pass as an env var. With JobSet you can also create different Jobs using different templates, but you have to specify that explicitly in the yaml.

--
To unsubscribe from this group and stop receiving emails from it, send an email to wg-batch+u...@kubernetes.io.

[Aldo Culquicondor]

You're looking for this https://kubernetes.io/docs/tasks/job/indexed-parallel-processing-static/

[Eduardo Franco]

Thanks Abdullah for the quick response. Ok, the IndexedJob actually looks pretty much exactly like what the array job in AWS Batch does, where each container gets an environment variable with its index in the array, and then orchestrating specific args is a matter of reading from some config file containing all the possible args once the container is running. This seems like the answer I'm looking for, thanks so much.