Questions about Cloud Scheduler

[miccloud]

Hi,
I have tried your AMI with Cloud Scheduler and Condor.
I would ask you few things:

1) Does Cloud Scheduler configure automatically the Ami Condor Worker
node, for joining the pool? (Amazon EC2)
Or I have to write particular script to do at execution node
startup for configure it with central manager?

2) Should I install other software as well as condors in Execution
node that are launched by Cloud Scheduler?

3) I have launched test.job in the AMI and I have read it.
What is the meaning for +VMAMI = "ami-fdee0094"? Is it a
specification for Cloud Scheduler or Condor?
In Condor manual I haven't read anyting about +VMAMI.
I think that it tell Cloud Scheduler to launch an AMI. But, if
there are already execution node, is this AMI launched anyway?

test.job:
Universe = vanilla
Executable = dumb.sh
Arguments = 4 10
Log = x.log
Output = x.out
Error = x.error
priority = 10
Requirements = VMType =?= "your.vm.type"
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
+VMAMI = "ami-fdee0094"
Queue


Thanks in advice.


--
Subscription settings: http://groups.google.com/group/cloudscheduler/subscribe?hl=en

[Patrick Armstrong]

Hi there,

On 23-Apr-10, at 12:41 AM, miccloud wrote:
> 1) Does Cloud Scheduler configure automatically the Ami Condor Worker
> node, for joining the pool? (Amazon EC2)

Yep, that's how your jobs get run when you submit them. The AMI for
the machine that's configured to do this is ami-fdee0094, which you
can try booting on its own if you want to examine it.

> Or I have to write particular script to do at execution node
> startup for configure it with central manager?

No, it's configured to do this automatically.

> 2) Should I install other software as well as condors in Execution
> node that are launched by Cloud Scheduler?

Well, to do any useful work you will probably have to. What type of
job do you want to run?

> 3) I have launched test.job in the AMI and I have read it.
> What is the meaning for +VMAMI = "ami-fdee0094"? Is it a
> specification for Cloud Scheduler or Condor?
> In Condor manual I haven't read anyting about +VMAMI.

It's a cloud scheduler specific attribute, that is used to determine
which AMI should be booted to run the job you submit. The equivalent
for Nimbus clusters is VMLoc, which corresponds to the URL of the VM
image to be booted.

> I think that it tell Cloud Scheduler to launch an AMI. But, if
> there are already execution node, is this AMI launched anyway?

If there is no idle node to run the job, then yes, it will start the
AMI. Cloud Scheduler will try to start as many VMs as there are jobs.
You can limit this by modifying the cloud_resources.conf file in /etc/
cloudscheduler/ .

--patrick

[Patrick Armstrong]

I've also added a page on the Cloud Scheduler Wiki about submitting
jobs.

http://wiki.github.com/hep-gc/cloud-scheduler/submitting-a-job-for-cloud-scheduler

[michele]

Hi,
and if I have a local worker node free, how do I tell to Cloud Scheduler to do not start a new AMI?
Have I to erase the line "VMAMI " from the job file?

Or Cloud Scheduler do it automatically?

Thanks a lot.

2010/4/23 Patrick Armstrong <patr...@uvic.ca>

If there is no idle node to run the job, then yes, it will start the AMI. Cloud Scheduler will try to start as many VMs as there are jobs. You can limit this by modifying the cloud_resources.conf file in /etc/cloudscheduler/ .

[Michael Paterson]

michele wrote:
> Hi,
> and if I have a local worker node free, how do I tell to Cloud
> Scheduler to do not start a new AMI?

> Have I to erase the line "*VMAMI* " from the job file?
For non-EC2 clusters, ie Nimbus, you can remove the VMAMI line from the
job file and use VMLoc to specify your image location.
+VMLoc = "http://myserver/vms/myvmimage.img.gz"
You'll also need to put your cluster information into the
cloud_resources.conf file so Cloud Scheduler will use your cluster.

>
> Or Cloud Scheduler do it automatically?
>
> Thanks a lot.
>
>

> 2010/4/23 Patrick Armstrong <patr...@uvic.ca <mailto:patr...@uvic.ca>>

[Patrick Armstrong]

Hey there,

On 26-Apr-10, at 1:21 AM, michele wrote:
> and if I have a local worker node free, how do I tell to Cloud
> Scheduler to do not start a new AMI?
> Have I to erase the line "VMAMI " from the job file?

This isn't really a supported use case for us, but if you just submit
a regular Condor job with no VM* attributes, cloud scheduler should
just ignore them. And you'll just be using plain old Condor. If you
have any problems with this, let us know on the list, and we should be
able to help with that.

> Or Cloud Scheduler do it automatically?

Nope, it's not that smart yet. :)

[Patrick Armstrong]

On 26-Apr-10, at 9:34 AM, Michael Paterson wrote:

> michele wrote:
>> Hi,
>> and if I have a local worker node free, how do I tell to Cloud
>> Scheduler to do not start a new AMI?
>> Have I to erase the line "*VMAMI* " from the job file?
> For non-EC2 clusters, ie Nimbus, you can remove the VMAMI line from
> the job file and use VMLoc to specify your image location.
> +VMLoc = "http://myserver/vms/myvmimage.img.gz"
> You'll also need to put your cluster information into the
> cloud_resources.conf file so Cloud Scheduler will use your cluster.

Hmm, Michele, are you talking about a worker node that you've started
manually somewhere, or one that you would like to boot with Nimbus?

[michele]

I have this scenario:

My central manager is in my network whith some worker node of my local pool.
So if there are not local machine free for a new job Cloud Scheduler has to intialize an AMI on Amazon.
If local machine are free the job is executed by them and Cloud Scheduler doesn't initialize AMI.
This AMI has to do the job.

2010/4/26 Patrick Armstrong <patr...@uvic.ca>

[miccloud]

I have set vm_slots: 3 in cloud_resources.conf. But if I launch a
file.job like this cloudscheduler always launch a new ami.

Universe = vanilla
Executable = /bin/hostname
Arguments = -f

Log = x.log
Output = x.out
Error = x.error
priority = 10

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

+VMAMI = "ami-fdee0094"

Queue

Why? Does cloudscheduler stop to launch ami when the machine are
equals to vm_slots quantity?
If I launch four time file.job it always setup a new machine.

And when cloudscheduler will be able to shut down the machine that are
idle for a long time?

Thanks a lot.

On Apr 27, 4:09 pm, michele <michelepo...@gmail.com> wrote:
> I have this scenario:
>
> My central manager is in my network whith some worker node of my local pool.
> So if there are not local machine free for a new job Cloud Scheduler has to
> intialize an AMI on Amazon.
> If local machine are free the job is executed by them and Cloud Scheduler
> doesn't initialize AMI.
> This AMI has to do the job.
>

> 2010/4/26 Patrick Armstrong <patri...@uvic.ca>