STARTING.

[Matthew Bascom]

I realize it´s a busy Monday...I´m starting an instance and it has been about 40 minutes since I ran createCluster.  Is this normal? 

From the AWS Management Console bootstrap, workspace, and update files are in place.

Also, when I submitJob, can the argument be any simple function?

Thank you.

Matt

[James Long]

My typical startup time for a cluster is 10-15 minutes. What status is
showing up in your R session? From the Elastic Map Reduce control
panel at Amazon, does it show a cluster up and running?

You probably should not be running the submitJob() function manually.
It's a support function that allows submission of segue specific jobs
to the cluster. The end user function for running jobs is the
emrlapply() function that performs an lapply() function in parallel
using Segue.

-J

[Matt Bascom]

Hi James,

The Elastic Map Reduce control panel still shows the cluster as
starting....¨2011-03-21 10:34 PST¨

- M.

On Mar 21, 11:31 am, James Long <jdl...@gmail.com> wrote:
> My typical startup time for a cluster is 10-15 minutes. What status is
> showing up in your R session? From the Elastic Map Reduce control
> panel at Amazon, does it show a cluster up and running?
>
> You probably should not be running the submitJob() function manually.
> It's a support function that allows submission of segue specific jobs
> to the cluster. The end user function for running jobs is the
> emrlapply() function that performs an lapply() function in parallel
> using Segue.
>
> -J
>

[James Long]

well then, it's not started yet ;)

how many nodes have you fired up and what size? I've only had the
start up take a long time on many really big instances (i.e. 30 xl ).
I spun up 20 small instances just a few minutes ago and it took 14
minutes.

Segue can't do anything until amazon allocates machines to you and
spins them up. So your R screen is probably just repeating "starting
<timestamp> over and over. The good news is you are not getting
charged until they are up and running.

-J

[Matt Bascom]

The createCluster parameters are set to default, so that would be two
instances with masterInstanceType = ¨m1.small¨.

They probably have a model that says, this guy has been here before,
since I´ll probably run parallel instances to calculate pi twice, sum
them, and divide by two. :)

- M.

On Mar 21, 11:59 am, James Long <jdl...@gmail.com> wrote:
> well then, it's not started yet ;)
>
> how many nodes have you fired up and what size? I've only had the
> start up take a long time on many really big instances (i.e. 30 xl ).
> I spun up 20 small instances just a few minutes ago and it took 14
> minutes.
>
> Segue can't do anything until amazon allocates machines to you and
> spins them up. So your R screen is probably just repeating "starting
> <timestamp> over and over. The good news is you are not getting
> charged until they are up and running.
>
> -J
>

[Matt Bascom]

But that wouldn´t be an apply function...hmm.

[James Long]

if it's been running that long on starting 2 instances, somethings
gone wrong. Kill your R session and then kill the EMR cluster from the
amazon control panel. Try it again and see what happens. Weird crap
like this happens sometimes.

-j

[Matt Bascom]

Okay. I´m starting down another road.

The first time I ran createCluster I had copy.image = TRUE. Now I
have copy.image = FALSE. The Elastic Map Reduce instance is still
STARTING and I´ll report back a time and whether there was a success.

Also, I tried setting location="us-west-1a", but received the
following error:

Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod",
cl, : Status Code: 400, AWS Request ID: 38883f5a-53f9-11e0-
a4c5-3522118eb217, AWS Error Code: ValidationError, AWS Error Message:
Specified Availability Zone is not supported

What is the naming convention for location?

Thank you, James!!

- M.

On Mar 21, 12:17 pm, James Long <jdl...@gmail.com> wrote:
> if it's been running that long on starting 2 instances, somethings
> gone wrong. Kill your R session and then kill the EMR cluster from the
> amazon control panel. Try it again and see what happens. Weird crap
> like this happens sometimes.
>
> -j
>

[James Long]

yeah I should have asked about the copying of the image. If you do
that, it will serialize your entire workspace and send it to S3. So if
your workspace is really big, it can take a LONG time depending on
your bandwidth.

According to this page: http://alestic.com/2009/07/ec2-availability-zones

there is no us-west-1a region. There is a us-west-1 region, however.

-J

[Matt Bascom]

According to http://status.aws.amazon.com/, there was performance
issue as early as 3-17-2011. I´m submitting an issue report here:
http://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/aws-report-issue1

- Matt

[James Long]

I don't recommend it. You have not identified that you have an issue
with AWS at all.

What, exactly, is the issue you are going to report?

-J

[Matt Bascom]

That an hour has elapsed and the AWS instance has not started.

- M.

On Mar 21, 2:11 pm, James Long <jdl...@gmail.com> wrote:
> I don't recommend it. You have not identified that you have an issue
> with AWS at all.
>
> What, exactly, is the issue you are going to report?
>
> -J
>

> On Mon, Mar 21, 2011 at 4:06 PM, Matt Bascom <mattbas...@gmail.com> wrote:
> > According tohttp://status.aws.amazon.com/, there was performance

> > issue as early as 3-17-2011.  I´m submitting an issue report here:

> >http://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/a...

[James Long]

Is this the cluster you created with copy.image=T? If so, you should
do more debugging on your end before you tell them they have a
problem. For example: How big is your workspace? How fast is your
internet
connection?

I'm just encouraging you to exercise courteous restraint. If you start
reporting system issues before you debug your end you will quickly
train AWS support team to ignore your questions.

-J

[Matthew Bascom]

Understood.  I´ll work on that.

- M.

[James Long]

Cool. I don't want you to dis-ingratiate yourself to the tech support
guys before you even get started!

Keep up the debugging and let me know if I can help!

-J

[Matthew Bascom]

Thank you, James!  :)

-M.