createCluster and startCluster

[Matthew Bascom]

Hello,

I´m giving this a try again.  Where is the first place I should look to trouble shoot with the following results? Shown are two results after running two functions - the second result comes with an error code:

###
# Start first code block:
> Cluster2 <- createCluster(numInstances=2,
+ cranPackages=c("Hmisc", "plyr"),
+ copy.image = FALSE, location="us-east-1b")

STARTING - 2011-04-04 06:35:18
...
STARTING - 2011-04-04 06:38:00
BOOTSTRAPPING - 2011-04-04 06:38:36
...
BOOTSTRAPPING - 2011-04-04 07:27:42
FAILED - 2011-04-04 07:28:15

### End first code block and result.
###

# Start second code block. This time I tried the startCluster function.

startCluster(Cluster2)  ## Is startCluster internal only?
STARTING - 2011-04-04 07:29:16
...
STARTING - 2011-04-04 07:31:45
BOOTSTRAPPING - 2011-04-04 07:32:24
...
BOOTSTRAPPING - 2011-04-04 08:21:13
FAILED - 2011-04-04 08:21:49
[1] "j-2WA3WV3UF5WT3"

###  End second code block and result
###

[James Long]

Matthew, I was able to make your code run if I didn't use a specified
Availability Zone (AZ) or if I set the AZ to us-east-1a, which is the
default in Segue.

I'm observing the same output as you: the cluster stays in "starting"
mode for a VERY long time.

For now, try again using us-east-1a and I will do some testing and try
to figure out what's going on.

-J

[Matthew Bascom]

Hi James,

One observation I made while running createCluster multiple times is the duplicate buckets of R files as viewed on the AWS Management Console.  What is the relationship of the Cluster to the AWS bucket? 

- M.

[James Long]

Matt,

When createCluster() is run it creates temp files locally and also
temp buckets in S3. If stopCluster() is run on a cluster the temp
buckets on S3 are deleted (or attempted to be deleted). You've found
some cases where the cluster fails to start completely. In those cases
the temp buckets on S3 are orphaned. This is a case when Segue will
leave the buckets behind and you have to delete them manually.

I haven't figured out a way to delete temp buckets that are orphaned.
In my own case I often run Segue from multiple machines so any given
instance of Segue does not know whether a temp bucket belongs to
another Segue process or if it is orphaned. The only thing I can think
of is have Segue do some house keeping and delete out buckets over a
week old... possibly with confirmation. This may require some
housekeeping functions that the user calls. I'll give this some
thought.

-J