Problem using segue

[CJ]

Hi

I've installed the segue package (via Hg) and am following the simple example at http://jeffreybreen.wordpress.com/2011/01/10/segue-r-to-amazon-elastic-mapreduce-hadoop/

When I createCluster, some instances come up, one gets bootstrapped, and then things go wrong. Here are a couple of examples:

Example 1:

> myCluster <- createCluster(numInstances=2)

STARTING - 2011-06-13 21:56:58
STARTING - 2011-06-13 21:57:29
STARTING - 2011-06-13 21:58:00
STARTING - 2011-06-13 21:58:30
STARTING - 2011-06-13 21:59:02
STARTING - 2011-06-13 21:59:33
BOOTSTRAPPING - 2011-06-13 22:00:03
SHUTTING_DOWN - 2011-06-13 22:00:34
FAILED - 2011-06-13 22:01:05

Example 2:

> myCluster <- createCluster(numInstances=2)

STARTING - 2011-06-13 22:50:36
STARTING - 2011-06-13 22:51:07
STARTING - 2011-06-13 22:51:38
STARTING - 2011-06-13 22:52:09
STARTING - 2011-06-13 22:52:39
STARTING - 2011-06-13 22:53:10
BOOTSTRAPPING - 2011-06-13 22:53:40
FAILED - 2011-06-13 22:54:11


I'd be grateful for advice. I'm also intrigued as to why there are 6 "STARTING" messages when I'm requesting two instances??

CJ

[James Long]

Hey CJ, the starting messages are the result of the poling timer.
Segue polls the EMR API every few seconds and returns the status of
the cluster. So the number of "starting" messages is a function of how
long it takes the cluster to come up, not how many machines are
running.

Now that does not explain the failure. I'll try to run a test later on
today to see what's up. Thanks for letting me know.

-J

[James Long]

CJ, I have reproduced the error. I'll review the bootstrap script and
try to figure out what's going on.

-J

[CJ]

Hi J

Thanks for looking into this for me. Segue looks awesome, and I'm really looking forward to using it.

CJ

[john locke]

Hi guys,

I am having the same issue. It looks like it completes Bootstrap
action item number 1, but fails on the second action item. My guess is
that R is not being installed correctly on the first bootstrapping
step.

The main failure of update.R is:
ec2-50-16-159-36.compute-1.amazonaws.com

Here is the stderr for the bootstrapLatestR.sh:
gpg: directory `/home/hadoop/.gnupg' created
gpg: new configuration file `/home/hadoop/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/hadoop/.gnupg/gpg.conf' are not yet
active during this run
gpg: keyring `/home/hadoop/.gnupg/secring.gpg' created
gpg: keyring `/home/hadoop/.gnupg/pubring.gpg' created
gpg: requesting key 381BA480 from hkp server pgp.mit.edu
gpg: /home/hadoop/.gnupg/trustdb.gpg: trustdb created
gpg: key 381BA480: public key "Johannes Ranke (CRAN Debian archive)
<jra...@uni-bremen.de>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg: imported: 1
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.
E: Wow, you exceeded the number of versions this APT is capable of.
E: Problem with MergeList /var/lib/apt/lists/
http.us.debian.org_debian_dists_testing_main_binary-i386_Packages
E: The package lists or status file could not be parsed or opened.

[James Long]

Yes, there's an issue with getting the latest version of R to install during bootstrap. Fortunately the EMR team is fond of R and the Segue project so I'm getting some help from the gents in Seattle.

In short Segue is not working until we can get this ironed out. This raises some important issues with the Segue architecture which I hope to address as soon as we get this flat tire fixed.

I'm really sorry for the inconvenience. Thanks for pointing this out, however. If it weren't for users letting me know Issues, I'd be in the dark.

Thx,

-J

Sent from my iPhone.

[other email]

That's pretty cool, I'm glad amazon is supportive. segue sounds very exciting and I'm looking forward to using it. 

[James Long]

OK guys. The EMR team at Amazon really came through for me. They
helped me craft a new bootstrap script that appears to load R
properly. The slight downside is that the version of R on each node
will be 2.11.1 which is about a year old.

I'm working on a solution to get around this, but at least getting to
R 2.11 on the nodes will make Segue useful to most of us.

Please test and let me know if you run into any issues.

I did NOT change the version number because I'm lazy. But I did
upgrade the tarball.

-JD

[CJ]

Brilliant, thanks. I look forward to trying it out.

-CJ