Aaron, et al:
I just updated Segue and pushed changes to Google code as well as
updating the tarball on google code.
When Segue fires up the Hadoop nodes, it fires them off with a
bootstrap script which loads a few things that R needs in order to run
properly. The first thing it does is load the latest version of R. The
most recent version of R needs the latest gcc in order to compile
properly. I had to add a line in the bootstrap script to load the
latest gcc. Then I had to encourage it to load the latest version of
littler. At any rate, I think it was about 3 lines of code.
For those who wonder how I debug this type of sorcery: I could tell
from the AWS console that the bootstraps routine was failing. So I
used the AWS console to fire up an interactive Pig session. I then
SSHed into the master node. From there I incrementally ran each line
of the bootstrap script until I got an error. Then I had to figure the
error out. Lather, rinse, repeat.
I've indexed the version number up to a whopping 0.05.
-J
On Mon, Jul 9, 2012 at 11:21 AM, James Long <
jdl...@gmail.com> wrote:
> Cool. Thanks.
>
> I'm able to reproduce the error and am currently trying to cook up a
> solution. Thanks for giving me such good error info. It makes ramping
> up the debugging much easier.
>
> -J
>
>
> On Mon, Jul 9, 2012 at 11:08 AM, Aaron <
aaron....@gmail.com> wrote:
>> I used the default, then tried m1.xlarge, i also tried the m1.small
>> but Segue overrode this with m1.large as per the 32 bit issue you
>> avoid.
>>
>> On Jul 9, 9:30 am, James Long <
jdl...@gmail.com> wrote:
>>> Aaron,
>>>
>>> It looks like there are some issues with the bootstrap script that
>>> loads up the instances with needed packages. I'll start debugging.
>>>
>>> Do you know which instance size you were using when you got the error?
>>>
>>> -J
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Jul 7, 2012 at 9:28 PM, Aaron <
aaron.walt...@gmail.com> wrote:
>>> > Hello There,
>>>
>>> > Been playing around as well. Made some steller progress as the simplicity
>>> > of the design but I'm getting the same error tonight? Using Segue files as
>>> > posted on July 5th. Tried it on my Mac and Ubuntu boxes with the same
>>> > results.
>>>
>>> > <snip from the logs>
>>> > Get:23
http://http.us.debian.orgtesting/non-free amd64 Packages [74.7 kB]
>>> > Fetched 17.3 MB in 13s (1242 kB/s)
>>> > Reading package lists...
>>> > Reading package lists...
>>> > Building dependency tree...
>>> > Reading state information...
>>> > Some packages could not be installed. This may mean that you have
>>> > requested an impossible situation or if you are using the unstable
>>> > distribution that some required packages have not yet been created
>>> > or been moved out of Incoming.
>>> > The following information may help to resolve the situation:
>>>
>>> > The following packages have unmet dependencies:
>>> > libc6-dev : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
>>> > libgcc1 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
>>> > libgfortran3 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
>>> > libgomp1 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
>>> > libstdc++6 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
>>>
>>> > On Tuesday, June 5, 2012 4:29:20 PM UTC-4, Hugh MacMullan wrote:
>>>
>>> >> It's alive! Thanks for the speedy response James, looking good.
>>> >> -Hugh
>>>
>>> >> On Jun 5, 12:33 pm, James Long <
jdl...@gmail.com> wrote:
>>> >> > A few things needed some love. But I could not get m1.small working
>>> >> > right. I changed a few things that would cause failures, I set segue
>>> >> > to reject m1.small and default to m1.large. And I changed the version
>>> >> > to 0.04. Give it a whirl now and let me know if it explodes.
>>>
>>> >> > -J
>>>
>>> >> > On Tue, Jun 5, 2012 at 9:04 AM, James Long <
jdl...@gmail.com> wrote:
>>> >> > > scratch that. I'm getting failures on m1.large.
>>>
>>> >> > > -J
>>>
>>> >> > > On Tue, Jun 5, 2012 at 8:57 AM, James Long <
jdl...@gmail.com> wrote:
>>> >> > >> Thanks for the head's up! I'm testing currently. I'm suspicious of
>>> >> > >> the
>>> >> > >> m1.small instance which is the default. It's 32 bit and all the other
>>> >> > >> instances are 64bit. It may be time to block m1.small and force using
>>> >> > >> 64 bit instances.... but first I have to ensure that's the real
>>> >> > >> issue.
>>>
>>> >> > >> -J
>>>
>>> >> > >> On Mon, Jun 4, 2012 at 4:08 PM, Hugh MacMullan
>>> >> > >> <
hughamacmulla...@gmail.com> wrote:
>>> >> > >>> Hi Team!
>>>
>>> >> > >>> Let me start of by saying "huzzah for Segue!" What a cool project.
>>> >> > >>> Simple is the way for my users for sure.
>>>
>>> >> > >>> I'm trying to start a new cluster (following the details on Jeffrey
>>> >> > >>> Breen's awesome "Abusing" article), and it seems to be failing at
>>> >> > >>> the
>>> >> > >>> R installation portion of the process, a dependency check failure
>>> >> > >>> (see
>>> >> > >>> OUTPUT below).
>>>
>>> >> > >>> Any workaround? Any other details I can provide?
>>>
>>> >> > >>> AMI was: ami-a9b362c0 (if that's not set in stone?)
>>>
>>> >> > >>> Thought I would report!
>>>
>>> >> > >>> -Hugh
>>>
>>> >> > >>> P.S. -- what I did:
>>> >> > >>>> library(segue)
>>> >> > >>> library(segue)
>>> >> > >>> Loading required package: rJava
>>> >> > >>> Loading required package: caTools
>>> >> > >>> Loading required package: bitops
>>> >> > >>> Segue did not find your AWS credentials. Please run the
>>> >> > >>> setCredentials() function.
>>> >> > >>>> setCredentials('MYCRED','MYCRED2')
>>> >> > >>> setCredentials('MYCRED', 'MYCRED2')
>>> >> > >>>> myCluster <- createCluster(numInstances=2)
>>> >> > >>> myCluster <- createCluster(numInstances=2)
>>>
>>> >> > >>> then STARTING ... BOOTSTRAPPING ... SHUTTING_DOWN ... FAILED
>>>
>>> >> > >>> =========== OUTPUT (stdout is the kicker) ===========
>>>
>>> >> > >>> controller:
>>> >> > >>> 2012-06-04T17:13:43.823Z INFO Fetching file
>>> >> > >>> 's3://rtmpveqqfdfzmifbjs0w-
>>> >> > >>> segue/bootstrapLatestR.sh'
>>> >> > >>> 2012-06-04T17:13:44.881Z INFO Working dir /mnt/var/lib/bootstrap-
>>> >> > >>> actions/1
>>> >> > >>> 2012-06-04T17:13:44.881Z INFO Executing
>>> >> > >>> /mnt/var/lib/bootstrap-actions/
>>> >> > >>> 1/bootstrapLatestR.sh
>>> >> > >>> 2012-06-04T17:14:11.902Z INFO Execution ended with ret val 100
>>> >> > >>> 2012-06-04T17:14:11.903Z ERROR Execution failed with code '100'
>>>
>>> >> > >>> stderr:
>>> >> > >>> + sudo tee /etc/apt/sources.list.d/R.list
>>> >> > >>> + gpg --keyserver
pgp.mit.edu --recv-key 381BA480
>>> >> > >>> gpg: directory `/home/hadoop/.gnupg' created
>>> >> > >>> gpg: new configuration file `/home/hadoop/.gnupg/gpg.conf' created
>>> >> > >>> gpg: WARNING: options in `/home/hadoop/.gnupg/gpg.conf' are not yet
>>> >> > >>> active during this run
>>> >> > >>> gpg: keyring `/home/hadoop/.gnupg/secring.gpg' created
>>> >> > >>> gpg: keyring `/home/hadoop/.gnupg/pubring.gpg' created
>>> >> > >>> gpg: requesting key 381BA480 from hkp server
pgp.mit.edu
>>> >> > >>> gpg: /home/hadoop/.gnupg/trustdb.gpg: trustdb created
>>> >> > >>> gpg: key 381BA480: public key "Johannes Ranke (CRAN Debian archive)
>>> >> > >>> <
jra...@uni-bremen.de>" imported
>>> >> > >>> gpg: no ultimately trusted keys found
>>> >> > >>> gpg: Total number processed: 1
>>> >> > >>> gpg: imported: 1
>>> >> > >>> + gpg -a --export 381BA480
>>> >> > >>> + sudo apt-key add jranke_cran.asc
>>> >> > >>> + sudo apt-get update
>>> >> > >>> + sudo DEBIAN_FRONTEND=noninteractive apt-get install -t testing
>>> >> > >>> --yes
>>> >> > >>> --force-yes --no-install-recommends r-base
>>> >> > >>> E: Broken packages
>>>
>>> >> > >>> stdout:
>>> >> > >>> # Change these lines if you don't want to use the main CRAN mirror.
>>> >> > >>> # debian R upgrade
>>> >> > >>> debhttp://
cran.r-project.org/bin/linux/debiansqueeze-cran/
>>> >> > >>> # deb-srchttp://
cran.r-project.org/cran/bin/linux/debiansqueeze-
>>> >> > >>> cran/
>>>
>>> >> > >>> OK
>>> >> > >>> Get:1
http://security.debian.orgsqueeze/updatesRelease.gpg [836 B]
>>> >> > >>> Ignhttp://
security.debian.org/squeeze/updates/contribTranslation-en
>>> >> > >>> Ignhttp://
security.debian.org/squeeze/updates/mainTranslation-en
>>> >> > >>> Ignhttp://
security.debian.org/squeeze/updates/non-freeTranslation-
>>> >> > >>> en
>>> >> > >>> Get:2
http://security.debian.orgsqueeze/updatesRelease [86.9 kB]
>>> >> > >>> Get:3
http://http.us.debian.orgsqueezeRelease.gpg [1672 B]
>>> >> > >>> Ignhttp://
http.us.debian.org/debian/squeeze/contribTranslation-en
>>> >> > >>> Ignhttp://
http.us.debian.org/debian/squeeze/mainTranslation-en
>>> >> > >>> Ignhttp://
http.us.debian.org/debian/squeeze/non-freeTranslation-en
>>> >> > >>> Get:4
http://http.us.debian.orgtestingRelease.gpg [836 B]
>>> >> > >>> Get:5
http://http.us.debian.org/debian/testing/contribTranslation-en
>>> >> > >>> [35.0 kB]
>>> >> > >>> Get:6
http://http.us.debian.org/debian/testing/mainTranslation-en
>>> >> > >>> [3857 kB]
>>> >> > >>> Get:7
http://security.debian.orgsqueeze/updates/maini386 Packages
>>> >> > >>> [229 kB]
>>> >> > >>> Get:8
http://http.us.debian.org/debian/testing/non-freeTranslation-
>>> >> > >>> en [62.7 kB]
>>> >> > >>> Get:9
http://http.us.debian.orgsqueezeRelease [111 kB]
>>> >> > >>> Get:10
http://http.us.debian.orgtestingRelease [190 kB]
>>> >> > >>> Get:11
http://cran.r-project.orgsqueeze-cran/Release.gpg [198 B]
>>> >> > >>> Ignhttp://
cran.r-project.org/bin/linux/debian/squeeze-cran/
>>> >> > >>> Translation-en
>>> >> > >>> Get:12
http://security.debian.orgsqueeze/updates/contribi386
>>> >> > >>> Packages [674 B]
>>> >> > >>> Get:13
http://security.debian.orgsqueeze/updates/non-freei386
>>> >> > >>> Packages [14 B]
>>> >> > >>> Get:14
http://cran.r-project.orgsqueeze-cran/Release [3651 B]
>>> >> > >>> Get:15
http://http.us.debian.orgsqueeze/maini386 Packages [6568 kB]
>>> >> > >>> Get:16
http://cran.r-project.orgsqueeze-cran/Packages [16.9 kB]
>>> >> > >>> Get:17
http://http.us.debian.orgsqueeze/contribi386 Packages [54.1
>>> >> > >>> kB]
>>> >> > >>> Hithttp://http.us.debian.orgsqueeze/non-freei386 Packages
>>> >> > >>> Get:18
http://http.us.debian.orgtesting/maini386 Packages/DiffIndex
>>> >> > >>> [7876 B]
>>> >> > >>> Get:19
http://http.us.debian.orgtesting/contribi386 Packages/
>>> >> > >>> DiffIndex [7819 B]
>>> >> > >>> Get:20
http://http.us.debian.orgtesting/non-freei386 Packages/
>>> >> > >>> DiffIndex [7819 B]
>>> >> > >>> Get:21
http://http.us.debian.orgtesting/maini386 Packages [5837 kB]
>>> >> > >>> Get:22
http://http.us.debian.orgtesting/contribi386 Packages [43.9
>>> >> > >>> kB]
>>> >> > >>> Get:23
http://http.us.debian.orgtesting/non-freei386 Packages [71.4
>>> >> > >>> kB]
>>> >> > >>> Fetched 17.2 MB in 18s (952 kB/s)
>>> >> > >>> Reading package lists...
>>> >> > >>> Reading package lists...
>>> >> > >>> Building dependency tree...
>>> >> > >>> Reading state information...
>>> >> > >>> Some packages could not be installed. This may mean that you have
>>> >> > >>> requested an impossible situation or if you are using the unstable
>>> >> > >>> distribution that some required packages have not yet been created
>>> >> > >>> or been moved out of Incoming.
>>> >> > >>> The following information may help to resolve the situation:
>>>
>>> >> > >>> The following packages have unmet dependencies:
>>> >> > >>> libc6-dev : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be
>>> >> > >>> installed