Error in lines[i]] : subscript out of bounds

[David]

Greetings,

I ran the two examples from Jeffrey Breen's post

http://jeffreybreen.wordpress.com/2011/01/10/segue-r-to-amazon-elastic-mapreduce-hadoop/

and in both cases after the WAITING phase I get the error

Error in lines[[i]] : subscript out of bounds

I am running segue on Ubuntu (as a guest in Virtualbox under a Win 7
host) with the tar package from here:

http://code.google.com/p/segue/downloads/list (which is
segue_0.02.tar.gz) (I did not build the
package myself).

Any ideas of what could be going on would be appreciated.

Thank you, David

[James Long]

David, that error tends to be the manifestation of many errors in Segue (unfortunately). The single biggest struggle I've had with maintaining Segue is keeping up with R changes. I suspect something changed with the R install on the nodes that's causing the problem. I'll try to run the example tomorrow and see if I can recreate. Thanks for the head's up.

-J

[David]

Hi James, I should have mentioned I am running R version 2.12.1

Actually, the second example in the linked blog post is yours, the pi
monte carlo
simulation.

I have found a couple of blog posts of people who have run these same
examples
with no mention of any trouble.

Thank you, David

On May 11, 8:14 pm, James Long <jdl...@gmail.com> wrote:
> David, that error tends to be the manifestation of many errors in Segue (unfortunately). The single biggest struggle I've had with maintaining Segue is keeping up with R changes. I suspect something changed with the R install on the nodes that's causing the problem. I'll try to run the example tomorrow and see if I can recreate. Thanks for the head's up.
>
> -J
>

> On May 11, 2011, at 9:39 PM, David <david.anis...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Greetings,
>
> > I ran the two examples from Jeffrey Breen's post
>

> >http://jeffreybreen.wordpress.com/2011/01/10/segue-r-to-amazon-elasti...

[James Long]

David, I just installed and ran a series of tests on my Ubuntu box
with Segue and had no problems. Can you try upgrading R to the latest
version and letting me know if that fixes things? I thought Segue
would run just fine under 2.12, but it's possible that the
compatibility was recently broken.

-J

[David]

James - I just updated R to version 2.13.0, then run the first example
and got the
same error again.

I had to reinstall the package for some reason. It required the
package caTools
as a dependency. So I installed that first. Then installed the segue
package locally.

Anything else I can try?

Thanks, David

[James Long]

Well this is getting to be quite a mystery. I have uploaded to google
code the latest build of the package. I replaced the old tar ball. So
try downloading the tar ball again and reinstalling Segue:

http://segue.googlecode.com/files/segue_0.02.tar.gz

-J

[David]

Yes! That worked. Awesome.

Wow. The possibilities boggle the mind.

Now I need to go see how really embarrassingly parallel my stuff
is........

Thank you! David

[James Long]

Fantastic!

I'm working on the next generation of Segue so stay tuned. Instead of continuing making changes, I think I'll build around a different core technology. So stay tuned.

Good luck!

-J

Sent from my iPhone.

[Vik Paruchuri]

Thank you for making such an interesting package!  I would love to make use of it, but I am having a few issues running emrlapply.

I am actually getting the same error referenced in the topic of this post, namely Error in lines[[i]] : subscript out of bounds .

I am running R 2.15 on Ubuntu Natty 11.04, and am using the latest build of segue.  Any insight would be much appreciated.

Thanks,
Vik

[James Long]

Thanks for your interest in Segue, Vic.

I'm not completely sure what's going on, but I'm able to reproduce the
error. It was pointed out to me earlier today in this stackoverflow
question:
http://stackoverflow.com/questions/10773680/emrlapply-of-segue-package-doesnt-work-on-simple-task

Here's what I've figured out so far:

No jobs are actually running on the EMR server. When an R job starts
one of the things it does first is load a few packages. One of those,
bitops, a dependance for caTools, is failing because of lack of a
namespace. That appears to be a symptom of some other issues. So I'm
trying to figure out what has changed recently that might cause this.
When I get it figured out I'll post an answer on stackoverflow and
also to this mailing list.

-J

[Vik Paruchuri]

Thanks a lot for the quick response, and I appreciate you looking into
the issue. I should have searched SO more thoroughly.

On May 29, 6:46 pm, James Long <jdl...@gmail.com> wrote:
> Thanks for your interest in Segue, Vic.
>
> I'm not completely sure what's going on, but I'm able to reproduce the
> error. It was pointed out to me earlier today in this stackoverflow

> question:http://stackoverflow.com/questions/10773680/emrlapply-of-segue-packag...

[James Long]

Actually I appreciate you hitting the mailing list. I don't check
stackoverflow as much as I check email so hitting me on email was a
very good idea.

I'm testing the fix right now. I had to learn so things about debian
package management.

-J

[James Long]

Well that turned out to be a royal pain.

New version of the tar ball is here:
http://code.google.com/p/segue/downloads/list
or you can clone the source and build it yourself, if you're into that
sort of thing.

In short, it appears that Amazon changed the EMR service to default to
the 1.0 version of the EMR AMI if no specific version was called.
Since Jan 1, the behavior had been to default to the latest version.
When I made the changes to default to a recent version I then had
issues with the current incarnation of Hadoop wanting output to be put
in a sub-bucket on S3. Not hard to do, but hard to figure out why it
was failing.

So all that's been patched up and it should work. Please let me know
if you have any issues.

-J

[Vik Paruchuri]

Thanks a lot for the quick fix, James.  I am going to test it out shortly, and I will let you know how it looks.

[Andrew Harris]

I'm getting this error now can can't figure out where it's coming from. I'm on R 2.15 and just downloaded the segue package 0.05. Is this from another amazon or R change or is there something I need to look into?

thanks,

andrew

[James Long]

This is a common error when things don't get set up on the worker
nodes properly. I'll try to test it soon, but it will likely be the
end of the week before I can make much progress.

Thanks for pointing this out.

-J

> --
> You received this message because you are subscribed to the Google Groups
> "Segue for R" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to segue-r+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Andrew Harris]

Do you have any guidance on where to look on how the nodes were being set up? I'm trying to load a few packages as well as R objects, but don't know how to validate that it has all been set up on each node correctly. 

I'm new to AWS and not quite sure how all of it is working. Thanks for looking into it when you get a chance.

[James Long]

The way I test is to spin up an EMR single instance (interactive mode
which I then SSH into) and manually run the bootstrapLatestR.sh script
and see what fails. Often I manually run each line of the script:

https://code.google.com/p/segue/source/browse/inst/bootstrapLatestR.sh

Let me know if you get time to try that and narrow down the bug. I'm
betting the install of R is failing (line 52).

-J

[James Long]

Andrew, I did some Segue testing yesterday and had no problems. Were
you able to reproduce the error?

-JD

On Mon, May 20, 2013 at 1:23 PM, Andrew Harris <andrew...@paceco.com> wrote: