EC2 launch-stacks Help

[Phatty Arbuckle]

Hi,

I've been working my way through the EC2 setup (https://github.com/geotrellis/geotrellis-ec2-cluster/blob/develop/deployment/README.md) and am having a problem with the launch-stacks command.  The VPC is created, but the MesosLeader and MesosFollower both end with ROLLBACK_COMPLETE with the CREATE_FAILED ending with invalid CIDR subnet specification.  This value seems to come from the config file IP setting, but I can't figure out what value actually will work here.  I've tried a few IPs.  What exactly does "CIDR notation of office IP addresses" mean?  What should this be set to?

Thanks for your help!

-Phatty

[Hector Castro]

Hi there,

CIDR here refers to Classless Inter-Domain Routing. [0] For the `IPAccess` variable, we are looking for a valid CIDR range to open up access to leader and follower instances via EC2 security groups.

If your IP address is 216.158.51.82, and you only want to open up access to this IP, then you'd supply 216.158.51.82/32.

Hope that helps,

--
Hector

[0] https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

> --
> You received this message because you are subscribed to the Google Groups "geotrellis-user"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Chris Brown]

Ah - that is meant to restrict external access to the cluster if you would like. In our case (Azavea) our office IP is 216.158.51.82 so we restrict access to "216.158.51.82/32".

If you do not care to restrict access, you can set it to "0.0.0.0/0". You'll still need the key-pair (.pem file) to ssh into a worker though.

--

You received this message because you are subscribed to the Google Groups "geotrellis-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Brown, GIS Software Developer

Azavea |  340 N 12th St, Ste 402, Philadelphia, PA
cbr...@azavea.com | P 215.558.6191 | F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea

[Phatty Arbuckle]

Chris & Hector,

Thank you!  I was confused by that because the value shows up in both the VPC template and the Leader/Follower.  When I add the /32 or /0 after the IP then it shows up in the VPC template as:

    "Parameters": {
        "OfficeCIDR": {
            "Default": "0.0.0.0/0/32", 
            "Description": "CIDR notation of office IP addresses to allow access from", 
            "Type": "String"
        }

 

and it shows up in the Follower/Leader template as:

"OfficeCIDR": {
            "Default": "0.0.0.0/0", 
            "Description": "CIDR notation of office IP addresses", 
            "Type": "String"
        },

The first one doesn't make all that much sense with the double "/" values.

This fixed that problem.  The next one I'm hitting is 

Value (<ARN of Mesos Leader Instance Profile (optional -- may not be necessary)>) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name

 

I have created the two IAM roles from the documentation: MesosLeaderInstanceProfile & MesosFollowerInstanceProfile.  I did not include them in the config file, as it said it was optional.  What do you recommend?

Maybe related or not...why would I receive "Builds finished but no artifacts were created." when running ./gt-stack.py create-ami meso-leader (or follower)?  This had previously worked and I saw AMI's listed in the Images area of EC2.  Unfortunately, I had to delete them and was trying to re-create them, but that never seems to happen now.

Thanks for your help!!

-Phatty

[Hector Castro]

On December 8, 2015 at 11:51:12, Phatty Arbuckle (pha...@gmail.com) wrote:
> Chris & Hector,
> Thank you! I was confused by that because the value shows up in both the
> VPC template and the Leader/Follower. When I add the /32 or /0 after the
> IP then it shows up in the VPC template as:
>
> > "Parameters": {
> > "OfficeCIDR": {

> > "Default": "*0.0.0.0/0/32*",

> > "Description": "CIDR notation of office IP addresses to allow access from",
> > "Type": "String"
> > }
> >
> >
> and it shows up in the Follower/Leader template as:
>
> > "OfficeCIDR": {
> > "Default": "0.0.0.0/0",
> > "Description": "CIDR notation of office IP addresses",
> > "Type": "String"
> > },
> >
> >
> The first one doesn't make all that much sense with the double "/" values.

That looks like an inconsistency on our end. I opened the following issue to address it:

  https://github.com/geotrellis/geotrellis-ec2-cluster/issues/42

> This fixed that problem. The next one I'm hitting is
>

> > Value (> > necessary)>) for parameter iamInstanceProfile.name is invalid. Invalid IAM

> > Instance Profile name
>
>
> I have created the two IAM roles from the
> documentation: MesosLeaderInstanceProfile & MesosFollowerInstanceProfile.
> I did not include them in the config file, as it said it was optional.
> What do you recommend?

I believe that the literal message inside of that setting is being read in as the value. The message the comment meant to convey is that if that setting is not present at all in the configuration, it will default to `MesosLeaderInstanceProfile` in the Leader case, and `MesosFollowerInstanceProfile` in the Follower case.

I think if you delete those lines from the configuration file altogether, it should get past that point.

> Maybe related or not...why would I receive "Builds finished but no
> artifacts were created." when running ./gt-stack.py create-ami meso-leader
> (or follower)? This had previously worked and I saw AMI's listed in the
> Images area of EC2. Unfortunately, I had to delete them and was trying to
> re-create them, but that never seems to happen now.

Hmm. Can you please provide more of the output from that command? In addition, can you also supply the entire command you're running?

[Phatty Arbuckle]

Hector,

Again, thank you!

I'm should have figured that configuration file change would work!  Thanks for pointing it out.  I removed that (and left the value blank) and it worked through that problem.  Now, my problem is:

19:24:25 UTC-0500 ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack MesosLeader-0000 The following resource(s) failed to create: [dnsPrivateRecords]. . Rollback requested by user.

19:24:24 UTC-0500 CREATE_FAILED    AWS::Route53::RecordSetGroup dnsPrivateRecords RRSet with DNS name zookeeper.service.geotrellis-spark.internal. is not permitted in zone vpc-0000.

19:24:20 UTC-0500 CREATE_IN_PROGRESS AWS::Route53::RecordSetGroup dnsPrivateRecords

19:24:16 UTC-0500 CREATE_COMPLETE  AWS::EC2::Instance MesosLeader

19:23:28 UTC-0500 CREATE_IN_PROGRESS AWS::EC2::Instance MesosLeader Resource creation Initiated

19:23:26 UTC-0500 CREATE_IN_PROGRESS AWS::EC2::Instance MesosLeader

19:23:24 UTC-0500 CREATE_COMPLETE  AWS::EC2::SecurityGroup sgMesosLeader

19:23:21 UTC-0500 CREATE_IN_PROGRESS AWS::EC2::SecurityGroup sgMesosLeader Resource creation Initiated

19:23:04 UTC-0500 CREATE_IN_PROGRESS AWS::EC2::SecurityGroup sgMesosLeader

19:22:48 UTC-0500 CREATE_IN_PROGRESS AWS::CloudFormation::Stack MesosLeader-00000 User Initiated

Any ideas?

As for the previous comment on the create-ami command, here's what I ran & my results:

sudo ./gt-stack.py create-ami MesosLeader

2015-12-09 00:26:55,831 foo [DEBUG]:Using access key found in shared credential file.

2015-12-09 00:26:55,831 foo [DEBUG]:Using secret key found in shared credential file.

- downloading role 'build-essential', owned by azavea

- downloading role from https://github.com/azavea/ansible-build-essential/archive/0.1.0.tar.gz

- extracting azavea.build-essential to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.build-essential

- azavea.build-essential was installed successfully

- downloading role 'ntp', owned by azavea

- downloading role from https://github.com/azavea/ansible-ntp/archive/0.1.0.tar.gz

- extracting azavea.ntp to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.ntp

- azavea.ntp was installed successfully

- downloading role 'java', owned by azavea

- downloading role from https://github.com/azavea/ansible-java/archive/0.2.1.tar.gz

- extracting azavea.java to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.java

- azavea.java was installed successfully

- downloading role 'zookeeper', owned by azavea

- downloading role from https://github.com/azavea/ansible-zookeeper/archive/0.4.0.tar.gz

- extracting azavea.zookeeper to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.zookeeper

- azavea.zookeeper was installed successfully

- dependency azavea.java is already installed, skipping.

- downloading role 'mesos', owned by azavea

- downloading role from https://github.com/azavea/ansible-mesos/archive/0.2.2.tar.gz

- extracting azavea.mesos to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.mesos

- azavea.mesos was installed successfully

- dependency azavea.java is already installed, skipping.

- downloading role 'marathon', owned by azavea

- downloading role from https://github.com/azavea/ansible-marathon/archive/0.2.1.tar.gz

- extracting azavea.marathon to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.marathon

- azavea.marathon was installed successfully

- dependency azavea.java is already installed, skipping.

- downloading role 'hdfs', owned by azavea

- downloading role from https://github.com/azavea/ansible-hdfs/archive/0.5.0.tar.gz

- extracting azavea.hdfs to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.hdfs

- azavea.hdfs was installed successfully

- dependency azavea.java is already installed, skipping.

- downloading role 'libgdal-java', owned by azavea

- downloading role from https://github.com/azavea/ansible-libgdal-java/archive/0.1.0.tar.gz

- extracting azavea.libgdal-java to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.libgdal-java

- azavea.libgdal-java was installed successfully

- downloading role 'spark', owned by azavea

- downloading role from https://github.com/azavea/ansible-spark/archive/0.3.0.tar.gz

- extracting azavea.spark to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.spark

- azavea.spark was installed successfully

- dependency azavea.java is already installed, skipping.

- downloading role 'accumulo', owned by azavea

- downloading role from https://github.com/azavea/ansible-accumulo/archive/0.2.0.tar.gz

- extracting azavea.accumulo to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.accumulo

- azavea.accumulo was installed successfully

- dependency azavea.build-essential is already installed, skipping.

- dependency azavea.java is already installed, skipping.

- downloading role 'pptpd', owned by azavea

- downloading role from https://github.com/azavea/ansible-pptpd/archive/0.2.0.tar.gz

- extracting azavea.pptpd to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.pptpd

- azavea.pptpd was installed successfully

- dependency azavea.iptables-persistent is already installed, skipping.

- downloading role 'graphite', owned by azavea

- downloading role from https://github.com/azavea/ansible-graphite/archive/0.4.0.tar.gz

- extracting azavea.graphite to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.graphite

- azavea.graphite was installed successfully

- dependency azavea.python is already installed, skipping.

- dependency azavea.pip is already installed, skipping.

- dependency azavea.apache2 is already installed, skipping.

- dependency azavea.git is already installed, skipping.

- dependency azavea.memcached is already installed, skipping.

- dependency azavea.collectd is already installed, skipping.

- downloading role 'grafana', owned by azavea

- downloading role from https://github.com/azavea/ansible-grafana/archive/0.1.0.tar.gz

- extracting azavea.grafana to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.grafana

- azavea.grafana was installed successfully

- dependency azavea.elasticsearch is already installed, skipping.

- downloading role 'nginx', owned by azavea

- downloading role from https://github.com/azavea/ansible-nginx/archive/0.2.0.tar.gz

- extracting azavea.nginx to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.nginx

- azavea.nginx was installed successfully

- downloading role 'apache2', owned by azavea

- downloading role from https://github.com/azavea/ansible-apache2/archive/0.2.2.tar.gz

- extracting azavea.apache2 to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.apache2

- azavea.apache2 was installed successfully

- downloading role 'statsite', owned by azavea

- downloading role from https://github.com/azavea/ansible-statsite/archive/0.1.1.tar.gz

- extracting azavea.statsite to /home/ubuntu/downloads/geotrellis-ec2-cluster-develop/deployment/ansible/roles/azavea.statsite

- azavea.statsite was installed successfully

- dependency azavea.git is already installed, skipping.

- dependency azavea.pip is already installed, skipping.

==> Builds finished but no artifacts were created.

I look forward to hearing your thoughts!

Cheers,

Phatty

[Rob Emanuele]

Hey Phatty,

What is your geotrellis-cluster.config say for "PrivateHostedZoneName"? It's "vpc-0000", yeah? It should be "geotrellis-spark.internal"... the option is there, which makes you think it could take on a different value, but the current master of geotrellis-ec2-cluster does not allow for that. I am working on a branch that does allow you to set the zone name differently (which would have to change also in the ansible packer "cluster_name" group_vars so that the two match), you might want to check it out: https://github.com/lossyrob/geotrellis-ec2-cluster/tree/upgrade-to-spark-1.5

However my guess is that if you change PrivateHostedZoneName to be "geotrellis-spark.internal", that should work out this problem.

Hope that helps!

- Rob

[Phatty Arbuckle]

Hi Rob,

I replaced vpc-0000 with geotrellis-spark.internal, but I get the same error message with the new value (see below).  What I'm confused about is that the vpc that it does list is not the one that launch-stacks created...but rather it's the default one on EC2.  I see that the launch-stacks one was created with the name GeoTrellisSparkVPC, but it's ID is not the one that is shows up in the error message below.

Error Message:

RRSet with DNS name mesos-leader.service.geotrellis-spark.internal. is not permitted in zone vpc-b51bfbd1.

Any thoughts?

Also, did you happen to understand my other question regarding create-ami Follower/Leader?

Thanks!!!

Phatty

[Hector Castro]

On December 9, 2015 at 14:27:16, Phatty Arbuckle (pha...@gmail.com) wrote:
> Hi Rob,
> I replaced vpc-0000 with geotrellis-spark.internal, but I get the same
> error message with the new value (see below). What I'm confused about is
> that the vpc that it does list is not the one that launch-stacks
> created...but rather it's the default one on EC2. I see that the
> launch-stacks one was created with the name GeoTrellisSparkVPC, but it's ID
> is not the one that is shows up in the error message below.
>
> Error Message:
> RRSet with DNS name mesos-leader.service.geotrellis-spark.internal. is not
> permitted in zone vpc-b51bfbd1.

Hard to tell what is going on here, but it seems like this is a failure attempting to create a DNS record inside of a Route 53 hosted zone named `vpc-b51bfbd1`. Can you provide a screenshot, or names of the hosted zones in your AWS account?

> Any thoughts?
>
> Also, did you happen to understand my other question regarding create-ami
> Follower/Leader?

I believe that failure is due to the case of the argument supplied to the `create-ami` subcommand. It should look like:

  ./gt-stack.py create-ami mesos-leader

vs.

  ./gt-stack.py create-ami MesosLeader

[Rob Emanuele]

One thing to note is that the Route 53 hosted zone does not get deleted when you delete the stack in CloudFormation. So it would be worth it to try deleting the stacks, deleting the Route 53 zone, and then trying again.

You can't register a DNS record that looks like "mesos-leader.service.geotrellis-spark.internal." in a hosted zone with a name like "vpc-XXX". The Hosted zone name needs to match the DNS record. So the hosted zone would need to be named "geotrellis-spark.internal", or even "service.geotrellis-spark.internal"

[Phatty Arbuckle]

Hector & Rob,

Thanks again.  Feeling like an idiot on the create-ami problem...of course capitalizing the correct way would make all the difference!!  My apologies for wasting your time on that one.

As for the DNS issue for vpc-b51bfbd1, it was the case that it needed to be deleted from the Route 53 console.  I figured out that one a little while ago.  Good news with that is that my Leader is now up & running.  Woot woot!!!

My latest troubleshooting looks like maybe a null value is being sent with the lcMesosFollower autoscaling launch configuration??  Maybe?  I'm having a hard time tracking that one down.  Thoughts?

CREATE_FAILED    AWS::AutoScaling::LaunchConfiguration lcMesosFollower 1 validation error detected: Value '' at 'iamInstanceProfile' failed to satisfy constraint: Member must have length greater than or equal to 1

CREATE_IN_PROGRESS AWS::AutoScaling::LaunchConfiguration lcMesosFollower

CREATE_COMPLETE  AWS::EC2::SecurityGroup          sgMesosFollower

CREATE_IN_PROGRESS AWS::EC2::SecurityGroup          sgMesosFollower Resource creation Initiated

CREATE_IN_PROGRESS AWS::EC2::SecurityGroup          sgMesosFollower

CREATE_IN_PROGRESS AWS::CloudFormation::Stack       MesosFollower-1bffa6ee0cc86a0b19 User Initiated

Cheers!

Phatty

[Rob Emanuele]

Are you setting the MesosLeaderInstanceProfile and MesosFollowerInstanceProfile in the geotrellis-cluster.config file? If so, do those IAM roles exist? If not, can you try putting in those options (it says that "may not be necessary" in the config template, but perhaps that's mistaken, this will be a good test to see if we should remove that comment). The roles I had set up in my account only have the " AmazonS3FullAccess" policy attached.

[Phatty Arbuckle]

I had not set them in the config file but the IAM roles do exist with the AmazonS3FullAccess policy.  I added 'MesosFollowerInstanceProfile" (didn't do leader since it completed successfully) to the config file to test it and that fixed it!!  Two follower instances were created!!  Thank you!!!

So, it looks like the number of followers is setup for a default of 2.   Is there a reason for that?

Additionally, the StackType I used was "accumulo", but I see there are plenty of other values listed.  Do any of those other ones work?  Is there documentation I could read as to what each one provides?

I'm so ecstatic I have Geotrellis EC2 up and running!!!

Cheers,

Phatty

[Rob Emanuele]

I believe the geotrellis-ec2-cluster deployment only supports Accumulo clusters currently, but you can run code that uses the Accumulo, HDFS or S3 backends of GeoTrellis with that cluster.

You can adjust the number of followers by modifying the instance counts in the Auto Scaling Group in the EC2 console. I think 2 was an arbitrary default.

Congratulations on getting the cluster up and running! :)

Cheers,

Rob

[Phatty Arbuckle]

Hey Rob & Hector,

Now that I'm up and running, I'm going to start deep diving a bit more into GeoTrellis.  I know this is off topic, but do you have some general info or documentation or tutorials on how GeoTrellis partitions/indexes/stores images in HDFS or Accumulo? What size tiles are stored? Also, how are the spectral bands stored?

As always, thanks so much for your prompt & informative help!!

-Phatty

[Phatty Arbuckle]

Rob & Hector,

I'm confused on something.  By following the EC2 project for deployment, would the VMs that are created provision all of the pieces (Mesos, Marathon, HDFS, Accumulo, Graphite, ElasticSearch, Grafana)?  Once those VMs are up & going in EC2 how do I verify everything is up & running & installed correctly?  I tried following some of the information on the readme for local development, but don't actually see anything running on the EC2 VMs that were created.  Thoughts?

Thanks for your help!

-Phatty

...

[Phatty Arbuckle]

Hey Guys,

Just to let you know I got past this problem (for the most part). I'm still wrangling with Hadoop, but will give it a little more troubleshooting before I post more questions.

I am curious about the previous question though...if you have time.

Thanks,

Phatty

...

[Rob Emanuele]

Hey,

Sorry about the lack of documentation, we're still filling out the features we need to code before the 0.10 release, and leaving the documentation bit for last (for shame...)

For checking everything is up and running, you might want to do an "ingest", which is the process of taking raster data in GeoTiff format and storing the indexed tiles in a supported backend (local file system, HDFS, Accumulo, or S3). You can use the "spark-etl" project, which you would have to build locally using the "sbt assembly" to create an "uber jar", upload it to the master server, and then run using spark-submit. The README in spark-etl has some information about how to run it: https://github.com/geotrellis/geotrellis/tree/master/spark-etl

Do you have some sample data you want to work with? Can you describe it? (Is it on s3? File format/projection/size?)

Spectral bands are handled by a MultiBandTile type inside of GeoTrellis. So we support reading and ingesting multi band geotiffs, and those tiles store all the band information inside each tile. You can create an ingest operation that cuts town on the bands, for example if you only wanted to ingest the red and infrared bands out of imagery.

To verify things are running properly, you can run the commands mentioned at the end of the readme here (https://github.com/geotrellis/geotrellis-ec2-cluster), that are talking about the vagrant box, but instead run them on the master node of your cluster by ssh'ing into it.

Hope this helps,

Rob

--