Typical workflow

[Michael Cianfrocco]

Our typical workflow on Amazon's EC2: 

1. Create EBS volume & upload data.
2. Start up p2.8xlarge instance for 2D classification or 3D analysis
3. Run your jobs!

[duanj...@gmail.com]

Hi Michael, 

I am new to AWS & starcluster. I have some questions about how to setup a cluster for EM analysis. 

1. Should I launche the EM-packages-in-the-Clouds AMI instead of the free t2.micro, in order to have access to all the EM softwares? Should I launche it from "starcluster ..." or from AWS console? 

2. I got many error when edit the .starcluster/config file. It did not work for me by simply copying the config context from https://sites.google.com/site/emcloudprocessing/. In the config file, which line should I pay attention to for my specific case? 

3. Once I launched the cluster, do the nodes automatically mounted? how many are they usually for the  EM-packages-in-the-Clouds AMI ? Where to define the number of node I want to use? 

4. What's the difference between an AMI and a node?  

Sorry, I got so many questions. I look forward to hearing from you. 

Jingqi 

[Michael Cianfrocco]

Hi Jingqi,

First question - what processing step are you trying to do first? 2D classification or 3D classification? Either way, you can bypass the starcluster software all together, and just run p2.xlarge or p2.8xlarge instances directly from the AWS console. I've done tests to see that these GPU instances are very fast for 3D, and also fast for 2D classification. 

For that information: 

https://sites.google.com/site/emcloudprocessing/home/relion2

So, that means you only need to launch a single machine to do all of your processing, no clusters required!

To answer one question - an AMI is the software environment that AWS loads onto your virtual machine ('Amazing Machine Image'). 

If you follow these steps you should be good to go: 

https://sites.google.com/site/emcloudprocessing/home/relion2#TOC-How-to-use-this-GPU-computing-resource

Let me know if that helped you get started,

Mike

[Michael Cianfrocco]

Reply from Jingqi: 

What if I want to stick on cluster rather than GPU, because some of the software I commonly used like sparx doesn't have full support in GPU version? What's the common flow to setup the cluster?

The best way to start a cluster is to follow the instructions that I describe here. You have two choices for starting starclutser - either launch a virtual machine on AWS that has starcluster pre-installed or install starcluster locally on your local machine. 

In either case, you need to edit the .starcluster/config file to provide information on the type of virtual machine to make into a cluster.

Mike