Re: [help] Create a MPI cluster using GENI nodes

[Vicraj Thomas]

Hello Ranajeet,

The geni-users mailing list is the best place to ask such questions
since there is a good chance somebody there has tried to run MPI on
GENI. If you are not a member of geni-users, please join that list
(http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList).

I believe Paul Ruth of RENCI is running MPI-based experiments on GENI.
He is on geni-users and might be able to help you.

Best wishes,

< Vic Thomas




On 3/27/14 4:30 PM, Ranajeet Anand wrote:
> Hello,
>
> I am a computer engineering student in Clemson University and I am
> working on the GENI testbed as a part of my course.
>
> I wanted to inquire if it is possible to install MPI (Message Passing
> Interface) on GENI nodes. I wish to create a homogenous cluster out of
> several GENI nodes and run parallel jobs using MPI on this cluster.
> Could you please inform me if this is possible? Also what steps would I
> need to undertake or what tools would I have to use to establish a
> cluster of MPI nodes on GENI?
>
> Any help in this direction would be greatly appreciated
>
> Thanks,
> Ranajeet Anand

[Paul Ruth]

Ranajeet,

GENI provides the basic compute and network infrastructure for your experiments and you can install just about any software you want inside of the virtual machines.   One way to do this would be to start some VMs and configure them with MPI after they are up.   Another way would be to create a custom image that already contains MPI (and possibly your application and data).   

I have an image that I use to run some MPI applications.  For now it only works on ExoGENI (although we are working on porting it to InstaGENI).   It might not be too much work for me to strip out my applications and data and share the image.    The image is Centos 6.3 with Open MPI installed from the yum repos.   Would this work for you?  If so, I can probably build this image later today and send you the url/hash.

How familiar are you with MPI?  What does your application do? Depending on what your application is requires you might need to customize the image a bit.

Paul

 

--
GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.

If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
--- You received this message because you are subscribed to the Google Groups "GENI Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ran...@g.clemson.edu]

 Hello Paul,

I am using an Ubuntu 12.04 image on VM nodes on InstaGENI. Is it possible to use a CentOS image on InstaGENI? If so, how do we create a cluster out of the VMs and run MPI jobs? I am familiar with the Palmetto Cluster computing system of Clemson University where there are job scripts to run MPI jobs.

My application is a parallel matrix multiplication that can be performed on any square number of processors (say, 4, 9, 16, etc). A root process will read input matrices stored in 2 files, scatter the data to all processes, and gather the final product into a product matrix and write it to a third file.

Regards,
Ranajeet

[Leigh Stoller]

> I am using an Ubuntu 12.04 image on VM nodes on InstaGENI. Is it possible to use a CentOS image on InstaGENI?

Yes, there is a CENTOS63-64-STD available on the InstaGeni racks, which
you can use for your VM.

Leigh

[Paul Ruth]

Like Leigh said, you can use Centos on InstaGENI and install any software you wish.    If you want to use the MPI image that I have been using, it will only work on ExoGENI. 

If you are running HPC applications that require significant computation you probably want to use ExoGEN anyway.  ExoGENI can provide more powerful VMs (up to 4 cores, 12 GB memory, 75 GB local disk, high bandwidth network connections: technically limited to 10 Gbps but have 4-5 Gbps limit in practice).    Further, the CPU cores are not oversubscribed, so you will get more predicable performance results.    

If you need more than the 4 racks that are available through the GENI portal we have other tools (Flukes) that provide access to 12 racks on 3 continents. 

Let me know if you want to try ExoGENI.

Paul

[Ranajeet Anand]

Hello Paul,

Yes, I would like to work on ExoGENI. If I understand correctly, if I use 4 racks, 16 cores should be available for the computation. If so, I think 4 racks should be sufficient.

Thanks,

Ranajeet

[Paul Ruth]

I have put together an MPI image that includes a simple "helloMPIworld"  example.   It will work on a single node or any number of nodes with any number of cores.

The image is available here:

URL:  http://geni-images.renci.org/images/standard/centos-MPI/centos6.3-MPI-v0.2.xml

Hash: 79f585b6ac301b219deffabd3bc05176807a68ad

If you are using Flack/Omni, add an ExoGENI VM and specify its image "Name" as the URL above and its "version" as the hash above.   If you are using Flukes the image URL and Hash above.

First, try creating a single VM on one of the ExoGENI racks.  It will work fine with several VMs connected by a broadcast network but try a single VM first.  Once the VM has booted login to the VM and follow the directions.  If you use the instance type XOXLarge you will have 4 cores, 12 GB memory, and 75 GB disk.   

Let me know if/when you have problems.

Paul

****Note these directions are in a README located in /home/mpiuser/README.helloMPIWorld***

Step 1: Become the mpiuser user 

> su mpiuser

> cd ~

Step 2: Add your compute nodes to your known_hosts files

For each compute node in your slice (including the first node)

> ssh-keyscan IP_OF_COMPUTE_NODE  >> ~/.ssh/known_hosts

Example: ssh-keyscan 172.16.0.1 >> ~/.ssh/known_hosts

Step 3: Add your compute nodes to your machines file 

For each compute node in your slice

> echo IP_OF_COMPUTE_NODE  >>  machines

Example: echo 172.16.0.1 >> machines

Step 4: Compile the application

>  /opt/bin/mpicc helloMPIWorld.c

Step 5: Copy the executable to each other worker

> scp a.out IP_OF_COMPUTE_NODE:.

Example: scp a.out 172.16.0.2:.

Step 6: Run the application

>  /opt/bin/mpirun -n 16 -machinefile machines a.out

Hello world from process 2 of 16 on node Node0

Hello world from process 6 of 16 on node Node0

Hello world from process 0 of 16 on node Node0

Hello world from process 14 of 16 on node Node0

Hello world from process 8 of 16 on node Node0

Hello world from process 10 of 16 on node Node0

Hello world from process 3 of 16 on node Node1

Hello world from process 12 of 16 on node Node0

Hello world from process 9 of 16 on node Node1

Hello world from process 5 of 16 on node Node1

Hello world from process 7 of 16 on node Node1

Hello world from process 4 of 16 on node Node0

Hello world from process 15 of 16 on node Node1

Hello world from process 1 of 16 on node Node1

Hello world from process 11 of 16 on node Node1

Hello world from process 13 of 16 on node Node1

[Ranajeet Anand]

Hi Paul,

Thank you for the information.

I tried to access the link but the centos6.3-MPI-v0.2.tgz  does not exist. Did you mean either of centos6.3-MPI-v0.2a.tgz, centos6.3-MPI-v0.2b.tgz, or centos6.3-MPI-v0.2c.tgz files on the server?

[Paul Ruth]

Try again with centos6.3-MPI-v0.2.tgz.   The others were test images I mad while creating the final version.   I must have forgotten to push out the final version.  It is there now.

Paul

[Paul Ruth]

Ranajeet,

Where you able to get an MPI cluster working on ExoGENI?

Paul

[Paul Ruth]

Its good to hear that you got this to work.  I'll try to answer some of your questions.

XOXLarge: 4 cores

XOLarge: 2 cores

XOMedium: 1 core

XOSmall: 1 core

More ExoGENI resource type information can be found here: https://wiki.exogeni.net/doku.php?id=public:experimenters:resource_types:start

When you run MPI you are specifying how many processes you want.  The system will allocate the processes to nodes in a round-robin fashion by default.   So, even if you have only one core on one node, you can create as many MPI processes as the node can handle without running out of resources (i.e. memory, file descriptors, etc.).    

If you just want to become familiar with MPI you will likely be able to run your 25 process job on a single node.   However, after you figure out how to use a single node,  you should try to use 2 or more nodes so that you learn how it works in the truly distributed environment.   Further, if you care about performance, you will likely want to limit MPI to assigning one processes per core and will need many nodes for interesting applications.

I'm not sure why you are seeing that error.   Does anyone else on the list know about that error?

Paul

On Tue, Apr 8, 2014 at 2:21 PM, Ranajeet Anand <ran...@g.clemson.edu> wrote:

Thanks Paul. I was able to run the MPI "hello world" program.

I wanted to ask how many cores are present on the XOXLarge, Medium and Small type of instances. You mentioned that the XOXLarge instance has 4 cores but I was able to specify more than 16 processes for the hello world program.

I need up to 25 processes to execute the MPI program, so I was wondering if I should set up 2 nodes of instances XOXLarge or a combination of large, medium and small would do.

Also, I tried to set up 1 more node of instance XOXLarge but Flack was unable to do so (Status was Allocation: Unallocated, Operational: Staged) so, I was wondering why that was happening

Thanks,
Ranajeet

On Tue, Apr 8, 2014 at 8:59 AM, Paul Ruth <paul...@gmail.com> wrote:

You should have sudo privileges.   

Try: 

sudo su mpiuser

Paul

On Mon, Apr 7, 2014 at 5:55 PM, Ranajeet Anand <ran...@g.clemson.edu> wrote:

Hello Paul,

I was working on setting up the ExoGENI VM on exogeni.net and set up the sliver type as XOXLarge. I was able to ssh to the node but I am unable to become the superuser.

Should the node login password be used for following step? Or is a specific password needed for that?

su mpiuser

Thanks,
Ranajeet Anand

[Ranajeet Anand]

Forgot to put this on the help group

---------- Forwarded message ----------
From: Ranajeet Anand <ran...@g.clemson.edu>
Date: Tue, Apr 15, 2014 at 11:11 AM
Subject: Re: [geni-users] Re: [help] Create a MPI cluster using GENI nodes
To: Paul Ruth <paul...@gmail.com>

Hi Paul,

I wanted to ask if you're aware if any of the ExoGENI aggregates are functional. Whenever I start Flack, the ExoGENI resources are not loaded and so I was wondering if there has been an outage of those aggregates,

Thanks,

Ranajeet

On Fri, Apr 11, 2014 at 6:10 PM, Ranajeet Anand <ran...@g.clemson.edu> wrote:

Hi Paul,

Alright, so that explains how I am able to use several processes even on the XOSmall instance. My algorithm can run on the MPI processes but I want to check the communication bandwidth between all the processes. Therefore, like you said, I'll have to attach one process to a single core and I wanted to ask if you could help me with that.

According to my algorithm, the processes are present in a rectangular grid-like fashion with each process connected to 4 neighboring processes. So, I'm thinking of setting up a grid of XOSmall instances and run my MPI program.

Thanks,
Ranajeet

[Paul Ruth]

None of the ExoGENI resources are up right now.   The should be up soon.

Paul

--
GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.
 
If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
---
You received this message because you are subscribed to the Google Groups "GENI Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+...@googlegroups.com.