Folding@Home on Amazon EC2

[Eric Hammond]

I built an Amazon EC2 AMI that starts Folding@Home when an instance is
created. I have not made this AMI public yet because I'm not sure I
have all of the configuration options right for this type of
environment.

The top program shows that the CPU is being used in the following
ways:
41% nice
25% idle
34% stolen

Amazon EC2 runs on Xen virtual machines, so I think that's where the
"stolen" cycles are going, but I'm not happy with the huge "idle"
count. Since Amazon charges by the hour, I'd like to use as much CPU
as possible.

Oddly, top also consistently reports that FahCore_81.exe uses 54%CPU
which doesn't match with the above.

Are there configuration options for Folding@Home that can be used on a
dedicated server to use as much of the resources as possible?

Is it possible to run without the "nice" value?

I'm using Ubuntu on EC2 in case that makes any difference:
http://ec2gutsy.notlong.com

--
Eric Hammond

[theMASS]

> Are there configuration options for Folding@Home that can be used on a
> dedicated server to use as much of the resources as possible?

By default the client is set to use 100% CPU. There is the option to
set priority
that by default is set to "idle" which as a general rule results in
the best performance.
However that may not be true when running on Amazon's Elastic Cloud.
Anyhow you can change
priority setting under "Advanced Options" when you run -configonly.

How many processors is your account set to use? I would try the SMP
client. I would
also take a look at notfred's diskless stuff. http://reilly.homeip.net/folding/cd.html
It's
basically a folding appliance that runs on a light Linux kernel and
can essentially be run like
a VMWare appliance or as a standalone app. It does auto detection at
launch and configures
itself. The source code is on the web site. If you get it working
let us know... I'm curious :)

...Also notfred is generally very accessible via the forum referenced
on the download page.

On Nov 29, 10:51 pm, Eric Hammond <ehamm...@thinksome.com> wrote:
> I built an Amazon EC2 AMI that starts Folding@Home when an instance is
> created. I have not made this AMI public yet because I'm not sure I
> have all of the configuration options right for this type of
> environment.
>
> The top program shows that the CPU is being used in the following
> ways:
> 41% nice
> 25% idle
> 34% stolen
>
> Amazon EC2 runs on Xen virtual machines, so I think that's where the
> "stolen" cycles are going, but I'm not happy with the huge "idle"
> count. Since Amazon charges by the hour, I'd like to use as much CPU
> as possible.
>
> Oddly, top also consistently reports that FahCore_81.exe uses 54%CPU
> which doesn't match with the above.
>

>

[Eric Hammond]

On Nov 30, 2:23 am, theMASS <mikemass...@gmail.com> wrote:
> By default the client is set to use 100% CPU.

Hmmm... It's not doing that here.

From my brief review of the "cpuusage" option, it only had an effect
on Windows, not Linux (in case that's the 100% CPU you were thinking
of).

> There is the option to set priority that by default is set to "idle" which as a
> general rule results in the best performance.

Here is the complete client.cfg

[settings]
username=ec2folding
team=45104
asknet=no
machineid=1

[http]
active=no
host=localhost
port=8080

When I looked at the "priority" option, I thought it seemed to only
change the process' "nice" value which shouldn't matter in theory
since the instance is not doing anything else with the CPU.

I would like to test with nice=0 if there is anyway to configure this.

> How many processors is your account set to use? I would try the SMP
> client.

The Amazon EC2 instance class I am testing with only has a single
(virtual) CPU.

> I would also take a look at notfred's diskless stuff.http://reilly.homeip.net/folding/cd.html

Thanks for the pointer. Looks pretty cool, though I don't see right
off how it would help on Amazon EC2.

[theMASS]

This sounds like the problem ..and it doesn't look good :(

http://developer.amazonwebservices.com/connect/thread.jspa?messageID=66816&#66816

[7im]

This link works better...

http://developer.amazonwebservices.com/connect/thread.jspa?messageID=66816

This one also has more info...

http://developer.amazonwebservices.com/connect/thread.jspa?messageID=66988

[Eric Hammond]

On Nov 30, 5:30 am, theMASS <mikemass...@gmail.com> wrote:
> This sounds like the problem ..and it doesn't look good :(

I should point out that I am already familiar with the "stolen" CPU
cycles problem. The version of top I am using on the Ubuntu AMI
reports that as "%st":

Cpu(s): 0.0%us, 0.0%sy, 40.5%ni, 25.0%id, 0.0%wa, 0.0%hi,
0.0%si, 34.5%st

I'm willing to accept that as one virtual machine instance on a dual
core machine, I'm not going to get the entire CPU (i.e., some of it is
"stolen" from me and reserved for somebody else's virtual machine
instance).

What concerns me is the "%id" idle time which seems to indicate that
my virtual machine has spare cycles that it could be using for the
Folding@home application but isn't.

If I run a simple shell loop, I can get the idle to 0%:

Cpu(s): 49.0%us, 0.0%sy, 5.0%ni, 0.0%id, 0.0%wa, 0.0%hi,
0.0%si, 46.0%st

This is the type of behavior I expect to see from EC2 with
approximately half of the CPU being "stolen" from the VM when I am
using as much CPU as possible.

What I'd like to do is figure out how to get the Folding@home
application to use as much CPU as possible.

With my basic tests, the application didn't seem to be waiting on
other things like disk or network IO and the amount of memory used is
only 285MB out of 1.7GB.

If somebody would like to examine this behavior in person, send me
your public ssh key and I can set up a temporary instance for you to
study.

[theMASS]

Do you have any folding numbers? i.e. time/step on a WU? I'd like to
see if the results confirm what top is reporting.

From what I understand about EC2 (which isn't much;)) The CPU's are
"virtual" and as newer faster physical CPUs are added to the cloud
only a percentage of a physical CPU/core is assigned as a CPU. I
don't know how this effects the way CPU usage is reported.

It's interesting that the stolen cycles are greater with the shell
loop than with F@H.

Lets make sure this is a real issue and not "virtual." ;)

[Eric Hammond]

theMASS wrote:
> Do you have any folding numbers? i.e. time/step on a WU? I'd like to
> see if the results confirm what top is reporting.

How do I check this?

Here are some things in the log that look like they might relate to
performance:

[06:21:35] + Benchmarking ...
[06:21:45] The benchmark result is 675
...
[06:21:57] Working on Unit 01 [November 29 06:21:57]
...
[06:21:58] Project: 3627 (Run 66, Clone 12, Gen 5)
...
[06:22:21] Completed 0 out of 1500000 steps (0%)
...
[07:21:46] Completed 675000 out of 1500000 steps (45%)

The first timestamp is from November 29 (GMT) and the last timestamp is
from December 01 so it's made it 45% of the way (WU?) in 49 hours.

Here's the underlying CPU for this particular instance (can vary):

vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 270
stepping : 2
cpu MHz : 2004.540
cache size : 1024 KB
bogomips : 4010.77

> It's interesting that the stolen cycles are greater with the shell
> loop than with F@H.

It is a bit odd. You start out with 100% idle and 0% stolen. Then
generally the stolen percent increases to match almost exactly the used
cpu percent. The way it ends up you can generally only use up to 50% of
the CPU. It makes Munin CPU usage graphs look odd, too.

I wish it could somehow just calculate what part of the CPU I own and
treat this as 100%.

Anyway, let me know if the above info means something to you or if you'd
like ssh access to an instance running the application.

Should it take 4.5 days to finish a WU? Are all WU's equal?

Am I right in thinking that $11 (EC2 fees) is a bit expensive for a WU?

If we are in the right ballpark, though, I would also like to test the
64-bit larger instance types. They are theoretically 4x and 8x the CPU
power of this one so a WU might get done in half a day for the same $11.

I was considering eventually setting up a web site where EC2 folding
instances would be run based on contributions, but I don't know how to
judge the value EC2 is offering for this particular project.

--
Eric Hammond

[Eric Hammond]

I think I'm almost done with my Folding@home on EC2 experiments.
Thanks to theMASS for a ton of help in getting different approaches to
work and in analyzing the results at 4am a couple nights.

My initial impression is that EC2 is probably not cost effective for
Folding@home compared to alternatives like buying cheap hardware or
(obviously) running it on spare cycles on systems you already use.

It has been, however, an interesting experiment in running
applications on EC2 and prompted me to build a 64-bit large/extra
large AMI for Ubuntu 7.10 Gutsy which should be released soon.

Tests:

For the first test, I used Christer Edwards' folding installer for
Ubuntu:
http://ubuntu-tutorials.com/category/folding/
This creates a single worker (not sure of terminology here) for each
CPU.

The small instance costs $0.10/hour to run (1/2 of a dual core CPU).
It started 29 Nov 06:22:21 and has completed 1200000 out of 1500000
steps (80%) as of 2 Dec 22:12:59. The live FAH log is available
directly from the small folding instance while I leave it running:
http://ec2folding.azeelo.com/folding/FAHlog.txt
Browse around that server for other info including Munin charts,
cpuinfo, and config.cfg.

The large instance costs $0.40/hour to run (2 dual core CPUs) and took
about 24+ hours to complete its first 2 WUs (concurrently). The live
FAH logs are available directly from the medium folding instance while
I leave it running:
http://ec2folding-large.azeelo.com/folding/CPU1-FAHlog.txt
http://ec2folding-large.azeelo.com/folding/CPU2-FAHlog.txt

I tried a large instance using the SMP client following instructions
here:
http://folding.stanford.edu/English/FAQ-SMP#ntoc29
I don't have the results handy, but they did not appear to be very
good from discussions with theMASS.

I tried an xlarge instance using the non-SMP client for a while at
$0.80/hour and it's performance per WU seemed equivalent to the small
instance.

The SMP client claims to be optimized for 4 cores, so I didn't try it
on the 8 core xlarge instance.

Results:

If my calculations are correct, the large instance was folding at
about $5/WU and the small and xlarge instances were about $11/WU. The
difference between large and xlarge doesn't make any sense to me which
leads me to...

Caveats:

- I just learned about Folding@home so I may not have installed and
tweaked the configuration to get maximum performance out of EC2.

- I may not understand how to configure EC2 to get the maximum
performance out of it, especially for this application, especially on
64-bit.

- The sample WUs that were assigned may have been unusually hard.
Apparently not all WUs are created equal, and I'm not sure the best
way to judge a platform's performance without running it for a long
time (which gets expensive on EC2).

Followup:

Followup on the idle CPU part of the thread: I believe that the "top"
program is reporting idle percentage incorrectly on the small instance
type. Both vmstat and Munin reported that the CPU was completely in
use with zero percent idle, while top consistently showed 25% idle.

Followup on "stolen" CPU cycles: The correct term is apparently
"steal" not "stolen". Approximately 50% of the perceived dual-core
CPU gets assigned to steal as the VM increases CPU usage on the small
instance type. The large and xlarge instance types have 2 and 4
complete dual-core CPUs available so no steal shows up.

Offer:

I'd hate for my beginner attempts to be taken as the last word in EC2
folding, but since there seems to be nobody else trying it out or
publishing the results I figured I should toss out what I had.

If anybody else would like to take a look at folding on EC2, I can
provide AMIs, recipes, and advice for getting it running. If you are
an expert with the folding software, I can even fire up some instances
on EC2 and give you ssh access to study them.

Best,
--
Eric Hammond
http://www.anvilon.com/

[Eric Hammond]

Thread continued on the new forum:

http://foldingforum.org/viewtopic.php?t=74