What is the difference between using cbtool and using SPEC Cloud?

[Yuna]

If I want to test the EC2 cloud, what are the pros and cons of using cbtool as a standalone as opposed to the entire SPEC Cloud tool? 

[Michael R. Hines]

Hi,

Excellent question. (CCing both mailing lists).

By itself, cbtool doesn't make any intelligent decisions ---- it just
uses probability distributions to drive load to a cloud under different
configurations chosen by the user. For cbtool results to be "meaningful"
at scale, you really need something like SPEC Cloud on the other side of
the API telling cbtool what to do and how to do it in real time. That's
what SPEC Cloud does. It evalutes what the cloud is doing in real time,
enforces quality of services, and creates meaningful metrics around the
API exposed by cbtool.

As a simple example, cbtool's API has no concept of money or cost. (SPEC
does not either, but only by design.)

By separating out what the benchmark "does" and the decisions it makes
based on that data, you can pick and choose what level of behavior at
which you want the benchmark to operate. This way, cbtool itself doesn't
dictate "how" to benchmark the cloud --- it just provides a mechanism to
do it easily. Without using the API, you're kind of limited to single-VM
or single-AI micro-benchmarks. The moment you want to scale to the level
of an entire cloud, you need something smarter on the other side of the
API that actually knows what's going on.

/*
* Michael R. Hines
* Platform Engineer, DigitalOcean.
*/

[Marcio Silva]

Hello Yuna,

SPECCloud will provide you with both very well-defined set of workloads (YCSB with Cassandra backend, K-Means on Hadoop) and specific Run Rules (e.g., run a profiling phase, then keep adding new copies of the workloads until certain thresholds are reached). 

This means that at the end of the run, you will have a 2 figures of merit (Scalability and Elasticy scores) and one metric (Mean Instance Provisioning Time). The SPECCloud "run rules executor" - which is basically a CBTOOL "client", issuing API requests to the former - will nicely process the data a calculate everything without any manual intervention.

If you just use the "pure" CBTOOL, you will have the freedom to expose your Cloud to other workloads (e.g., HPCC, FIO, UNIXBENCH, Giraph, SPECjbb), but then it will be up to you the design of the "run rules" (e.g., how many copies of each workload? what should be the load level for each copy), the processing of the automatically collected raw metrics (e.g., application bandwidth, application latency) and the extraction/composition of figures of merit for your Cloud. 

My own personal evaluation of the two options, as one the two co-creators of CBTOOL: while this can certainly be done (CBTOOL was actually created as an experimental framework to test memory overcommit in Clouds), unless you have a very definite use case (e.g., a container cloud for HPC) or specific cloud resource(s) that you want to stress (e.g., Cloud block storage), you should stick to SPECCloud. 

Regards,

Marcio

-------------------------------------------------------------
Marcio A. Silva, PhD.
Software Engineer
DataCenter Systems Software
IBM Thomas J. Watson Research Center
Yorktown Heights NY 10598-0218
phone: 1-914-945-2911, fax: 1-914-945-4254
e-mail: mar...@us.ibm.com