Physical hardware for Round 13 and beyond

[Brian Hauer]

Effective imminently, for reasons outside our control, the Peak hardware environment is being decommissioned and will not be replaced as we had originally expected.

As a result, we aim to wrap up Round 12 a bit unconventionally: by just publishing the data we collect from a run that is executing right now.  This would be the last set of data collected from the dual Xeon E5 servers at Peak.

But going forward, starting with Round 13, we will need a new physical hardware environment.  We've always liked sharing results from a physical environment and a cloud environment.  Some people prefer to see physical hardware because the results cannot be affected by concurrent heavy utilization by other virtual machines.  Others prefer seeing cloud results because that is more aligned with their own application's production deployment.

Some options:

1. We can look for another generous hosting firm willing to participate in this project.
2. We have some servers we operate at our office that could host VMs with committed CPU cores.  This would not be a physical hardware environment, but would be (potentially) more predictable and consistent than a public cloud.
3. We might be able to put together a small hardware environment in a manner analogous to our original "i7" environment from earlier rounds.  Those i7 machines had been re-purposed workstations, a new environment would probably be purpose-built servers, but low-end.  Think ~$500 per server.
   

A challenge we had in early rounds with our i7 workstations was that a large number of frameworks were network-limited by our 1-gigabit Ethernet for the plaintext and JSON tests, and maybe even for other test types.  Ten gigabit Ethernet is still quite expensive, so for option #3 above, I would consider low-power CPUs such as 4-core Intel Atoms.  In other words, it's physical hardware, but something far less powerful than we've seen to-date, making it (presumably) more difficult to be network limited by 1-gigabit.  Though I suspect that there will still be a few that run up against the network's capacity.

I'm open to other thoughts and input!

[Fredrik Widlund]

Hi Brian, 

Some feedback from a participant in the benchmark.

Personally I feel that an important purpose of the benchmark is to find realistic limits to what currently is achievable with production quality setups. If we are not pushing the limits, then the results of the benchmarks risk being less interesting or even not very relevant. Personally I feel that scaling up with regards to cpu cores and network bandwidth is important and I think that 16 cores and a single 10 GbE should be a minimum requirement, and ideally we should try to scale up even further.

On the positive side I assume access to such production hardware for the benchmark is only required for a very limited amount of time every now and then.

I had a look around and found for example https://www.packet.net/bare-metal/ that offer what they call a "HIGH I/O MONSTER" (2 × E5-2640 v3 with 2x10 GbE LACP) bare bone server for $1.75/hour. I not familiar with the company, but perhaps that or something similar could be an option. Perhaps it is also possible to find a sponsor for these resources.

On a different note I would suggest adding for example a CoreOS setup, since many interesting and relevant networking features have been release in later kernels, for example lockless tcp listeners.

Kind regards,

Fredrik Widlund

--
You received this message because you are subscribed to the Google Groups "framework-benchmarks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchm...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-benchmarks.
For more options, visit https://groups.google.com/d/optout.

[Naoki INADA]

I like GCP. Network performance (throughput and latency) is exellent.
n1-highcpu-32 seems good to me.

[l...@hivelocity.net]

Hello Brian,

What type of hardware would you be looking for and how many servers?

Thanks,

-Lee Linton

Hivelocity Hosting

l...@hivelocity.net

https://www.hivelocity.net

https://stream.hivelocity.net

[l...@hivelocity.net]

Also brian I wanted to mention that a current customer of ours reached out to me regarding this and the possibility of Hivelocity sponsoring, or helping out to the best of our ability.

Thanks,

-Lee

Hivelocity

l...@hivelocity.net

On Tuesday, February 23, 2016 at 12:59:13 PM UTC-5, Brian Hauer wrote:

[Brian Hauer]

Hi Lee,

Thanks for reaching out on this matter!

An ideal configuration would be five machines in the roles below:

1. Linux load generator.
2. Linux web/app server.
3. Linux database server.
4. Windows web/app server.
5. Windows database server.

Presently, our project's toolset has evolved without keeping the Windows side up to date, so although an ideal configuration would include Windows servers, a minimal configuration could omit those (that is, a configuration of three servers would be workable).

We'd like all of the servers to be (roughly) equivalent in specification for simplicity in understanding the results.  That is, although some people may use a larger database server than what they use for web/app, we'd prefer to just keep everything the same specification.  Our project uses databases but is not a database benchmark, so an extra-powerful database server is not needed.

Specification wise, here is what is important to us and what's not important:

1. Network capacity is highly important.  An ideal configuration would provide 10 gigabit Ethernet.
2. CPU capacity is modestly important.  8 or more HT cores would be nice.
3. Memory capacity is not very important.  None of our current tests are large-memory tests.
4. Disk capacity is not very important.  We just need space to install software and gather data; the total need is measured in tens of gigabytes.
5. Disk speed is modestly important.

If you'd prefer concrete numbers rather than these rough ideas, just let me know.

Again, I really appreciate you reaching out to us about this!  If you'd like to discuss this further outside the mailing list, just let me know.