Test Assets Proposal (Try 2)

[Tom Kent]

Proposal
========
I would like to (again) propose that the Boost organization purchase machines for the exclusive use of running our current regression test suite, with the goal of having a faster turnaround for runs on more configurations.

I am volunteering to host these machines myself and provide the power/cooling (estimated at $15/mo) and internet access required for them; as well as the most substantial cost: the time to configure and maintain them.

**Here's what I am asking the steering committee for**

The cost to the Boost organization would be not-to-exceed $1800 for the purchase of three machines, to be used exclusively for boost, roughly with the specs found here: http://pcpartpicker.com/user/teeks99/saved/6WrFf7

The rest of this message gives some background and addresses some concerns that were raised previously. I am hoping that discussion this time can be limited to a yes/no vote on the above by the steering committee.

About me
========

I'm a software engineer at Boeing and have been consistently one of the largest test runners for Boosts regression runs for the last 6+ years. I am also involved with the Boost community by performing the windows build that is uploaded to sourceforge for each beta/release, running an instance of the regression report generator, participating in peer reviews, making some (very) minor pull requests, and using Boost daily in my day-job. All in all, I spend about 15 hours each month of my volunteer time helping with the Boost project.
I currently have three machines that spend the majority of their time performing regression runs. However, two of those need to be taken offline from time to time for other work. I expect this to increase in the next few months as I ramp-up my projects I am working on for my masters degree, though the results I am generating should still always be available.

Other Concerns
==============

With my previous proposal along these lines, there was a lot of discussion about alternatives that bogged down the process. I would like to raise those concerns here (and hopefully address them) so that we don't need to discuss them again in depth.

Boost should move to per-commit tests
-------------------------------------

I strongly agree with this idea. The boost community would be better served if each library was running per-commit tests through a system like travis, appveor, etc. However, there are only a few libraries that are setup for this currently, and I believe that those already have the computing resources they need. The biggest hurdle to this processes in the re-architecting that each library will need to go through. In most cases this will probably be a small task, however it is not something that can easily be automated. This has been happening slowly for the last several years, however I do not anticipate it being available in all libraries for several more years. If the Boost organization wishes to use its resources to support this transition, it could investigate purchasing subscriptions from these groups or more to the point sponsor developers to make the necessary changes to the libraries.

In contrast, the submitted proposal would only work with the regression test system as it exists today. However, in the future this hardware could be re-purposed for these tasks.

We should start more advanced testing
-------------------------------------

There has been talk about static analysis and fuzzing tests of libraries. This also could be highly beneficial to the Boost project and would also require a substantial (possibly a lot more than is being requested here) investment in computing resources. Unfortunately to incorporate this into existing projects would take a substantial amount of developer effort.

The submitted proposal is only attempting to deal with the existing regression test system. In the future, this hardware (if it is of sufficient reliability, which it may not be as it is not server class hardware) could be re-purposed for these tasks.

There are cheaper options
-------------------------

There are definitely cheaper computers, however I have found that this combination of processor + ram + storage is a sweet spot for running Boosts regression test suite. There is a small premium that is paid for a lower power/cooling/size setup than would be typical for a desktop PC, but this is no more than 10% of the cost. I am estimating that I will pay approximately $15/mo for these three computers in power/cooling and I feel it is fair to request hardware that would not increase this cost.

There are more reliable options
-------------------------------

Again, there are definitely more reliable options. Server class hardware is designed to be substantially more reliable than consumer equipment. Unfortunately there is a steep price to pay for this. I estimate that purchasing server class equipment would lower the rate of regression runs for this amount of money by 2/3 to 1/2. In my experience consumer class equipment is sufficient for regression runs. Over the last ~8 years I have run regression tests on >15 computers, some consumer and some server-class and I have not seen any difference in run output nor have I had any of the computers die from the running of these tests. The only issue I have seen is the loss of an early model SSD due to too many writes from the tests, I am confident that the proposed SSD will not suffer this problem for many years.

Cloud machines would avoid the need for purchasing hardware
-----------------------------------------------------------

I would equally be interested in providing configuration and maintenance for cloud machines that the Boost organization would be interested in purchasing to run the regression test suite. Indeed I have already managed numerous runs using both Microsoft's Azure and Amazon's AWS. However, the cost of these machines is such that running them for approximately three months at the same rate as would be available with the machines in the proposal would equal the purchase cost of those machines.

There are cheaper ways to get cloud equipment, for example using Amazon's spot instances, which would mesh nicely with the loose reliability requirements of the regression test runs, however, this would require a developer to create a system for programatically interacting with Amazon's infrastructure, something which I do not have the time to do. Even then, it would still only take approximately one year to equal the purchase cost in the proposal for a similar amount of regression runs.

I would like to thank the Boost steering committee for their consideration of this proposal.

Sincerely,

Tom Kent

[Rob Stewart]

On February 23, 2016 9:01:50 PM EST, 'Tom Kent' via Boost Steering Committee <boost-s...@googlegroups.com> wrote:
>Proposal
>========
>I would like to (again) propose that the Boost organization purchase
>machines for the exclusive use of running our current regression test
>suite, with the goal of having a faster turnaround for runs on more
>configurations.
>
>I am volunteering to host these machines myself and provide the
>power/cooling (estimated at $15/mo) and internet access required for
>them;
>as well as the most substantial cost: the time to configure and
>maintain them.
>
>**Here's what I am asking the steering committee for**
>The cost to the Boost organization would be not-to-exceed $1800 for the
>purchase of three machines, to be used exclusively for boost, roughly
>with the specs found here:
>http://pcpartpicker.com/user/teeks99/saved/6WrFf7

[snip answers to previous questions]

Thanks for your proposal, Tom. I think your rationale is quite good for the issues raised. I appreciate your willingness to do all that you've proposed, including paying for the electricity and Internet access, plus the cooling load in your house.

While the Azure costs would be higher, that approach would avoid those extra burdens on you. I wonder if we can get Microsoft to donate the resources needed. Otherwise, I like your proposal, especially since testing is evolving and the demand for the current testing regime may well fade away by the time the equipment in question reaches end of life.

___
Rob

(Sent from my portable computation engine)

[Eric Niebler]

Considering Tom's willingness to do the grunt work and had demonstrated himself in the past, I feel this would be a good use of Boost's funds. Then again, I'm not familiar with Boost's financial situation.

\e

[Robert Ramey]

> [snip answers to previous questions]
>
> Thanks for your proposal, Tom. I think your rationale is quite good for the issues raised.

> I appreciate your willingness to do all that you've proposed,including paying for the

> electricity and Internet access, plus the cooling load in your house.
>
> While the Azure costs would be higher, that approach would avoid those extra burdens on you.
> I wonder if we can get Microsoft to donate the resources needed.
Otherwise, I like your proposal,
>
> especially since testing is evolving and the demand for the current testing regime may well fade
> away by the time the equipment in question reaches end of life.

Funny, it never occurred to me use Azure, or AWS or whatever.

I think it would be very worthwhile to chat up the right people in
microsoft about making a free Azure account for doing the testing that
Tom does. I think they could be convinced that it it would be very much
in their interest if they didn't have to provide the management that Tom
provides. On the other hand, Tom would have to learn the procedures to
deal with this platform.

I looked at travis and concluded I need more time to look at it and that
it only has a few platforms available - not the whole set that Tom tests
on. So I would say:

a) the money involved in fronting this cost is well worth it to boost.
b) It's a small amount in comparison to boosts current financial position.
c) Tom has demonstrated that he will do the work which makes this pay off.

BUT

d) I'd like us to think bigger and consider what it would take to get a
free access to Azure (or other) and to get Tom to run all his tests there.

Robert Ramey

[Jon Kalb]

Boost’s financial situation is such that we can fund this without any risk to any of our current or planned operations.

In addition to the funds raised by C++Now, Boost receives some funds from corporate matching of donated time. For example, one Boeing employee that volunteers his time working on testing Boost libraries has put in for over 250 hours of matching. I’m not certain what the match rate is, but even at minimum wage, that exceeds the cost of this request.

Jon

[Eric Niebler]

Can we come to some agreement about what will happen to the machines if/when Tom becomes too busy to manage them, or moves on to other things? I don't want to see $1800 of Boost's assets collecting dust in Tom's basement.

\e

--
You received this message because you are subscribed to the Google Groups "Boost Steering Committee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boost-steerin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mca...@boost.org]

I suspect the Foundation will be insistent that we come up with a proposal of what to do with the assets. The simplest course of action is to have Kent notify the Committee if he gets to the point of wanting to pass that baton and we could find a new home.

michael

[mca...@boost.org]

Thank you Kent for the thoughtfully crafted proposal. You have been a long-time active supporter of Boost's testing, deployment, and packaging. What most of the readers probably don't know is that your efforts are graciously added to by your employer as financial support to Boost. Thank you for all you do to support Boost and the community.

[Jon Kalb]

Steering Committee,

Thank you for voting, the matter is settled.

Tom,

Thank you for your patience with this process (and for all the work you do for Boost). The committee has approved your request.

Please work with Bradley to coordinate the acquisition.

Bradley,

Please let this email serve as notice that the Boost Steering Committee has approved this request for Tom to purchase hardware for use as test machines.

Thank you all.

Jon

[Tom Kent]

On Fri, Feb 26, 2016 at 9:07 AM, Jon Kalb <jon...@boost.org> wrote:

Steering Committee,

Thank you for voting, the matter is settled.

Tom,

Thank you for your patience with this process (and for all the work you do for Boost). The committee has approved your request.

Thank you all for taking the time to consider this proposal, and for putting your trust in me.

I didn't want to reply earlier to avoid getting off track, but I wanted to address a couple things that came up from the proposal:

1) I will consider these test assets to be available for use at the Boost organization's discretion. If the committee feels they would be better used somewhere else (or for a different purpose), I will help facilitate their move. Likewise if I am no longer able to run the tests.

2) I think that approaching Microsoft to ask for a donation of Azure credit for use in boost testing is a great idea. Over the last couple years I have run the boost test suite on Azure instances a number of times. If they were to provide instances, I would be happy to volunteer to run the regression tests on those as well. Who wants to spearhead this effort?

Sincerely,

Tom Kent

 

[Rob Stewart]

On February 26, 2016 5:19:45 PM EST, Tom Kent <tee...@gmail.com> wrote:
>
>Thank you all for taking the time to consider this proposal, and for
>putting your trust in me.

Thanks for your efforts, Tom.

>2) I think that approaching Microsoft to ask for a donation of Azure
>credit
>for use in boost testing is a great idea. Over the last couple years I
>have
>run the boost test suite on Azure instances a number of times. If they
>were
>to provide instances, I would be happy to volunteer to run the
>regression
>tests on those as well. Who wants to spearhead this effort?

The question I have is whether and how SFC should be involved in that effort.

Jon, would Amazon have reason to want to donate their services, too?