getting rid of subtests in roachtest

[Andrei Matei]

Hey Peter,

Given that momentum has built for revamping roachtest and getting it to have a proper work scheduler which can reuse clusters across tests (as alluded to in 28921), would you agree that, in this brave new world, subtests don't make sense any more and thus we can get rid of the concept?

For context, today, I believe the only purpose that subtests serve is indicating that different tests can share a cluster (i.e. all subtests of a top-level test share a cluster). The (roachprod) cluster is wiped in between siblings, but not otherwise destroyed and recreated.
Parent tests do not contain any logic, only leaves do. Only roots can specify a cluster spec. It's unclear what, if anything, intermediate nodes (not roots, not leaves) can specify (maybe only skipping?). It's unclear who's supposed to specify other things like MinVersion when there's a hierarchy.

Thanks,

- a_m

[Peter Mattis]

Is the roachprod cluster automatically wiped in between siblings? I thought it wasn't and that various subtests do that manually. Perhaps I'm mistaken.

One item to be aware of is the special setup that the jepsen tests do. That setup is time consuming and the subtests want to share an already set up cluster. I had thoughts on adding some sort of tag to clusters to indicate that some initialization had been performed, but never got around to doing it as the existing solution for the jepsen tests was sufficient. If you're revisiting how subtests work you might need to do something in this area. We definitely don't want every jepsen subtest running `initJepsen` from scratch.

[Andrei Matei]

On Fri, Oct 12, 2018, 10:54 AM Peter Mattis <pe...@cockroachlabs.com> wrote:

Is the roachprod cluster automatically wiped in between siblings? I thought it wasn't and that various subtests do that manually. Perhaps I'm mistaken.

Today subtests do the wiping manually I think, but I would like to make that automatic, between what today are sibling subtests and between root tests. That would be a good thing, wouldn't it?

One item to be aware of is the special setup that the jepsen tests do. That setup is time consuming and the subtests want to share an already set up cluster. I had thoughts on adding some sort of tag to clusters to indicate that some initialization had been performed, but never got around to doing it as the existing solution for the jepsen tests was sufficient. If you're revisiting how subtests work you might need to do something in this area. We definitely don't want every jepsen subtest running `initJepsen` from scratch.

Yes, I'm planning on dealing with Jepsen by introducing a cluster tagging mechanism, applied by a test once it performed things like initJepsen(). Separately, tests would use the same tag in their cluster spec. One way or another, these features in concert would make the scheduler give big preference to running sequences of similarly tagged tests on the same cluster.

This would all be orthogonal to the wiping; clusters would still be `roachprod wiped` between tests, but that doesn't destroy the initJepsen() stuff (which is package installs and such).

[Peter Mattis]

On Fri, Oct 12, 2018 at 11:44 AM Andrei Matei <and...@cockroachlabs.com> wrote:

On Fri, Oct 12, 2018, 10:54 AM Peter Mattis <pe...@cockroachlabs.com> wrote:

Is the roachprod cluster automatically wiped in between siblings? I thought it wasn't and that various subtests do that manually. Perhaps I'm mistaken.

Today subtests do the wiping manually I think, but I would like to make that automatic, between what today are sibling subtests and between root tests. That would be a good thing, wouldn't it?

Yes. I was just a bit confused by your email regarding what you are planning to do vs the current state. Wiping a cluster between subtests is a good thing.

 

One item to be aware of is the special setup that the jepsen tests do. That setup is time consuming and the subtests want to share an already set up cluster. I had thoughts on adding some sort of tag to clusters to indicate that some initialization had been performed, but never got around to doing it as the existing solution for the jepsen tests was sufficient. If you're revisiting how subtests work you might need to do something in this area. We definitely don't want every jepsen subtest running `initJepsen` from scratch.

Yes, I'm planning on dealing with Jepsen by introducing a cluster tagging mechanism, applied by a test once it performed things like initJepsen(). Separately, tests would use the same tag in their cluster spec. One way or another, these features in concert would make the scheduler give big preference to running sequences of similarly tagged tests on the same cluster.

This would all be orthogonal to the wiping; clusters would still be `roachprod wiped` between tests, but that doesn't destroy the initJepsen() stuff (which is package installs and such).

Sounds good to me.

[Tobias Schottdorf]

FYI, the synctest roachtest should also not have the cluster passed to a subsequent roachtest since it sets up some random filesystem stuff that `wipe` can't reset.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/CANKgOKh74mRrZ032Yf7Ye1fo%3DBTEd-1A0eo6aJgUuJvwOJT7kQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

-- Tobias

[Andrei Matei]

Good to know; I was wondering if there are cases like this and thought I'd have to find out the hard way. If anybody knows of more, please let me know.
I think we (I) should make it the test's responsibility to destroy a cluster or otherwise mark it as filthy in these cases.

[Andrei Matei]

I am confused by the "also" part in "synctest roachtest should also not have the cluster passed", though. To the best of my knowledge, a Jepsen cluster can be reused by other tests; the packages it installs and the other special setup is benign, I think.

[Tobias Schottdorf]

Jepsen messes with lots of system internals like the clocks or in some future also the file system. It seems to work fine (the clusters are already reused between Jepsen tests) but mixing them with non-Jepsen tests seems like a bad idea. For example, if the last Jepsen test crashes while having messed with the clocks, it could leave the next roachtest in an awkward place.

--

-- Tobias

[Andrei Matei]

+ Ben

Hmmm so then what do you reckon would be the right abstraction here?
It seems to me that either we want Jepsen clusters to be reused by both Jepsen and non-Jepsen tests alike, or not reused by anybody. Does it make sense to say that they can be reused only by other "Jepsen tests"?
Thus, I think the Jepsen tests should be under contract to restore the things that they mucked with and could affect other tests back to sane values, and, if they fail to do so, destroy the cluster (the test runner can help with implementing this contract, by example by allowing testsreturn an error to the test runner which says that a cluster cannot be reused and so it must be destroyed).

Alternatively, I guess we could also imagine some machinery that allows a cluster to be reused by some tests but not others if it really makes sense, but that would probably make a scheduler uglier.

[Tobias Schottdorf]

ISTM that the right abstraction is not to reuse resources across tests. If our primary motivation for that is cutting down on the runtime of tests (is there another one?), another approach would be to decouple provisioning and destroying of resources from the tests (i.e. prefetch the resources for the tests and destroy in the background as they get released without blocking the next test). Or we could go a cheap route and just move the destroy part into a background goroutine. Creating appears to be much faster than destroying in my experience.

--

-- Tobias

[Peter Mattis]

Part of the rationale for the scheduler is so that we can better utilize our cloud quota. If we destroy asynchronously in the background it will be hard for the scheduler to utilize all of the quota.

FYI, I noticed today that destroying clusters on AWS is really fast.

~ time roachprod create -c aws -n 1 peter-test

...

real 0m36.694s

user 0m6.361s

sys 0m1.390s

~ roachprod destroy peter-test

...

real 0m3.954s

user 0m1.961s

sys 0m0.363s

Compare this to GCE:

~ time roachprod create -c gce -n 1 peter-test

...

real 0m49.709s

user 0m5.965s

sys 0m1.331s

~ time roachprod destroy peter-test

...

real 1m29.029s

user 0m2.097s

sys 0m0.409s

Sort of crazy that GCE takes twice as long to destroy a cluster as it does to create it.

[Andrei Matei]

Well but Jepsen tests for example really do want to reuse clusters because their "creation" does take a long time. And even regular `roachprod create` takes one minute. Compound over many tests and it adds up.
I'm also trying to move to a model where you tell roachtest how much GCP quota it's allowed to use, and then it runs everything as fast as possible while staying under that quota. In this model, cluster destruction would be just as blocking for tests as cluster creation.

[Ben Darnell]

On Fri, Oct 12, 2018 at 2:46 PM Andrei Matei <and...@cockroachlabs.com> wrote:

+ Ben

Hmmm so then what do you reckon would be the right abstraction here?
It seems to me that either we want Jepsen clusters to be reused by both Jepsen and non-Jepsen tests alike, or not reused by anybody. Does it make sense to say that they can be reused only by other "Jepsen tests"?

Yes, I think the policy we want is that jepsen clusters can only be reused by other jepsen tests. Generally, I'd give tests a "cluster reuse tag" and only reuse clusters across tests with the same tag. (I'd probably also destroy the clusters of any failing tests to start clean on the next one). Be conservative about cluster reuse.

Another alternative would be to set up pre-baked machine images with the time-consuming parts of the setup already done to them. 

-Ben

[Tobias Schottdorf]

>  Another alternative would be to set up pre-baked machine images with the time-consuming parts of the setup already done to them.

and we might have to do that anyway because the test flakiness related to provisioning is only going to go up as we add more testing.

--

-- Tobias

[Tobias Schottdorf]

(though that's a real infra problem to solve since we're going to want to target tests at multiple cloud providers)

--

-- Tobias

[Andrei Matei]

It seems to me that either we want Jepsen clusters to be reused by both Jepsen and non-Jepsen tests alike, or not reused by anybody. Does it make sense to say that they can be reused only by other "Jepsen tests"?

Yes, I think the policy we want is that jepsen clusters can only be reused by other jepsen tests. Generally, I'd give tests a "cluster reuse tag" and only reuse clusters across tests with the same tag. 

But why? (My message was implying that I don't really see how this policy makes sense). Are Jepsen tests generally somehow more resilient to the kind of mucking that some of them do to the machines than other, more naive, tests?

> (I'd probably also destroy the clusters of any failing tests to start clean on the next one). Be conservative about cluster reuse.

I'd rather be more liberal :). You can imagine that a change causes a lot of tests to fail. We don't want to do away with all the benefits of cluster reuse for these runs.
I believe the vast majority of tests are fine with cluster reuse after a wipe. The wipe destroys everything cockroach-related. Only "special" tests do things outside of cockroach.
No?


Perhaps a middle way would be to add a dirty tag to some select tests (including all the Jepsen ones) which means that the cluster is to be destroyed if the cluster fails. And so the test itself would be free of any contractual burden to destroy or reset anything (given that it fails); the burden would all fall on the higher-level test runner.
Would everybody be on board with this?

[Tobias Schottdorf]

> I'd rather be more liberal :)

100% what Ben said. We shouldn't reuse on failure, these bugs are going to be horrible. This is starting to get uncomfortably close to this messy world in which you try to use Puppet to enforce a certain state and I thought the world had agreed that that was just a bad idea. I'm getting more and more convinced that we should steer clear of reuse and focus on efficient provisioning/destroying instead. Yes, we might not utilize our quota 100% but it'll get close enough.

--

-- Tobias

[Andrei Matei]

If we shouldn't reuse on failure, then should we reuse on success? If yes, how come?

[Tobias Schottdorf]

Maybe I wasn't clear above: The more I think about it, the more I'd rather not reuse at all.

--

-- Tobias

[Peter Mattis]

Do you mind the cluster reuse for subtests? If we create/destroy clusters for every subtest, then the acceptance test time will increase by ~30m (from ~13m to ~45m). I'm pretty sure nobody is going to like that.

[Andrei Matei]

You don't make anything of the fact that `roachprod wipe` (which is to be done between all tests) is already a very powerful tool?
The difference from the Puppet reference is that I think the vast majority of our tests play in a nice sandbox that's easy to... wipe.

[Peter Mattis]

My take is that most tests don't do anything funky with the cluster and thus reusing clusters between tests is fine. There are a handful of tests such as clearrange, jepsen, and synctest which muck with the nodes in a cluster (i.e. changing the filesystem to zfs or charybdefs, tweaking ntp, etc). Tagging the cluster-spec for those tests so that the clusters cannot be reused doesn't seem particularly onerous or error prone. 

[Ben Darnell]

On Fri, Oct 12, 2018 at 3:28 PM Andrei Matei <and...@cockroachlabs.com> wrote:

It seems to me that either we want Jepsen clusters to be reused by both Jepsen and non-Jepsen tests alike, or not reused by anybody. Does it make sense to say that they can be reused only by other "Jepsen tests"?

Yes, I think the policy we want is that jepsen clusters can only be reused by other jepsen tests. Generally, I'd give tests a "cluster reuse tag" and only reuse clusters across tests with the same tag. 

But why? (My message was implying that I don't really see how this policy makes sense). Are Jepsen tests generally somehow more resilient to the kind of mucking that some of them do to the machines than other, more naive, tests?

Jepsen tests are more resilient to the kind of mucking that other jepsen tests do, but they're no more resilient in general. Jepsen is mostly idempotent. For example, every jepsen test at startup uses ntp to synchronize the clocks once and then turns ntpd off. They do enough cleanup to leave things in a state where another jepsen test can run, but they don't try to reverse everything they do (or even enumerate everything so it could be reversed). 

 

> (I'd probably also destroy the clusters of any failing tests to start clean on the next one). Be conservative about cluster reuse.

I'd rather be more liberal :). You can imagine that a change causes a lot of tests to fail. We don't want to do away with all the benefits of cluster reuse for these runs.

If something causes a lot of tests to fail, I'd probably rather just cut everything off than go through the process of running every test while everything's broken. But that seems like a tricky heuristic and I'm OK with reusing clusters after failed tests in most cases (we have occasionally seen jepsen tests have cascading failures because of this, but it's rare). 

 

I believe the vast majority of tests are fine with cluster reuse after a wipe. The wipe destroys everything cockroach-related. Only "special" tests do things outside of cockroach.

Yes, I think most tests would be fine with cluster reuse, so they could get a default tag. Just don't mix things across the different categories of clusters. 

-Ben

[Tobias Schottdorf]

As a counterpoint, we already have lots of test that reformat the file systems (not often for good reason) as well as remounting with nobarrier or not and this is essential to some tests. I really think it's a bad idea to throw all of it into a pot. Yes, there are some tests that are fine to be grouped. But I think that already today you'd get lots of funky behavior if you just reused the same cluster for all tests. So if there's reuse, the reuse should be opt-in and perhaps limited to across subtests. Also, please consider that as we add lots of tests that *can't* be grouped (think: 20 performance tests), the situation will deteriorate roughly to where it is today, whereas if we streamlined the provisioning, it would actually get better.

--

-- Tobias

[Andrei Matei]

Tobi, please tell me more about these performance tests. Depending on what they do exactly, I suspect they wouldn't be good candidates for a roachtest run that runs many tests with --parallelism because the tests would compete for local CPU. Is running with --parallelism=1 what you had in mind when you say that "streamlined provisioning" would help? If so, then I guess we could do your suggestion of creating and destroying clusters concurrently when the --parallelism specified is not enough to saturate the given GCP quota. It could be a cool feature. But it's not what I want to focus on at the moment.

What I want to focus on for now are the use cases where one wants to run many tests in parallel (e.g. CI) or stress a couple of tests, and do the test running as fast as possible subject to a quota constraint.

For this, I think maybe we're converging on something. How about the following technicalities:

- The notion of "subtests" goes away. Subtests used to be about cluster reuse, now we're getting smarter and more flexible about that.

- Each test will specify a cluster reuse policy. This refers to who can reuse a cluster after it, not what cluster it itself can run on. The possible policies are:

    - "no-reuse" - The default (default for safety, but also very rarely used). Nobody can reuse the cluster after me. It doesn't matter if I succeed or fail. Not even another run of myself can reuse this cluster.

    - "only-tagged<foo>" - Only tests that have this policy with tag "foo" can reuse the cluster. It doesn't matter if the test succeeds or fails. Jepsen tests would all share a tag here. The scheduler would be smart enough to generally prefer running another "foo" (e.g. another run of the test that just ran if --count is specified) if there's anything left to run, rather than destroying the cluster.
    - "any" - Any other test can follow me on this cluster (after a wipe is performed, which always is regardless of this policy). So, a "no-reuse" or a "only-tagged" test can follow too, assuming the cluster spec matches (this goes against Ben's "no mixing of categories" opinion, but I think that what he intended was just prohibiting reuse in the other direction: no regular test after a Jepsen test).

   

Note that a test's success or failure never matters w.r.t. cluster reuse.

All tests except the Jepsens, clearrange and synctest will specify the "any" policy (let me know if you know of more funkies).
Jepsen tests will specify "only-tagged<jepsen>".
I don't know what clearrange and syntest should specify - "only-tagged<clear/sync>" or "no-reuse". Please tell me.

What do y'all think?

[Tobias Schottdorf]

`--parallelism` affects only the test runner, right? Real perf tests wouldn't want to use the test runner's CPU for anything. They start workloads on a remote node. (Maybe I'm misunderstanding your point here).

My point of mentioning performance tests is that they want to control their environment which typically includes the file system, whether it's local ssd or not, whether the device is mounted nobarrier or not, etc. But I may not have a big point here because lots of that needs to be specified when creating the node. Filesystems etc aren't (but perhaps that will change), but the tests can assert that they're in the right environment (and we're likely going to have N workloads to be tested in each environment, so reuse makes sense once you've set up the right one).

You're right that with `--parallelism=10` the streamlining goes away to some extent. But now I'm not so sure that cluster reuse is really that pressing a problem if the provisioning time is what counts, can't --parallelism=auto just run the tests as fast as it can? Sure, it's a little less than optimal, but how much less? Or are we re-doing various apt-gets and disk reformats many times if we don't keep the hardware around?

> (let me know if you know of more funkies).

Anything that reformats (search for zfs) or calls RemountNoBarrier which notably includes TPCC.

Maybe some of my aversion to this is that roachtests do things that should really be baked into the cluster they're getting passed. For example, in my `synctest` roachtest, I call `roachprod install charybdefs`. The test should just get a cluster that has that set. Similar for file systems and mount options, this will also make it more straightforward to support multiple providers.

Anyway, my stance has softened somewhat. Your labels sound OK.

--

-- Tobias

[Andrei Matei]

`--parallelism` affects only the test runner, right? Real perf tests wouldn't want to use the test runner's CPU for anything. They start workloads on a remote node. (Maybe I'm misunderstanding your point here).

Throughout this thread, I'm talking about the `roachprod run` command. I now guess you're talking about `roachprod bench tpcc`, where (at least for the tpcc bench, I see we ssh elsewhere and run `workload` there). `roachprod bench` uses the same infra (and I'm not planning on changing that), but I don't know to what extent cluster can be reused there since they presumably have different cluster specs; more below.
But now I'm again confused about your "prefetching of resources" idea. What kind of `roachprod bench` invocation would benefit from it? No need to respond if this has become irrelevant to the current thread.

 

My point of mentioning performance tests is that they want to control their environment which typically includes the file system, whether it's local ssd or not, whether the device is mounted nobarrier or not, etc. But I may not have a big point here because lots of that needs to be specified when creating the node. Filesystems etc aren't (but perhaps that will change), but the tests can assert that they're in the right environment (and we're likely going to have N workloads to be tested in each environment, so reuse makes sense once you've set up the right one).

A test has a "cluster spec". I think stuff like the filesystem prolly belongs there. Of course, there's no reuse between things with different specs. I'm planning to use some sort of DeepEquals for comparing two specs.

 

You're right that with `--parallelism=10` the streamlining goes away to some extent. But now I'm not so sure that cluster reuse is really that pressing a problem if the provisioning time is what counts, can't --parallelism=auto just run the tests as fast as it can? Sure, it's a little less than optimal, but how much less? Or are we re-doing various apt-gets and disk reformats many times if we don't keep the hardware around?

For the Jepsen tests in particular, yes, there's a setup with apt-gets and such which takes like 7 min I think.
For others, we're just talking about saving the `roachprod create` and `roachprod destroy`.
 

> (let me know if you know of more funkies).

Anything that reformats (search for zfs) or calls RemountNoBarrier which notably includes TPCC.

Ack.
 

👍

 

[Tobias Schottdorf]

I was talking about roachtest run (which is I think also what you're talking about). Maybe some benchmarks use the test runner for the workload, but that seems wrong. Could you link to one?

I know next to nothing about `roachprod bench tpcc` and perhaps roachtest isn't even going to be used to get reliable performance results, but so far it has looked to me that it is. Peter or Nathan would know whether there is anything you should be aware of here.

--

-- Tobias

[Andrei Matei]

I've got a question about the use of RemountNoBarrier() method. It seems to have proliferated some and is used by some tests that otherwise don't do sketchy things, so I'm considering having some of these tests share a only-tagged<remount-no-barrier> cluster reuse policy, so that clusters can be reused among these tests.
But my question is - would there be anything wrong with making all our tests call RemountNoBarrier() (or, rather, making roachprod give you a no-barrier cluster in the first place when used by roachtest)? If I understand correctly, this barrier is used by journaling file systems (or perhaps the way we use the option, only by ext4) to ensure safety in case of an unclean machine restart. Our tests don't seem to ever do machine restarts as far as I can see (e.g. the chaos module only shuts down crdb processes). So, should we make nobarrier the default for all our tests so that tests that currently use this option don't need to be special any more?

[Tobias Schottdorf]

The reason tests set varying nobarriers is because it has a huge impact on sync performance. My understanding is that `-o nobarrier` tells the kernel that the device has power loss protection and so it needs to sync "less" (i.e., the SSD isn't going to lose what you told it to write even if power cuts out). Tests behave very differently depending on whether nobarrier is set and we always set it for our performance testing (unless we're explicitly testing how bad the absence of nobarrier is). I think whatever you do in this department shouldn't change the existing tests.

If the cluster config specified the filesystem configuration (which seems like the right thing to do to me) then you could have a ext4-nobarrier and an ext4-barrier version (and zfs, and xfs, ...). I don't think this shoehorns well into your labels.

--

You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/CAPqkKgmL1mHbLnLsyGGGd0QSUTY3YUwf0rPc1zMzRGPC6hHYZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

-- Tobias

[Andrei Matei]

I understand why some tests want the no-barrier option (to get faster fsyncs); what I'm not sure of is if there'd be a downside to enabling it for all tests, or at least by default. What's the argument against doing that? Just that we want more realistic performance characteristics (ironically, for the "non-performance" tests)? 

[Tobias Schottdorf]

I'm not opposed to making it the default, but tests should be able to chose which one they get. And I don't think that you should change what each test gets today (at least not with your refactor).

--

-- Tobias

[Andrei Matei]

Yeah, agreed. FWIW, I also agree that both the no-barrier option and the filesystem type belong in the cluster config.