Shuffle insert fixtures at ActiveRecord unit test

[Yasuo Honda]

Hi,

I'd like to propose shuffle insert fixtures at ActiveRecord unit tests

to find and address tests which depend on non-deterministic sort order easily.

* Background

Since [Unlock minitest for Rails' test suite](https://github.com/rails/rails/pull/29271)

has been merged to master, there are some fixes made to address non-deterministic sort order. 

https://github.com/rails/rails/pull/30228

https://github.com/rails/rails/pull/30152

https://github.com/rails/rails/pull/30133

All of these fixes found when ActiveRecord unit tests are executed with PostgreSQL database.

It is likely due to PostgreSQL performs append write, do not perform in place update.

Here, I do not mean which architecture is good or bad. I wanted to say it is an application responsibility to write SQL statement to guarantee sort order. 

Changing load fixture order would help to find and address tests depending on un-guaranteed sort order.

* Test

I have created a `shuffle_fixtures` branch and updated `insert_fixtures` method 

to shuffle fixtures and executed ActiveRecord unit tests with postgresql adapter.

- `shuffle_fixtures` branch

https://github.com/yahonda/rails/tree/shuffle_fixtures

- Commit to shuffle fixtures

https://github.com/yahonda/rails/commit/c41361aa35078101c0b840497fae92078c511ff1

* Test results with postgresql adapter 

"5654 runs, 15575 assertions, 13 failures, 0 errors, 2 skips"

https://gist.github.com/yahonda/7898ec69325dcb70c5e6efdc7f74d174

I'd like to hear from developers working on ActiveRecord development.

Thanks,

--

Yasuo Honda

[Matthew Draper]

> I have created a `shuffle_fixtures` branch and updated `insert_fixtures` method
> to shuffle fixtures and executed ActiveRecord unit tests with postgresql adapter.
>
> - `shuffle_fixtures` branch
> https://github.com/yahonda/rails/tree/shuffle_fixtures

Hmmm, interesting.


Introducing deliberate randomness seems like a good idea, to flush out accidental ordering assumptions in the tests — particularly our own.

But I’m not sure whether a full shuffle is going to focus on the right set of assumptions:

* It changes the order in which IDs are assigned to records, so .order(:id) can no longer be trusted as a simple “in fixture order” definition.

* It *doesn’t* change the order rows will be returned relative to their IDs: even after this, most of the time User.all.to_a will match User.order(:id).to_a.



I wonder if there’d be any merit to an “order torture” config setting, which adds an `ORDER BY random()`-type clause to every relation query…



Matthew

[Yasuo Honda]

Thanks for the reply.

> But I’m not sure whether a full shuffle is going to focus on the right set of assumptions: 

Agreed.  I feel "flush everything" is too aggressive for what I want. 

Since I just tested with all shuffled fixtures. I'd like to have some time and take a look at each failure

and take necessary actions for them.

Thanks,

--

Yasuo Honda