I decided to break down and compile php from source following Sammy Kaye Powers' videos. After doing this I realized that I could commit the docker image to docker hub, so that whoever didn't want to go through all the install steps could just download a ready to go container. Here's the page for the repo:
The gist of it (assuming you have docker installed on your machine) is below.
Of course this image only has php 7.3 on it - so phpqa is still useful for testing multiple versions without compiling them all - but I'm still not sure how / if you can get the .out files to inform your test writing.
Sorry for slamming your mailboxes - I hope you find this useful!
Clone Docker Image
1. docker login
2. docker pull otteford/madison-phptestfest-2017-rough
3. docker run otteford/madison-phptestfest-2017-rough
4. docker ps -a
-note/copy the container-id
5. In a separate terminal tab/window
docker exec -t -i CONTAINER-ID bash -l
Leave docker interactive shell: Ctrl-p + Ctrl-q
(I'm not sure this always works)
Run all tests:
sapi/cli/php run-tests.php -P
Run specific test:
sapi/cli/php run-tests.php -P tests/basic/001.phpt
make test TESTS=tests/basic/001.phpt
make test TESTS="tests/basic/001.phpt ext/standard/tests/random"