----------------------------------------------------------------------
Ran 11906 tests in 171.474s
FAILED (errors=1, skipped=1152, expected failures=4)
}}}
This is from the 1.11 alpha1 tarball, rather than git itself.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
* severity: Normal => Release blocker
* cc: David Sanders (added)
* component: Uncategorized => contrib.staticfiles
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:1>
Comment (by David Sanders):
Going to need more information than that. What are the steps to reproduce?
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:2>
Comment (by Tim Graham):
Extracting the tarball and running the tests from there reproduced it for
me.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:3>
Comment (by David Sanders):
I can't reproduce with those instructions.
{{{
[root@fce6b18201fb tests]# ./runtests.py
"staticfiles_tests.test_storage.TestCollectionCachedStorage"
Testing against Django installed in '/tmp/django/Django-1.11a1/django'
with up to 2 processes
Creating test database for alias 'default'...
Creating test database for alias 'other'...
System check identified no issues (0 silenced).
...................
----------------------------------------------------------------------
Ran 19 tests in 0.802s
OK
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
}}}
{{{
root@fce6b18201fb tests]# ./runtests.py
"staticfiles_tests.test_storage.TestCollectionCachedStorage.test_corrupt_intermediate_files"
Testing against Django installed in '/tmp/django/Django-1.11a1/django'
with up to 2 processes
Creating test database for alias 'default'...
Creating test database for alias 'other'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.042s
OK
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
}}}
That's running with Python2.7 on a CentOS 7.3 Docker image.
Tim, are you reproducing by running all tests or just the targeted tests
cases like in my example output above?
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:4>
Comment (by Chris Lamb):
Replying to [comment:2 David Sanders]:
> Going to need more information than that. What are the steps to
reproduce?
Sorry about that, I should have been clearer when referring to the
tarball. I'm running the entire testsuite.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:5>
Comment (by David Sanders):
Replying to [comment:5 Chris Lamb]:
> Replying to [comment:2 David Sanders]:
> > Going to need more information than that. What are the steps to
reproduce?
>
> Sorry about that, I should have been clearer when referring to the
tarball. I'm running the entire testsuite.
Does the error still occur when you run only the single test or the group
of tests like in my example output above?
Entire testsuite takes my Macbook 20+ minutes to run at least, so I'd like
to narrow down the reproduce case if it's to be tracked down in a
reasonable amount of time.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:6>
Comment (by Chris Lamb):
> Does the error still occur when you run only the single test or the
group of tests like in my example output above?
I'm clearly failing at communicating today - my bad! It only appears when
I run the entire testsuite. If I just run the single test (or its parent
group), the test does not fail.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:7>
Comment (by David Sanders):
Replying to [comment:7 Chris Lamb]:
> > Does the error still occur when you run only the single test or the
group of tests like in my example output above?
>
> I'm clearly failing at communicating today - my bad! It only appears
when I run the entire testsuite. If I just run the single test (or its
parent group), the test does not fail.
Well, that's good info, since it seems to show there's something elsewhere
in the testsuite breaking it.
However, that's a real bummer considering how long it takes me to run the
entire testsuite, tracking it down is going to be a PITA. Perhaps I can
selectively delete test modules that are unlikely to be the interference
and that'll speed things up...
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:8>
Comment (by Simon Charette):
David, you might want to try the `--bisect` or `--pair` options of
`runtests.py` to speed up the process.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:9>
Comment (by David Sanders):
Replying to [comment:9 Simon Charette]:
> David, you might want to try the `--bisect` or `--pair` options of
`runtests.py` to speed up the process.
Thanks for the tip Simon, both of those were quite speedy compared to
running the full test suite.
Unfortunately neither revealed any problems. Ran them both 3 times to
ensure there wasn't anything inconsistent:
{{{
./runtests.py --pair
"staticfiles_tests.test_storage.TestCollectionCachedStorage.test_corrupt_intermediate_files"
./runtests.py --bisect
"staticfiles_tests.test_storage.TestCollectionCachedStorage.test_corrupt_intermediate_files"
}}}
I did, however, have to remove one group of tests because they were
failing for me and seem unrelated - is anyone able to do run
`./runtests.py "staticfiles_tests.test_views"` and have it succeed? For me
that always fails complaining about `AUTH_USER` problems. If run with
other tests that error doesn't occur, only when it is run alone.
Running the entire testsuite for me hangs and I've killed it after 40
minutes without it reproducing the original error.
Chris, could you try running `./runtests.py
"staticfiles_tests.test_views"` and responding with your results? If that
run causes an error, can you try deleting the `test_views.py` file under
`staticfiles_tests` and then run the entire testsuite again and see if the
failure still reproduces?
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:10>
Comment (by Chris Lamb):
Replying to [comment:10 David Sanders]:
> Chris, could you try running `./runtests.py
"staticfiles_tests.test_views"` and responding with your results?
Fails with
https://gist.github.com/lamby/b424c3310fade40245f046b712913f44/raw
> If that run causes an error, can you try deleting the `test_views.py`
file under `staticfiles_tests` and then run the entire testsuite again and
see if the failure still reproduces?
Fails with
https://gist.githubusercontent.com/lamby/647a790e935ec11c25221e69bc951263/raw
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:11>
Comment (by David Sanders):
Replying to [comment:11 Chris Lamb]:
> Replying to [comment:10 David Sanders]:
> > Chris, could you try running `./runtests.py
"staticfiles_tests.test_views"` and responding with your results?
>
> Fails with
https://gist.github.com/lamby/b424c3310fade40245f046b712913f44/raw
>
> > If that run causes an error, can you try deleting the `test_views.py`
file under `staticfiles_tests` and then run the entire testsuite again and
see if the failure still reproduces?
>
> Fails with
https://gist.githubusercontent.com/lamby/647a790e935ec11c25221e69bc951263/raw
Thanks! So it looks like `staticfiles_tests.test_views` can't be run by
itself for anyone, that probably should be fixed in general since each
test should be able to be run in isolation and not fail.
For the second case it looks like your setup didn't appreciate deleting
that test for some reason, not sure why. Need to find a way to exclude
that test to see if it's the one causing problems.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:12>
Comment (by Chris Lamb):
Any progress? Would love to see this alpha version in Debian
experimental...
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:13>
Comment (by David Sanders):
Replying to [comment:13 Chris Lamb]:
> Any progress? Would love to see this alpha version in Debian
experimental...
Sorry Chris, I haven't been working on it. I've got too much else on my
plate and no easy way to reproduce this (for some reason running the full
testsuite from the tarball hangs for me).
We do know that the test in isolation succeeds, so it's some interaction
with other tests, however both `--bisect` and `--pair` don't yield any
information.
Can any Django test gurus take a look to try to find what the test
interaction might be?
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:14>
Comment (by Chris Lamb):
If it helps, if I limit parallelism with "taskset --cpu-list 0" the
testsuite does not fail.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:15>
Comment (by Tim Graham):
I have a [https://github.com/django/django/pull/7942 PR] to fix the
`test_views` failure when run in isolation.
I can still reproduce a non-deterministic failure of
`test_corrupt_intermediate_files` when running only the staticfiles tests
(actually only `staticfiles_tests.test_storage`). I use a loop to repeat
the tests until it fails:
{{{
for i in {1..500};
do
python tests/runtests.py staticfiles_tests.test_storage
retcode=$?
if [ $retcode = 1 ]; then
echo $i;
echo 'failed';
break;
fi
done
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:16>
Comment (by David Sanders):
Thanks Tim, that script helps me quickly reproduce on my setup.
I've further narrowed it and confirmed what Chris said about parallelism
being a factor. If you use `--parallel 1` to disable parallelism it won't
reproduce.
I've narrowed it down to the parallel interaction between two test cases.
It makes sense, the latter is also dealing with `window.png` and changes
the modification timestamp on it which changes the hash.
{{{
for i in {1..500};
do
python tests/runtests.py
staticfiles_tests.test_storage.TestCollectionCachedStorage
staticfiles_tests.test_storage.TestCollectionHashedFilesCache
retcode=$?
if [ $retcode = 1 ]; then
echo $i;
echo 'failed';
break;
fi
done
}}}
That's all the digging I've got time for tonight. I'll start to look into
a fix tomorrow, unless someone wants to pounce on it before me now that
it's narrowed down.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:17>
Comment (by Claude Paroz):
To isolate the test, you can get inspiration from
`tests.i18n.utils.RunInTmpDirMixin`, that is create a temporary directory
to copy the static tree in, override the STATICFILES_DIRS setting to
include that temp dir, then remove the dir in the test tearDown.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:18>
Comment (by GitHub <noreply@…>):
In [changeset:"11856ea44e4dd1ff2358c559c0d04ac848832c23" 11856ea4]:
{{{
#!CommitTicketReference repository=""
revision="11856ea44e4dd1ff2358c559c0d04ac848832c23"
Refs #27741 -- Fixed staticfiles_tests.test_views when run in isolation.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:19>
Comment (by Tim Graham <timograham@…>):
In [changeset:"7dda48bf18454b34e8e0a305dbfda08ecfe8db25" 7dda48b]:
{{{
#!CommitTicketReference repository=""
revision="7dda48bf18454b34e8e0a305dbfda08ecfe8db25"
[1.11.x] Refs #27741 -- Fixed staticfiles_tests.test_views when run in
isolation.
Backport of 11856ea44e4dd1ff2358c559c0d04ac848832c23 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:20>
Comment (by Tim Graham):
I'm feeling a bit lazy about isolating the tests using the approach Claude
suggested. How about this patch, at least for now:
{{{ #!diff
diff --git a/tests/staticfiles_tests/test_storage.py
b/tests/staticfiles_tests/test_storage.py
index 7e7b31f..11d1cf3 100644
--- a/tests/staticfiles_tests/test_storage.py
+++ b/tests/staticfiles_tests/test_storage.py
@@ -290,7 +290,7 @@ class TestCollectionCachedStorage(TestHashedFiles,
CollectionTestCase):
# high enough.
configured_storage.max_post_process_passes = 1
# File without intermediates that can be rehashed without a
problem.
- self.hashed_file_path('cached/css/img/window.png')
+ self.hashed_file_path('cached/css/fonts/font.svg')
# File with too many intermediates to rehash with the low max
# post-process passes.
err_msg = "The name 'cached/styles.css' could not be hashed with
%r." % (configured_storage._wrapped,)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:21>
Comment (by Chris Lamb):
Replying to [comment:21 Tim Graham]:
> I'm feeling a bit lazy about isolating the tests using the approach
Claude suggested. How about this patch, at least for now:
Can you elaborate why this would fix it? :) In isolation it seems a bit
arbitrary.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:22>
Comment (by Tim Graham):
From David's comment:17, "I've narrowed it down to the parallel
interaction between two test cases. It makes sense, the latter is also
dealing with window.png and changes the modification timestamp on it which
changes the hash." If the test uses some other file that's not modified in
another test, there's no issue as far as I've tested.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:23>
Comment (by David Sanders):
Replying to [comment:23 Tim Graham]:
> From David's comment:17, "I've narrowed it down to the parallel
interaction between two test cases. It makes sense, the latter is also
dealing with window.png and changes the modification timestamp on it which
changes the hash." If the test uses some other file that's not modified in
another test, there's no issue as far as I've tested.
I've actually got a patch for it isolated, Tim. Just finished it up the
other day, but I've been too swamped to get it cleaned up and in a PR.
Sorry about that.
Should I submit the PR for the stable/1.11.x branch and then do a separate
PR for master?
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:24>
Comment (by Tim Graham):
The PR can go to master and I'll backport from there.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:25>
Comment (by David Sanders):
Replying to [comment:25 Tim Graham]:
> The PR can go to master and I'll backport from there.
Ok, I've made it against master. My attempt to rebase after fixing a typo
didn't seem to trigger another build on Jenkins (as it has in the past) so
I had to do a second commit to trigger the clean build.
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:26>
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"8e3f9d3ee20fdadef6f2a28c71fb7e142b415059" 8e3f9d3e]:
{{{
#!CommitTicketReference repository=""
revision="8e3f9d3ee20fdadef6f2a28c71fb7e142b415059"
Fixed #27741 -- Isolated TestCollectionHashedFilesCache in a tmpdir.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:27>
Comment (by Tim Graham <timograham@…>):
In [changeset:"fe86a785a1079f646e7ad452f6fac08ec4d3378c" fe86a78]:
{{{
#!CommitTicketReference repository=""
revision="fe86a785a1079f646e7ad452f6fac08ec4d3378c"
[1.11.x] Fixed #27741 -- Isolated TestCollectionHashedFilesCache in a
tmpdir.
Backport of 8e3f9d3ee20fdadef6f2a28c71fb7e142b415059 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27741#comment:28>