Question on files used by model_forms tests
Karen Tracey
IOError: [Errno 78] File name too long: u'/tmp/test2_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________.txt'
That's not, of course, the file name specified in the test (found in tests/modeltests/model_forms/models.py), where the file is merely specified as 'test2.txt'. I'm guessing the underscores are coming from the save_FOO_file behavior [2] of adding underscores when a file of the actually-specified name already exists. But why did a file of the actually-specified name already exist, plus one with one underscore, two underscores, etc, up to the point that the file name got too long? How did the machine get into this state?
The tests appear to be coded to delete the files they create. Was there ever a problem where the files weren't being deleted, which could have led to the buildbot getting into this state?
There seems to be one machine (flanagan-sparc) that is used for testing both the Python 2.5 branch and trunk (Python 2.6). Could simultaneous runs of the tests for different levels of Python have produced this situation?
Some weirdness that could give someone (who knows more about these tests than I) a hint: the errors show up for the files test2.txt, test3.txt, test2.png, and test3.png but NOT test1.txt or test.png. What's special about the ones that don't show the problem -- anything?
Any feedback/ideas on what led to this situation on the buildbot would be appreciated.
Karen
[1] http://www.python.org/dev/buildbot/community/all/
[2] http://www.djangoproject.com/documentation/db-api/#save-foo-file-filename-raw-contents
Marty Alchin
> The Python community buildbots [1] for Django are failing many model_forms
> tests with errors like this:
>
> IOError: [Errno 78] File name too long:
> u'/tmp/test2_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________.txt'
>
Yowza! We'd been warned[1] that this was possible, but I never
expected it to come up during tests. Thanks for the heads-up about
that buildbot, I had no idea there was one running anywhere but
buildbot.djangoproject.com.
> That's not, of course, the file name specified in the test (found in
> tests/modeltests/model_forms/models.py), where the file is merely specified
> as 'test2.txt'. I'm guessing the underscores are coming from the
> save_FOO_file behavior [2] of adding underscores when a file of the
> actually-specified name already exists. But why did a file of the
> actually-specified name already exist, plus one with one underscore, two
> underscores, etc, up to the point that the file name got too long? How did
> the machine get into this state?
I've run into that a few times on these tests while debugging my file
storage patch[2], but since that hasn't been committed yet, I know
it's nothing to do with that.
> The tests appear to be coded to delete the files they create. Was there
> ever a problem where the files weren't being deleted, which could have led
> to the buildbot getting into this state?
None that I know of, and I've been watching the file stuff pretty
closely lately.
> There seems to be one machine (flanagan-sparc) that is used for testing both
> the Python 2.5 branch and trunk (Python 2.6). Could simultaneous runs of
> the tests for different levels of Python have produced this situation?
Hrm. I can't say I've tried two simultaneous test runs, since it takes
15 minutes for me to run just one, but I'm not sure that's the
problem. Each test would know the actual filename that was used on the
filesystem, and it would be deleting that one, rather than getting
mixed up with another one.
Then again, there's also been some concern[3] about a race condition
between picking a name and writing to the file. I don't see how that
would cause this, though, since that would show the two tests writing
to the same file, and trying to delete the same file. It would cause
corruption in the file's contents, and probably IOErrors when trying
to delete, but I can't imagine it would actually leave the file lying
around to get spotted next time. Especially not reliably enough to go
through that many iterations. I suppose it's possible, but it's not my
first guess.
> Some weirdness that could give someone (who knows more about these tests
> than I) a hint: the errors show up for the files test2.txt, test3.txt,
> test2.png, and test3.png but NOT test1.txt or test.png. What's special
> about the ones that don't show the problem -- anything?
That's a useful tip, since it at least helps narrow things down a
little bit. I don't see anything particularly special about those two
cases, but I'll do some digging in the buildbot's history and see if I
can spot where things went so horribly wrong. I've spent a good deal
of time debugging those tests, and it's not fun, but I'll do what I
can.
-Gul
[1] http://code.djangoproject.com/ticket/5485#comment:3
[2] http://code.djangoproject.com/ticket/5361
[3] http://code.djangoproject.com/ticket/4948
Marty Alchin
unfortunately, is only two weeks worth), I don't see any clear cause
for the errors that are showing up. That same error has been happening
for as long as those buildbots have been running the full test suite,
which makes me thing they were running different buildbots before,
when the problem first occurred. Then, maybe a switch in buildbots, or
maybe just their names, started a new set of logs that only contain
the "too long" error.
I don't know who to talk to about the environment those buildbots are
running in, but they may well just need to "rm test*.txt" in their
temp directory and try the tests again. At least then we'd be able to
get a more useful error if the process starts all over again. It's
also quite possible that the original problem is no longer present,
and simply wiping out those files could be enough to get things back
on track.
Karen, do you happen to know who to relay this information to?
-Gul
Karen Tracey
Karen
Gary Wilson Jr.
> On Thu, Jul 31, 2008 at 5:23 PM, Karen Tracey <kmtr...@gmail.com> wrote:
>> There seems to be one machine (flanagan-sparc) that is used for testing both
>> the Python 2.5 branch and trunk (Python 2.6). Could simultaneous runs of
>> the tests for different levels of Python have produced this situation?
>
> Hrm. I can't say I've tried two simultaneous test runs, since it takes
> 15 minutes for me to run just one, but I'm not sure that's the
> problem. Each test would know the actual filename that was used on the
> filesystem, and it would be deleting that one, rather than getting
> mixed up with another one.
I can say that I've hit test failures for this when running more than
one test run at a time. Of course, the failures I was seeing were of
the nature "Expected test.png, got test_.png", and I do believe I had to
manually rm them afterwards. I didn't look any further into the issue
though, sorry.
Gary
Karen Tracey
Karen, do you happen to know who to relay this information to?
OK, I found & contacted the owner and he's cleaned out /tmp so we'll see how things look with the next couple of runs. Trunk is currently building....
Karen
Marty Alchin
> OK, I found & contacted the owner and he's cleaned out /tmp so we'll see how
> things look with the next couple of runs. Trunk is currently building....
And just like that, the latest trunk passes model_forms with flying
colors! Thanks for getting in touch and following up on this, Karen.
-Gul