waitress: intermittent failures with python 3.2

[Lorenzo Gil Sanchez]

Hi everybody,

sorry if this is not the proper place to report waitress problems. I
didn't find a better place but I'll happy to send it somewhere else if
that is more appropiate.

I'm finding intermittent failures when running the waitress test suite
with Python 3.2. The tests located in
waitress/tests/test_functional.py that serve the file
waitress/tests/fixtureapps/groundhog1.jpg occasionally fail. I have
modified the tests suite so when they fail they copy the served file
into a directory under tmp. With this modification I can see the
problem is that the image is not completely served. The tests fail in
this assertion: self.assertEqual(cl, len(response_body)). The served
file's size is smaller than the Content-Length header in the response.

With python 2.6 and python 2.7 this doesn't happen.

If I run this command:

for i in `seq 100`; do tox -e py32 ; done

I got 57 failures on a specific run. If I run it again I got similar
failures but not always the same.

I modified the SubprocessTests.stop_subprocess() method and added a
time.sleep(1) call at the end with the hope to decrease the number of
failures but it didn't help at all.

Can anyone confirm this problem? I'd like to help fixing the bug but
I'm afraid that I'm not very familiar with waitress low level code so
any pointer will be appreciated.

Note: my goal is to package waitress for Fedora/EPEL.

Best regards,

Lorenzo Gil

[Marius Gedminas]

On Sun, Oct 21, 2012 at 07:55:12PM +0200, Lorenzo Gil Sanchez wrote:
> I'm finding intermittent failures when running the waitress test suite
> with Python 3.2.

...

> If I run this command:
>
> for i in `seq 100`; do tox -e py32 ; done
>
> I got 57 failures on a specific run. If I run it again I got similar
> failures but not always the same.

...

> Can anyone confirm this problem? I'd like to help fixing the bug but
> I'm afraid that I'm not very familiar with waitress low level code so
> any pointer will be appreciated.

I was unable to reproduce this on Ubuntu 12.04 (x86-64) with 10 runs of
tox -e py32, but I *was* able to reproduce it when I tried a 100 runs
after that.

One of the failures I saw was

FAIL: test_notfilelike_longcl_http11 (waitress.tests.test_functional.TestFileWrapper)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mg/src/waitress/waitress/tests/test_functional.py", line 1114, in test_notfilelike_longcl_http11
self.assertEqual(cl, len(response_body)+10)
AssertionError: 45458 != 16271

so it looks like the same thing you saw.

In total I've seen 8 failures out of 110 runs.

Marius Gedminas
--
After having done some test using hi-tech istruments (moving my mouse
during a kernel build) [...]
-- Davide Libenzi on lkml

[Lorenzo Gil Sanchez]

2012/10/23 Marius Gedminas <mar...@gedmin.as>:

> On Sun, Oct 21, 2012 at 07:55:12PM +0200, Lorenzo Gil Sanchez wrote:
>> I'm finding intermittent failures when running the waitress test suite
>> with Python 3.2.
> ...
>> If I run this command:
>>
>> for i in `seq 100`; do tox -e py32 ; done
>>
>> I got 57 failures on a specific run. If I run it again I got similar
>> failures but not always the same.
> ...
>> Can anyone confirm this problem? I'd like to help fixing the bug but
>> I'm afraid that I'm not very familiar with waitress low level code so
>> any pointer will be appreciated.
>
> I was unable to reproduce this on Ubuntu 12.04 (x86-64) with 10 runs of
> tox -e py32, but I *was* able to reproduce it when I tried a 100 runs
> after that.
>
> One of the failures I saw was
>
> FAIL: test_notfilelike_longcl_http11 (waitress.tests.test_functional.TestFileWrapper)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/mg/src/waitress/waitress/tests/test_functional.py", line 1114, in test_notfilelike_longcl_http11
> self.assertEqual(cl, len(response_body)+10)
> AssertionError: 45458 != 16271
>
> so it looks like the same thing you saw.
>

Yes, it's exactly the same failure.

> In total I've seen 8 failures out of 110 runs.
>

I must say, the machine I'm using is quite old and slow (5 years).
Maybe that helps to reproduce the bug.

[Damien Baty]

Hello,

Le 21/10/2012 19:55, Lorenzo Gil Sanchez a �crit :

>
> I'm finding intermittent failures when running the waitress test
> suite with Python 3.2.

> [...]

> The tests fail in this assertion: self.assertEqual(cl,
> len(response_body)). The served file's size is smaller than the
> Content-Length header in the response.
>

> Can anyone confirm this problem?

I did reproduce the bug you mention. On each test run, at least one of
the tests in "test_function.TestFileWrapper" failed (although it was not
always the same ones).

The problem is that the test reads from the socket before all data has
been sent. It happens to me only on tests in TestFileWrapper, which
sends a 48 Kb file. I guess that this is more prone to happen on
low-end, busy machines. I opened a pull request with a simple fix
(https://github.com/Pylons/waitress/pull/2). Perhaps you could try
reproducing the bug with my branch and see if it fixes it for you too?

Note: this is is a "bug" in the tests, _not_ in the implementation of
Waitress.

Cheers,

--
Damien Baty

[Lorenzo Gil Sanchez]

2012/10/25 Damien Baty <dam...@damienbaty.com>:
> Hello,

>
> Le 21/10/2012 19:55, Lorenzo Gil Sanchez a écrit :
>>
>> I'm finding intermittent failures when running the waitress test
>> suite with Python 3.2.
>> [...]
>> The tests fail in this assertion: self.assertEqual(cl,
>> len(response_body)). The served file's size is smaller than the
>> Content-Length header in the response.
>>
>> Can anyone confirm this problem?
>
> I did reproduce the bug you mention. On each test run, at least one of
> the tests in "test_function.TestFileWrapper" failed (although it was not
> always the same ones).
>
> The problem is that the test reads from the socket before all data has
> been sent. It happens to me only on tests in TestFileWrapper, which
> sends a 48 Kb file. I guess that this is more prone to happen on
> low-end, busy machines. I opened a pull request with a simple fix
> (https://github.com/Pylons/waitress/pull/2). Perhaps you could try
> reproducing the bug with my branch and see if it fixes it for you too?

Sure! I'll do that as soon as I get my hands into my home's computer
which is the slow one where I did the tests. That should be this
afternoon (CEST tz).

>
> Note: this is is a "bug" in the tests, _not_ in the implementation of
> Waitress.
>

Good to know.


> Cheers,
>
> --
> Damien Baty
>
> --
> You received this message because you are subscribed to the Google Groups "pylons-devel" group.
> To post to this group, send email to pylons...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-devel...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
>

[Lorenzo Gil Sanchez]

2012/10/25 Damien Baty <dam...@damienbaty.com>:
> Hello,

>
> Le 21/10/2012 19:55, Lorenzo Gil Sanchez a écrit :
>>
>> I'm finding intermittent failures when running the waitress test
>> suite with Python 3.2.
>> [...]
>> The tests fail in this assertion: self.assertEqual(cl,
>> len(response_body)). The served file's size is smaller than the
>> Content-Length header in the response.
>>
>> Can anyone confirm this problem?
>
> I did reproduce the bug you mention. On each test run, at least one of
> the tests in "test_function.TestFileWrapper" failed (although it was not
> always the same ones).
>
> The problem is that the test reads from the socket before all data has
> been sent. It happens to me only on tests in TestFileWrapper, which
> sends a 48 Kb file. I guess that this is more prone to happen on
> low-end, busy machines. I opened a pull request with a simple fix
> (https://github.com/Pylons/waitress/pull/2). Perhaps you could try
> reproducing the bug with my branch and see if it fixes it for you too?

I have tested your fixes (in the master branch since Chris already
merged them) and it works perfectly!

Thanks a lot for fixing this tests' bug.