Test Errors
Bill Sumerlin
failure (i.e. the remote server did not return a result or generated
some sort of error), or both?
When I examined the results.csv file, I expected to find an error
count in the error column but instead discovered 10055 socket error
codes which indicates insufficient local system memory on socket I/
O... which could be from either/both transmission or reception buffer
overruns. At only 450 transactions/second, I did not expect
insufficient buffers with very small send/receive messages on a quad-
core windows based system.
I have not thoroughly looked (just googling on the net), but is there
an option to turn on graphing of error counts corresponding to
transmission results. I always find these valuable when attempting to
improve server stability at high request/transmission rates.
Bill Sumerlin
generation errors (e.g., 10055 socket error) are counted and treated
as the same. Furthermore, the results (response rates, etc.) are NOT
adjusted for the errors, so if/when errors start to occur, the
reported results are not accurate and in all likelihood inflated.
When running server stress tests, it is important to determine when
those errors start to occur, and whether the server properly recovers.
If a server does not recover, sooner or later that server could create
a problem when in production.
Corey Goldberg
not sure I understand. If your server is responding with errors
*really fast*, the results will show that rate.
I agree there needs to be a lot of work done in the results and
graphing. I will accept patches/pull-requests of code. Do you have
interest in collaborating around those areas? If not, then add
well-defined descriptions of features/fixes to the issue tracker and
I'll get to them eventually.
regards,
-Corey
Bill Sumerlin
factor out and graph the errors, and testing Multi-Mechanize in PC,
Mac OS X and Linux environments, I have the following observations:
1. If the server you are testing against is a high performance
server, Multi-Mechanize (and the Python Mechanize library working with
the local OS) will generate "local" errors, either due to running out
of "free" file handles, sockets and/or memory before the server you
are testing against has any problems. So, it is VERY IMPORTANT to
differentiate between a local system error and a remote server error.
I am going to have to dig through the Mechanize code to figure out how
to differentiate between the two types of errors. Currently, the
returned generic error code is not enough.
2. There is a noticeable reported server performance difference
depending on the system type you run Multi-Mechanize on. For example,
I've found that the latest IMac (OS X Lion) processes significantly
more transactions/tests faster than a Quad Core Windows 7 PC. I
suspect that the Linux system will be even faster. Why? I believe it
has to do with the OS and how quickly it can open, close sockets and
process inbound and outbound messages.
3. If the test is run long enough, the test client in the Mac
environment performs a partial recovery and starts communicating
successfully with the remote server. That implies that resource
constraints may be system related versus the Multi-Mechanize processes
themselves. The PC Windows 7 environment does not seem capable of
recovering during a single test.
4. Simultaneously running Multi-Mechanize against the same server
does NOT significantly change any of the single test client results. I
find that somewhat interesting. My challenge is to determine what the
maximum sustainable throughput for a production server is. At this
point, I can't even determine the number of Multi-Mechanize test
clients I will need to determine that. So, I've pulled in another
test tool (httperf), and automated it, only to find that it too
suffers the same local OS issues.
The bottom line is that it is essential to differentiate between local
client errors and remote server errors, and then to provide some
guidance on how to improve the local test environment to reduce the
local client errors.