Hi Matt,
I have just spent some time running tests on my own Eucalyptus/Walrus server which seem to confirm my suspicions that Walrus does not cope well with more than a few simultaneous connections. I'll discuss my testing for a bit, then offer a potential work around below.
I ran a number of tests uploading and deleting objects in a single thread. I tried uploading different object sizes, from just a few bytes to 8MB, and never saw any problems even uploading hundreds of objects.
As soon as I added multi-threading to the mix, my Walrus server responded with an Internal Server error or two and then hung completely. The only remedy for this was to restart the cloud and cc components.
In my tests I could produce hanging failures somewhat regularly with 10 simultaneous connections, with the chance of a hang decreasing when I used fewer threads. Also, when using multiple threads I noticed that odd things would happen even when the uploads didn't obviously fail. For example, Walrus' listing of objects in the database can get out of sync with the contents of the object file storage area (/var/lib/eucalyptus/bukkit).
I'm not sure exactly which version of Eucalyptus/Walrus I'm using (is there any way to find out the version number?) but my recommendation for now is to use only a single HTTP connection when interacting with Walrus.
You can configure the JetS3t library and applications to use a single HTTP connection with the following settings in jets3t.properties:
s3service.max-thread-count=1
s3service.admin-max-thread-count=1
httpclient.max-connections=1
Can you try these settings and confirm whether they work-around the issue?
James
---
http://www.jamesmurty.com