Parallel uploads of results to S3

6 views
Skip to first unread message

shlomo....@gmail.com

unread,
Aug 13, 2008, 8:19:10 AM8/13/08
to lifegu...@googlegroups.com
Hi,

Currently, lifeguard's AbstractBaseService uploads the service's
MetaFile results to S3 serially.

Here is a small patch to allow parallel uploading of the service's
result MetaFiles to S3, using jets3t's
S3ServiceSimpleMulti.putObject(). The number of threads that jets3t
will use to upload is controlled by jets3t's property
"s3service.max-thread-count", which is by default 4.

.. Shlomo

paraUpload.diff.txt

shlomo

unread,
Aug 13, 2008, 12:34:10 PM8/13/08
to lifeguard-dev
A similar patch can be applied to IngestorBase. If you're interested
let me know and I'll send it.

.. Shlomo
>  paraUpload.diff.txt
> 3KViewDownload

dkav...@gmail.com

unread,
Aug 28, 2008, 6:48:39 PM8/28/08
to lifeguard-dev
I didn't get this in there, but you're totally on the money with this.

David

David Kavanagh

unread,
Oct 6, 2009, 7:10:35 AM10/6/09
to lifeguard-dev
Oi! I finally got to this!
Reply all
Reply to author
Forward
0 new messages