synchronize - multi threaded s3 service
linuxphan
DOWN load content to S3.
The logic is:
1)upload content
2)when content complete - upload a manifest
3)app logic start to download and process latest (from manifest)
content from somewhere else
Prior to 0.8.1 there was no multi-threading options, so this worked
with the exception of the file size limit.
Now in 0.8.1, the invoking of the synchronize service to upload is
returning before the upload job is actually complete and the next step
in the application is firing too soon
Should I be invoking org.jets3t.apps.synchronize.Synchronize with a
different StorageService? Currently, the StorageService is built with
the provider credentials from
org.jets3t.service.impl.rest.httpclient.RestS3Service
It seems the console(main method) Synchronize waits for all threads to
complete before releasing. I would like the Synchronize.run() method
to do the same.
thanks
James Murty
The Synchronize application has always been multi-threaded, nothing
has changed there. The run method will also finish its operations
before returning, it doesn't spawn any background threads so there
shouldn't be anything special you need to do.
I don't think Synchronize has changed in any way that would affect
your use of it, perhaps something in your application or usage has
changed?
> --
> You received this message because you are subscribed to the Google Groups "JetS3t Users" group.
> To post to this group, send email to jets3t...@googlegroups.com.
> To unsubscribe from this group, send email to jets3t-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jets3t-users?hl=en.
>
>
linuxphan
getting:
WARN [FileComparer] Identified an object that looks like a directory
placeholder created by Panic's Transmit application. If this object
was indeed created by Transmit, it will not be handled properly unless
the JetS3t property "filecomparer.ignore-panic-dir-placeholders" is
set to true. S3Object [key=idx4/http://dummy.network.com/libraries/
testing/1304647910944/data/index/,bucket=dummy-dev,lastModified=Fri
May 06 02:11:55 UTC 2011, dataInputStream=null]
Metadata={ETag="d41d8cd98f00b204e9800998ecf8427e", Date=Fri May 06
02:11:59 UTC 2011, Content-Length=0, jets3t-original-file-date-
iso8601=2011-05-06T02:11:50.000Z, id-2=rrdMmQbyEeafDjx1xbjf0C/A5ro0nz//
EL2iQS8BGWqN+EzGc3bbJ7i1y/71M5Ds, request-id=ADB8DE9016AFBC62, Last-
Modified=Fri May 06 02:11:55 UTC 2011, Content-Type=binary/octet-
stream}
...
same message for each subdirectory
then:
ERROR [S3ServiceMulti$ThreadGroupManager] A thread failed with an
exception. Firing ERROR event and cancelling all threads
java.io.FileNotFoundException: /opt/backend/dummy/directory/repository/
1304647917427/conf (Is a directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at
org.jets3t.service.multithread.DownloadPackage.getOutputStream(DownloadPackage.java:
120)
at org.jets3t.service.multithread.S3ServiceMulti
$DownloadObjectRunnable.run(S3ServiceMulti.java:2026)
at java.lang.Thread.run(Thread.java:662)
I hadn't seen this message prior to upgrading from 0.6.1 to 0.8.1.
This may have come in 0.7x or later, I'm not sure. It seems it
doesn't recognize my key structure for up/downloading files? That's
why it says the files aren't there when I try to process them in the
next step.
I found a related thread here: http://goo.gl/9Bm60 but not a
resolution yet
I've added filecomparer.ignore-panic-dir-placeholders=true in
synchronize.properties but that doesn't seem to make a difference like
the message states.
I use a key structure for my files with the following naming
convention: idxN/http://domain_name.com/libraries/library_name/
timestamp/ with subdirectories under that
linuxphan
the upload wasn't done by Panic's Transmit
all UP/DOWN synchronizing is done via Jets3t
any pointers appreciated
thanks
linuxphan
James Murty
I think your issue is due to changes to JetS3t and Synchronize that
were intended to make the tools more compatible with other S3 client
tools. These changes have been happening from version 0.7.2 onwards,
and particularly in release 0.8.0.
In short, object's with key names ending in a slash '/' character will
generally be treated as directory place-holder objects, because that
is the strategy used by many other tools include Amazon's own online
console. Previously, objects needed to have special JetS3t-specific
metadata to be considered directory place-holders.
Can you check which version of JetS3t you're running? I'm pretty sure
the warning about the "filecomparer.ignore-panic-dir-placeholders" is
only from versions before 0.8.0.
Can you upgrade to the latest 0.8.1 version of Jets3t then try a sync
and let me know what error messages you get? If you still have
problems with the latest version I will be in a better position to
debug the problem and provide a fix, I can't provide much support for
older versions.
Cheers,
James
linuxphan
jar's
the upload is done to S3 via jets3t synchronize, and it appears to be
successful following the path in the error message
but the download seems to fail with the placeholder issues of not
knowing key structure vs directory
thanks
linuxphan
the 0.8.1 codebase.
i found that the jvm container my app was running in had an older
version of jets3t running in another war
i no longer have that particular issue
thanks for the help
James Murty
placeholders in 0.8.0 so I'm happy it's working for you.