Could be used use more than one s3-endpoint at the same time?

[Emanuele Garuglieri]

Hi to everybody,

i need to address the following scenario:

- I need to archive on amazon S3 only the A object.

- I need to archive on seeweb S3 only the B object.

I have defined two jetS3 property file but, although I have centralized the creation of S3RestService using the correct property file, the S3RestService use always the last properties file loaded in memory!!

Can be done this scenario without reload the properties file ( in a synchonized way )  every time that I need to use the S3RestService API?

I'm looking forward to hearing from you,

Thanks to everybody,

Ema

[James Murty]

Hi,

You should be able to have two separate service instances that each
talk to a different back-end, but you will need to create a
JetS3tProperties object for each one and pass in these properties when
you construct the service.

The JetS3tProperties class has a number of convenience methods to
populate the properties from a file or input stream, so it should just
be a matter of loading your two properties files into two property
class instances before you create your services.

Hope this helps,
James

> --
> You received this message because you are subscribed to the Google Groups
> "JetS3t Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/jets3t-users/-/LbxwMpVdg2EJ.
> To post to this group, send email to jets3t...@googlegroups.com.
> To unsubscribe from this group, send email to
> jets3t-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/jets3t-users?hl=en.

[Emanuele Garuglieri]

Hi James,

I have already developed and test it and it work!

Thank you! Really much!

Regards,

Ema

[Emanuele Garuglieri]

Hi James,

I didn't solve completely my issue :(

I solved the loading problem with the correct jets3 property file but sometimes an exception will be raise and I'm not able to reproduce it!!

I just need to use two jets3 property for use two different s3Storage provider, the CloudFrontService i use only the Amazon's one.

The Exception will be raise AFTER the creation of a new RestS3Service or CloudFrontService, when I try to use an utility method of this class.

For example ( I override some path for privacy :) ): 

2013-02-04 16:45:40 [http-8080-3] DEBUG JetS3Utility  - Entering in monitoring phase.

2013-02-04 16:45:40 [http-8080-3] DEBUG JetS3Utility  - Will be use the jets3FilePath=C:\XXXXXXXXXXX/jets3t.amazon.properties

2013-02-04 16:45:40 [http-8080-3] DEBUG JetS3Utility  - Created jets3t

2013-02-04 16:45:40 [http-8080-3] DEBUG JetS3Utility  - Created awsCredentials with awsAccessKeyDecrypt=XXXXXXXXXXXXXXXXXXXX; awsSecretAccessKeyDecrypt=XXXXXXXXXXXXXXXXXXXX

2013-02-04 16:45:40 [http-8080-3] DEBUG PluginProxyUtil  - About to attempt auto proxy detection under Java version:1.6.0_04-b12

2013-02-04 16:45:40 [http-8080-3] DEBUG PluginProxyUtil  - Sun Plugin 1.4+ proxy detection class not found, will try failover detection

2013-02-04 16:45:40 [http-8080-3] DEBUG PluginProxyUtil  - Using failover proxy detection...

2013-02-04 16:45:40 [http-8080-3] DEBUG PluginProxyUtil  - Plugin Proxy Config List Property:null

2013-02-04 16:45:40 [http-8080-3] DEBUG PluginProxyUtil  - No configured plugin proxy list

2013-02-04 16:45:40 [http-8080-3] DEBUG JetS3Utility  - Created Serv...@org.jets3t.service.CloudFrontService@17f4743

org.jets3t.service.CloudFrontServiceException: CloudFront Request failed

at org.jets3t.service.CloudFrontService.performRestRequest(CloudFrontService.java:326)

at org.jets3t.service.CloudFrontService.listDistributionsImpl(CloudFrontService.java:370)

at org.jets3t.service.CloudFrontService.listStreamingDistributions(CloudFrontService.java:429)

at org.jets3t.service.CloudFrontService.listStreamingDistributions(CloudFrontService.java:455)

at xxxxx.reload(xxxxx.java:66)

at xxxxx.reload(xxxxx.java:38)

at xxxxx.getInstance(xxxxx.java:56)

[...]

Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)

at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)

at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)

at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)

at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)

at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)

at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:562)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)

at org.jets3t.service.CloudFrontService.performRestRequest(CloudFrontService.java:278)

... 70 more

An another example with RestS3Service (with seewebproperties):

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Entering createS3RestService...

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Entering in monitoring phase.

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Will be use the jets3FilePath=C:\xxxxx/jets3t.seeweb.properties

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Created jets3t

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Created awsCredentials with awsAccessKeyDecrypt=XXXXXX; awsSecretAccessKeyDecrypt=XXXXX

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - About to attempt auto proxy detection under Java version:1.6.0_04-b12

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Sun Plugin 1.4+ proxy detection class not found, will try failover detection

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Using failover proxy detection...

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Plugin Proxy Config List Property:null

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - No configured plugin proxy list

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - About to attempt auto proxy detection under Java version:1.6.0_04-b12

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Sun Plugin 1.4+ proxy detection class not found, will try failover detection

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Using failover proxy detection...

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - Plugin Proxy Config List Property:null

2013-02-04 17:44:54 [Thread-19] DEBUG PluginProxyUtil  - No configured plugin proxy list

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Created Serv...@org.jets3t.service.impl.rest.httpclient.RestS3Service@19e4d25

2013-02-04 17:44:54 [Thread-19] DEBUG JetS3Utility  - Entering createS3RestService...

2013-02-04 17:44:54 [Thread-19] DEBUG SeewebS3Utility  - Entering deleteS3Object

2013-02-04 17:44:54 [Thread-19] DEBUG SeewebS3Utility  - BucketName : XXXXX and s3ObjectKey: XXXXXXXXXXX

2013-02-04 17:44:54 [Thread-19] ERROR SeewebS3Utility  - Cannot delete S3 object XXXXXXXXXX on bucket XXXXXXXXXX

org.jets3t.service.ServiceException: Request Error: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:574)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestDelete(RestStorageService.java:1138)

at org.jets3t.service.impl.rest.httpclient.RestStorageService.deleteObjectImpl(RestStorageService.java:1529)

at org.jets3t.service.StorageService.deleteObject(StorageService.java:1058)

The credential are correct in every each loading :(

I'm really really sad...Could you help me?

This is my Utility class with static and synchronized (with the Monitor static class variable) method for retrieve the RestS3Service instance

public static RestS3Service createS3RestService(String provider, String awsAccessKeyDecrypt, String  awsSecretAccessKeyDecrypt) throws FileNotFoundException, IOException, S3ServiceException {

trace.debug("Entering createS3RestService...");

RestS3Service s3Service =  null;

synchronized (MONITOR) {

trace.debug("Entering in monitoring phase.");

String jets3FilePath = ServiceManager.getConfigurationServices().getConfigFilePath("/jets3t."+provider+".properties");

trace.debug("Will be use the jets3FilePath="+jets3FilePath);

Jets3tProperties jets3t = new Jets3tProperties();

jets3t.loadAndReplaceProperties(new FileInputStream(jets3FilePath), provider+"jetS3");

//Jets3tProperties.getInstance(Constants.JETS3T_PROPERTIES_FILENAME).loadAndReplaceProperties(new FileInputStream(jets3FilePath), provider+"jetS3");

trace.debug("Created jets3t");

AWSCredentials awsCredentials = new AWSCredentials(awsAccessKeyDecrypt, awsSecretAccessKeyDecrypt);

trace.debug("Created awsCredentials with awsAccessKeyDecrypt=" + awsAccessKeyDecrypt + "; awsSecretAccessKeyDecrypt="+awsSecretAccessKeyDecrypt);

s3Service = new RestS3Service(awsCredentials, provider, new BasicCredentialsProvider(), jets3t);

trace.debug("Created Services@"+s3Service);

//s3Service = new RestS3Service(awsCredentials);

}

trace.debug("Entering createS3RestService...");

return s3Service;

}

and the CloudFrontService instance

public static CloudFrontService createCloudFrontService(String provider, String awsAccessKeyDecrypt, String  awsSecretAccessKeyDecrypt) throws CloudFrontServiceException, FileNotFoundException, IOException {

trace.debug("Entering createCloudFrontService...");

CloudFrontService cloudFrontService =  null;

synchronized (MONITOR) {

trace.debug("Entering in monitoring phase.");

String jets3FilePath = ServiceManager.getConfigurationServices().getConfigFilePath("/jets3t."+provider+".properties");

trace.debug("Will be use the jets3FilePath="+jets3FilePath);

Jets3tProperties jets3t = new Jets3tProperties();

jets3t.loadAndReplaceProperties(new FileInputStream(jets3FilePath), provider+"jetS3");

trace.debug("Created jets3t");

AWSCredentials awsCredentials = new AWSCredentials(awsAccessKeyDecrypt, awsSecretAccessKeyDecrypt);

trace.debug("Created awsCredentials with awsAccessKeyDecrypt=" + awsAccessKeyDecrypt + "; awsSecretAccessKeyDecrypt="+awsSecretAccessKeyDecrypt);

cloudFrontService = new CloudFrontService(awsCredentials, provider, new BasicCredentialsProvider(), jets3t);

trace.debug("Created Services@"+cloudFrontService);

}

return cloudFrontService;

}

And those are my two jets3 configuration files :

Seeweb:

#########################

# Low-level configuration of jets3t toolkit behaviour.

#

# For detailed documentation, please see:

# http://www.jets3t.org/toolkit/configuration.html

#

# Properties set in this file will affect services and applications 

# using the JetS3t library.

#########################

###

# StorageService (apply to both S3 and Google Storage services)

###

storage-service.internal-error-retry-max=5

storage-service.disable-live-md5=false

###

# ThreadedStorageService (multi-threaded wrapper for S3 or Google services)

###

threaded-service.max-thread-count=2

threaded-service.admin-max-thread-count=20

threaded-service.ignore-exceptions-in-multi=false

###

# S3ServiceMulti (multi-threaded wrapper for S3 service)

###

s3service.max-thread-count=2

s3service.admin-max-thread-count=20

s3service.ignore-exceptions-in-multi=false

###

# RestS3Service

###

s3service.https-only=true

s3service.s3-endpoint=seewebstorage.it

s3service.s3-endpoint-http-port=80

s3service.s3-endpoint-https-port=443

s3service.disable-dns-buckets=false

s3service.default-bucket-location=US

s3service.enable-storage-classes=true

s3service.default-storage-class=STANDARD

#s3service.default-storage-class=REDUCED_REDUNDANCY

# Uncomment this directive to use Amazon S3's server-side encryption

#s3service.server-side-encryption=AES256

# Virtual path for non-standard servers, such as Eucalyptus

#s3service.s3-endpoint-virtual-path=/services/Walrus

###

# GoogleStorageService

###

gsservice.https-only=true

gsservice.gs-endpoint=commondatastorage.googleapis.com

gsservice.gs-endpoint-http-port=80

gsservice.gs-endpoint-https-port=443

###

# CloudFrontService

###

cloudfront-service.internal-error-retry-max=5

###

# REST/HTTP HttpClient properties

###

httpclient.max-connections=20

#httpclient.max-connections-per-host=5

httpclient.retry-max=5

httpclient.connection-timeout-ms=60000

httpclient.socket-timeout-ms=60000

httpclient.stale-checking-enabled=true

#httpclient.useragent=My User Agent String

#httpclient.read-throttle=50

httpclient.authentication-preemptive=false

httpclient.proxy-autodetect=true

httpclient.requester-pays-buckets-enabled=false

# TCP window size hints for kernel 

#httpclient.socket-receive-buffer=65536

#httpclient.socket-send-buffer=65536

#httpclient.connection-manager-timeout=0

###

# GUI application properties

###

gui.verboseErrorDialog=true

###

# Upload properties

###

uploads.stream-retry-buffer-size=131072

uploads.storeEmptyDirectories=true

###

# Download properties

###

downloads.restoreLastModifiedDate=true

###

# File/Object comparison properties

###

#filecomparer.skip-symlinks=false

#filecomparer.use-md5-files=false

#filecomparer.generate-md5-files=false

#filecomparer.md5-files-root-dir=/custom/path/for/md5/files

#filecomparer.skip-upload-of-md5-files=false

#filecomparer.assume-local-latest-in-mismatch=false

###

# Encryption Algorithm used by JetS3t applications.

#

# This setting does not enable encryption in the JetS3t toolkit as

# a whole. For this, refer to the Javadocs for the EncryptionUtil

# and ObjectUtils classes.

#

# Note that some cryptography options will only be available

# if the Java Cryptography Extension (JCE) Unlimited Strength 

# Jurisdiction Policy Files are installed.

###

crypto.algorithm=PBEWithMD5AndDES

###

# XML Parsing properties

###

xmlparser.sanitize-listings=true

###

# Amazon DevPay Settings

#

# Default values for DevPay user and product tokens.

###

#devpay.user-token=

#devpay.product-token=

and the Amazon version is exactly the same, except for this part:

s3service.s3-endpoint=s3.amazonaws.com

s3service.s3-endpoint-http-port=80

s3service.s3-endpoint-https-port=443

s3service.disable-dns-buckets=false

[James Murty]

Hi Emanuele,

The javax.net.ssl.SSLPeerUnverifiedException error you are getting isn't really to do with JetS3t specifically, it's probably because your Java installation does not recognise the Amazon SSL certificates and is therefore rejecting secure connections to S3.

To fix this you will likely need to update your Java installation, or your system in general, to get up-to-date certificates.

---
jamesmurty.com

To unsubscribe from this group and stop receiving emails from it, send an email to jets3t-users...@googlegroups.com.

To post to this group, send email to jets3t...@googlegroups.com.

Visit this group at http://groups.google.com/group/jets3t-users?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Emanuele Garuglieri]

Hi James,

thank you for the quickly answer!

But if it is a certificates problem, I couldn't never be able to use the S3Service or CloudFrontServier, do you agree?

Or do you think it could be happen sometimes in a random way?

How can I retrieve up-to-date certificates?

Best Regards,

Emanuele Garuglieri

[James Murty]

You have customised one endpoint with s3service.s3-endpoint=seewebstorage.it so this may cause SSL exceptions if that service doesn't have valid SSL certificates, or your system doesn't recognise them.

> But if it is a certificates problem, I couldn't never be able to use the S3Service or CloudFrontServier, do you agree?

If your Java installation or system has outdated certificates this would only affect Java programs.

I can't help much more I'm afraid. Try using JetS3t to talk to S3 without any of your config customisations, and if that works OK then there's a problem with your customisations. If you can't talk to S3 at all even with normal JetS3t configs, then it's definitely Java/system certificate problems.