What is most resource friendly way to create and reuse HPCloudObjectStorageClient

[n3phele]

Hi jclouders,

I am trying to optimize my GAE startup times and memory consumption
with jclouds. My code uses the HPCloudObjectStorageClient rather than
the Blobstore interface.

I have two questions:

1. Im wondering if there is a lower cost way to instantiate my client
than I do currently.

My current code looks like

context = new BlobStoreContextFactory().createContext("hpcloud-
objectstorage", this.accessKey, this.secretKey);

HPCloudObjectStorageClient client =
HPCloudObjectStorageClient.class.cast(context.getProviderSpecificContext().getApi());

Is their a cheaper way to create the client, using some other lower
level methods?

2. I have a variety of requests to process, and these can use
different credentials to access the cloud.

Right now, I am currently caching the clients, along the lines of:

Map<String, BlobStoreContext> cache;

When a request arrives, I look to see if I have a context already in
the cache. Is there a better way of doing this?

Thanks in advance,
\n

[Adrian Cole]

Hi, n3phele.

Part of the overhead you are experiencing is looking up the provider
by name. If you look it up by type, it will be faster. Also, if you
are not using BlobStore, you can skip that view. Finally, you should
be using the google appengine module.

ex.
RestContext<HPCloudObjectStorageClient,
HPCloudObjectStorageAsyncClient> context =
ContextBuilder.newBuilder(new HPCloudObjectStorageProviderMetadata())
.credentials("tenant:accessKey", "secret")
.modules(ImmutableSet.<Module>of(new
AsyncGoogleAppEngineConfigurationModule()))
.build();

try using Stopwatch and see if it helps!

-A

> --
> You received this message because you are subscribed to the Google Groups "jclouds" group.
> To post to this group, send email to jcl...@googlegroups.com.
> To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.
>

[Adrian Cole]

Note that also, you can use guava LoadingCache/CacheBuilder, which is
easy to use for caching contexts.

-A

On Sun, Jun 3, 2012 at 2:14 PM, n3phele <n3p...@gmail.com> wrote:

[n3phele]

Adrian

Thx. Just trying to upload the first set of changes. GAE is acting weirdly  slooowly precompiling 485 files. I guess Ive been moved! 

Can you give me a code snippet example of  LoadingCache/CacheBuilder to get me started?

Best

\n

[n3phele]

Adrian

It works! It is better, but there is still a pretty stiff startup cost for the first swift client request. Part of this may be the keystone exchange.

This is first time the app does a jclouds request... Note that it is already running so this is not the first time it is processing requests.. (Around 12s)

2. I2012-06-03 16:20:05.750

   com.google.inject.internal.util.$FinalizableReferenceQueue$SystemLoader loadFinalizer: Not allowed to access system class loader.
   

3. I2012-06-03 16:20:05.825

   com.google.inject.internal.util.$FinalizableReferenceQueue <init>: Failed to start reference finalizer thread. Reference cleanup will only occur when new references are created.
   java.lang.reflect.InvocationTargetException
   	at com.google.appengine.runtime.Request.process-5267caf7bb30b833(Request.java)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:43)
   	at com.google.inject.internal.util.$FinalizableReferenceQueue.<init>(FinalizableReferenceQueue.java:124)
   	at com.google.inject.internal.util.$MapMaker$QueueHolder.<clinit>(MapMaker.java:787)
   	at com.google.inject.internal.util.$MapMaker$WeakEntry.<init>(MapMaker.java:946)
   	at com.google.inject.internal.util.$MapMaker$Strength$1.newEntry(MapMaker.java:312)
   	at com.google.inject.internal.util.$MapMaker$StrategyImpl.newEntry(MapMaker.java:498)
   	at com.google.inject.internal.util.$MapMaker$StrategyImpl.newEntry(MapMaker.java:419)
   	at com.google.inject.internal.util.$CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2029)
   	at com.google.inject.internal.Annotations$AnnotationChecker.hasAnnotations(Annotations.java:116)
   	at com.google.inject.internal.Annotations.isBindingAnnotation(Annotations.java:180)
   	at com.google.inject.Key.ensureIsBindingAnnotation(Key.java:366)
   	at com.google.inject.Key.strategyFor(Key.java:338)
   	at com.google.inject.Key.get(Key.java:272)
   	at com.google.inject.internal.AbstractBindingBuilder.annotatedWithInternal(AbstractBindingBuilder.java:82)
   	at com.google.inject.internal.ConstantBindingBuilderImpl.annotatedWith(ConstantBindingBuilderImpl.java:49)
   	at org.nnsoft.guice.rocoto.configuration.ConfigurationModule$1.toValue(ConfigurationModule.java:112)
   	at org.nnsoft.guice.rocoto.configuration.ConfigurationModule.bindProperties(ConfigurationModule.java:152)
   	at org.nnsoft.guice.rocoto.configuration.ConfigurationModule.bindProperties(ConfigurationModule.java:127)
   	at org.jclouds.config.BindPropertiesToExpandedValues$1.bindConfigurations(BindPropertiesToExpandedValues.java:53)
   	at org.nnsoft.guice.rocoto.configuration.ConfigurationModule.configure(ConfigurationModule.java:67)
   	at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
   	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
   	at com.google.inject.spi.Elements.getElements(Elements.java:101)
   	at com.google.inject.spi.Elements.getElements(Elements.java:92)
   	at org.nnsoft.guice.rocoto.Rocoto.expandVariables(Rocoto.java:51)
   	at org.nnsoft.guice.rocoto.Rocoto.expandVariables(Rocoto.java:46)
   	at org.jclouds.config.BindPropertiesToExpandedValues.configure(BindPropertiesToExpandedValues.java:49)
   	at com.google.inject.AbstractModule.configure(AbstractModule.java:59)
   	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
   	at com.google.inject.spi.Elements.getElements(Elements.java:101)
   	at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133)
   	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
   	at com.google.inject.Guice.createInjector(Guice.java:95)
   	at com.google.inject.Guice.createInjector(Guice.java:72)
   	at com.google.inject.Guice.createInjector(Guice.java:62)
   	at org.jclouds.ContextBuilder.expandProperties(ContextBuilder.java:302)
   	at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:254)
   	at org.jclouds.ContextBuilder.build(ContextBuilder.java:535)
   	at org.jclouds.ContextBuilder.build(ContextBuilder.java:487)
   	at n3phele.storage.swift.CloudStorageImpl.getJcloudsContext(CloudStorageImpl.java:387)
   	at n3phele.storage.swift.CloudStorageImpl.getFileList(CloudStorageImpl.java:194)
   	at n3phele.storage.CloudStorage$1.getFileList(CloudStorage.java:67)
   	at n3phele.service.rest.impl.RepositoryResource.list(RepositoryResource.java:204)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:43)
   	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
   	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
   	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
   	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
   	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
   	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
   	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
   	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
   	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1483)
   	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1414)
   	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)
   	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)
   	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)
   	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
   	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
   	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
   	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
   	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
   	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
   	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   	at org.mortbay.jetty.Server.handle(Server.java:326)
   	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
   	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
   	at java.lang.Thread.run(Thread.java:679)
   Caused by: java.security.AccessControlException: access denied (java.lang.RuntimePermission modifyThreadGroup)
   	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:355)
   	at java.security.AccessController.checkPermission(AccessController.java:567)
   	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   	at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
   <continued in next message>

4. I2012-06-03 16:20:05.825

   <continued from previous message>
   at java.lang.Thread.init(Thread.java:353)
   	at java.lang.Thread.<init>(Thread.java:479)
   	at com.google.inject.internal.util.$Finalizer.<init>(Finalizer.java:92)
   	at com.google.inject.internal.util.$Finalizer.startFinalizer(Finalizer.java:81)
   	... 88 more

5. I2012-06-03 16:20:17.379

   n3phele.storage.swift.CloudStorageImpl getFileList: Found ua_16may2012.zip

   This is the next time around.. (about 700ms - but this includes some other request processing )

   
   2012-06-03 16:20:24.019
   n3phele.service.rest.impl.BasicSecurityFilter$Authorizer isUserInRole: Checking role authenticated with user
   
   I2012-06-03 16:20:24.800
   n3phele.storage.swift.CloudStorageImpl getFileList: Found agent-with-swift.tgz
   
   

   Is this what you would expect?

   Best

   -N

[Adrian Cole]

One thing you can do is pre-emptively create the context on servlet
context startup. I'd expect some delays, but one way to be sure about
what is one thing vs another is to set the log category for
"jclouds.headers" to FINE

Also, make sure you use the Async client whenever possible, as this
does work in GAE. Ex. if you have several swift requests, you can
issue them all, then block on the Futures later.

Here's some code I played around with:
https://github.com/jclouds/jclouds/blob/master/demos/googleappengine/src/main/java/org/jclouds/samples/googleappengine/GetAllResourcesController.java

I hope this helps!
-A

p.s. this is the cachebuilder I was referring to.
http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheBuilder.html

[n3phele]

Thx again.

& the cachebuilder does look interesting.