HikariCP on Google App Engine

[Ray Vanderborght]

Wondering if anyone here has gotten HikariCP to work successfully on appengine?

I work on a site that gets a fair amount of traffic and we recently added a postgres datasource with no connection pooling to our app.

With this setup we quickly hit an appengine imposed limit on number of socket connections per day, which broke the part of the app that relies on postgres. The limit is a pretty high number, but it seems not high enough. For the time-being google has increased our threshold, and that has resolved our immediate problem, but it seems that connection pooling is really the right answer in this case. We'd love to use HikariCP.

These stackoverflow questions seem to indicate that some work went into making HikariCP possible on GAE, but it looks like it still has issues:

http://stackoverflow.com/questions/21457736/connection-pool-on-app-engine-with-cloud-sql

http://stackoverflow.com/questions/26841106/jdbc-connection-pool-compatible-with-app-engine

Anybody able to report they got this working?

Thanks,

Ray

[Brett Wooldridge]

In particular what have you tried? Have you tried configuring the ThreadFactory using the GAE factory? If so, what is the error/failure that you are seeing?

[Ray Vanderborght]

Well, unfortunately the outlook seems grim... I tried this code:

    public Connection jdbcConnection(final Environment env) throws Exception {
      Class.forName("org.postgresql.Driver");

      final PostgresAccount acct = env.getPostgresAccount();

      final String url = acct.getJdbcUrl("hub");

      HikariConfig config = new HikariConfig();
      config.setThreadFactory(ThreadManager.backgroundThreadFactory());
      config.setJdbcUrl(url);
      config.addDataSourceProperty("cachePrepStmts", "true");
      config.addDataSourceProperty("prepStmtCacheSize", "250");
      config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");

      HikariDataSource ds = new HikariDataSource(config);
      return ds.getConnection(acct.getRole(), acct.getPassword());

//    return DriverManager.getConnection(url, acct.getRole(), acct.getPassword());
   }

And got this result:

java.lang.NoClassDefFoundError: java.util.concurrent.atomic.LongAdder is a restricted class. Please see the Google  App Engine developer's guide for more details.

at com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime.java:52)

at com.zaxxer.hikari.util.Sequence$Java8Sequence.<init>(Sequence.java:78)

at com.zaxxer.hikari.util.Sequence$Factory.create(Sequence.java:54)

at com.zaxxer.hikari.util.QueuedSequenceSynchronizer.<init>(QueuedSequenceSynchronizer.java:60)

at com.zaxxer.hikari.util.ConcurrentBag.<init>(ConcurrentBag.java:98)

at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:103)

at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)

at com.gearlaunch.hub.guice.HubModule.jdbcConnection(HubModule.java:532)

at com.gearlaunch.hub.guice.HubModule$$FastClassByGuice$$8f0f12f8.invoke(<generated>)

at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:272)

at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:172)

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)

at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)

at com.gearlaunch.hub.analytics.sql.RequestConnectionProvider.get(RequestConnectionProvider.java:30)

at com.gearlaunch.hub.analytics.sql.RequestConnectionProvider.get(RequestConnectionProvider.java:14)

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)

at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)

at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)

at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)

at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)

at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:167)

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)

at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)

at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)

at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93)

at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)

at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80)

at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:62)

at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:984)

at com.gearlaunch.hub.util.GuicyDeferredTask.run(GuicyDeferredTask.java:25)

at com.google.apphosting.utils.servlet.DeferredTaskServlet.performRequest(DeferredTaskServlet.java:142)

at com.google.apphosting.utils.servlet.DeferredTaskServlet.service(DeferredTaskServlet.java:101)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)

at com.gearlaunch.hub.page.RouterFilter.doFilter(RouterFilter.java:63)

at com.gearlaunch.hub.util.AbstractFilter.doFilter(AbstractFilter.java:48)

at com.gearlaunch.hub.util.WebApplicationExceptionFilter.doFilter(WebApplicationExceptionFilter.java:28)

at com.gearlaunch.hub.util.AbstractFilter.doFilter(AbstractFilter.java:48)

at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)

at com.gearlaunch.hub.analytics.sql.RequestConnectionFilter.doFilter(RequestConnectionFilter.java:30)

at com.gearlaunch.hub.util.AbstractFilter.doFilter(AbstractFilter.java:48)

at com.gearlaunch.hub.util.InternetExplorerFilter.doFilter(InternetExplorerFilter.java:40)

at com.gearlaunch.hub.util.AbstractFilter.doFilter(AbstractFilter.java:48)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)

at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)

at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:98)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:512)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[Ray Vanderborght]

It just occurred to me that LongAdder is a Java8 only class and GAE is on Java7... Our app runs on Java8 locally (in dev) but we deploy to Java7 jvms because AppEngine. The reason for this setup it that we use lambdas in our codebase and have retrolambda to crosscompile classes down to Java7 for deployment.

Does HikariCP detect at runtime whether Java8 features are available? Is there a flag we can set to force it to treat it as Java7?

[Brett Wooldridge]

HikariCP should detect Java 8.  For some reason the detection is not working on GAE.

HikariCP does:

classloader.loadClass("java.util.concurrent.atomic.LongAdder")

which should throw a ClassNotFoundException on Java 7.  It is strange that Google is trapping that (somehow) -- probably through their custom ClassLoader.

But there is good news.  We did have the forethought to provide an override.  We've never had to use it before.  If you set this VM property:

-Dcom.zaxxer.hikari.useAtomicLongSequence=true

Then HikariCP will use an AtomicLong instead of LongAdder.  Though there may be other issues looming, if Google is thwarting our Java 8 detection...

-Brett

[Brett Wooldridge]

By the way, ds.getConnection(username, password) is not supported (anymore).  You need to configure the username and password in the HikariConfig.

-Brett

[Ray Vanderborght]

Thanks for the tips. I'll try the flag, but in the meantime I was able to get past the Java8 problem by switching to the older java6 hikari build.

However, after also changing my test to be a bit more realistic by re-using the datasource, I get the same exception seen in the stackoverflow question I posted earlier:

http://stackoverflow.com/questions/26841106/jdbc-connection-pool-compatible-with-app-engine

Namely, this:

Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThreadGroup")

@Provides
@Singleton
public HikariDataSource pooledDataSource(final Environment env) {

   final PostgresAccount acct = env.getPostgresAccount();
   final String url = acct.getJdbcUrl("hub");

   final HikariConfig config = new HikariConfig();

   config.setThreadFactory(ThreadManager.backgroundThreadFactory());
   config.setJdbcUrl(url);
   config.addDataSourceProperty("cachePrepStmts", "true");
   config.addDataSourceProperty("prepStmtCacheSize", "250");
   config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");

   config.setConnectionTestQuery("select 1");
   config.setUsername(acct.getRole());
   config.setPassword(acct.getPassword());

   HikariDataSource ds = new HikariDataSource(config);

   return ds;
}

/**
 * Get a new, in this case unpooled connection. Normal connections are provided by the RequestConnectionProvider
 * TODO: add a connection pool
 */
@Provides @Raw
public Connection jdbcConnection(final HikariDataSource ds) throws Exception {
   Class.forName("org.postgresql.Driver");

   return ds.getConnection();
}

[Brett Wooldridge]

That is a strange exception to get, as that is the purpose of the GAE ThreadFactory.  Are your sure your application is configured as a "backend" application?  "Frontend" applications are not allowed to create background threads.  Can you post the full stack trace from this exception?

-Brett

[Ray Vanderborght]

Ah, that is the problem. Backends are deprecated in favor of Modules now, but after a closer look at the docs, background threads are only allowed on modules that are set to "manual" or "basic" scaling, and we're set to "automatic" scaling, which doesn't support them. We really like that automatic scaling for our app, so I think we're going to have to take a step back re-evaluate the options.