Column Limit in Presto

[Shanmuga sundaram Swaminathan]

Hi,
Is there any limit on number of columns for a table in presto?

[David Phillips]

Currently, there is no exact hard limit, but we've seen stuff break with more than few thousand.

[Shanmuga sundaram Swaminathan]

Thanks David Phillips,

I have three tables each with approximately 600 columns and trying to do full join on those three tables. I got error as below:

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Method code too large!
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
at com.facebook.presto.sql.gen.ExpressionCompiler.compilePageProcessor(ExpressionCompiler.java:87)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.enforceLayout(LocalExecutionPlanner.java:376)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:305)
at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:256)
at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:149)
at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:110)
at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:73)
at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:249)
at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:263)
at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:115)
at sun.reflect.GeneratedMethodAccessor725.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: Method code too large!
at org.objectweb.asm.MethodWriter.a(Unknown Source)
at org.objectweb.asm.ClassWriter.toByteArray(Unknown Source)
at com.facebook.presto.sql.gen.CompilerUtils.defineClasses(CompilerUtils.java:114)
at com.facebook.presto.sql.gen.CompilerUtils.defineClass(CompilerUtils.java:78)
at com.facebook.presto.sql.gen.CompilerUtils.defineClass(CompilerUtils.java:84)
at com.facebook.presto.sql.gen.ExpressionCompiler.compileProcessor(ExpressionCompiler.java:128)
at com.facebook.presto.sql.gen.ExpressionCompiler.compileAndInstantiate(ExpressionCompiler.java:93)
at com.facebook.presto.sql.gen.ExpressionCompiler.access$300(ExpressionCompiler.java:42)
at com.facebook.presto.sql.gen.ExpressionCompiler$1.load(ExpressionCompiler.java:53)
at com.facebook.presto.sql.gen.ExpressionCompiler$1.load(ExpressionCompiler.java:48)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 64 more

How can I solve this? Will the above error be solved if I increase the maximumSize of LoadingCache? If I increase, will there be any performance issues?

[Dain Sundstrom]

That is the kind of error you get when you have to many columns.

-dain

> --
> You received this message because you are subscribed to the Google Groups "Presto" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Shanmuga sundaram Swaminathan]

Thanks Dain,

If it is because of many columns, I have around 1800 columns, then How can I solve this? Will the above error be solved if I increase the maximumSize of LoadingCache in ExpressionCompiler.java(Line no.87)? If I increase, will there be any performance issues?