Classloader issues; conflicting classes even with ClassLoaderSafeEventListener

[gray]

Hello,

I'm running into an issue that I initially worked around by using an apache based HttpClient instead of anything backed by a jersey rest client. At this point, due to performance requirements, I feel that I need to utilize the client described below.

Trying to use the Apache AtlasClientV2 REST client within a Presto plugin consistently gives me this error due to the fact that I have to include the following dependencies in my pom.xml:

'''
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<version>2.17</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>2.17</version>
</dependency>
'''

When I do not include the above dependencies, there is no implementation method for a few of the methods required by AtlasClientV2. I get the following errors in that case:

'''
java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:120)
at org.apache.atlas.AtlasBaseClient.getAddressIfActive(AtlasBaseClient.java:516)
at org.apache.atlas.AtlasBaseClient.selectActiveServerAddress(AtlasBaseClient.java:500)
at org.apache.atlas.AtlasBaseClient.determineActiveServiceURL(AtlasBaseClient.java:291)
at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:420)
at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:148)
at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:106)
at com.eventbrite.integral.atlas.ApacheAtlasClient.setupAtlasClient(ApacheAtlasClient.java:43)
at com.eventbrite.integral.atlas.ApacheAtlasClient.getInstance(ApacheAtlasClient.java:23)
at com.eventbrite.integral.presto.handlers.HandleSelect.<clinit>(HandleSelect.java:33)
at com.eventbrite.integral.presto.PrestoQueryHandler.createDispatcher(PrestoQueryHandler.java:80)
at com.eventbrite.integral.presto.PrestoQueryHandler.<init>(PrestoQueryHandler.java:49)
at com.eventbrite.integral.presto.PrestoEventListener.<init>(PrestoEventListener.java:42)
at com.eventbrite.integral.presto.PrestoEventListenerFactory.create(PrestoEventListenerFactory.java:19)
at com.facebook.presto.eventlistener.EventListenerManager.setConfiguredEventListener(EventListenerManager.java:82)
at com.facebook.presto.eventlistener.EventListenerManager.loadConfiguredEventListener(EventListenerManager.java:67)
at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)
at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)
'''

'''
2018-11-07T10:31:14.624-0800 ERROR main com.facebook.presto.server.PrestoServer org.apache.atlas.utils.AuthenticationUtil.isKerberosAuthenticationEnabled(Lorg/apache/hadoop/security/UserGroupInformation;)Z
java.lang.NoSuchMethodError: org.apache.atlas.utils.AuthenticationUtil.isKerberosAuthenticationEnabled(Lorg/apache/hadoop/security/UserGroupInformation;)Z
at org.apache.atlas.AtlasBaseClient.getClient(AtlasBaseClient.java:283)
at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:436)
at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:164)
at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:106)
at com.eventbrite.integral.atlas.ApacheAtlasClient.setupAtlasClient(ApacheAtlasClient.java:43)
at com.eventbrite.integral.atlas.ApacheAtlasClient.getInstance(ApacheAtlasClient.java:23)
at com.eventbrite.integral.presto.handlers.HandleSelect.<clinit>(HandleSelect.java:33)
at com.eventbrite.integral.presto.PrestoQueryHandler.createDispatcher(PrestoQueryHandler.java:80)
at com.eventbrite.integral.presto.PrestoQueryHandler.<init>(PrestoQueryHandler.java:49)
at com.eventbrite.integral.presto.PrestoEventListener.<init>(PrestoEventListener.java:42)
at com.eventbrite.integral.presto.PrestoEventListenerFactory.create(PrestoEventListenerFactory.java:19)
at com.facebook.presto.eventlistener.EventListenerManager.setConfiguredEventListener(EventListenerManager.java:82)
at com.facebook.presto.eventlistener.EventListenerManager.loadConfiguredEventListener(EventListenerManager.java:67)
at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)
at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)
'''

After including jersey core and container dependencies, I get the below error.

'''
2018-11-07T10:56:03.709-0800 ERROR main com.facebook.presto.server.PrestoServer ClassCastException: attempting to castjar:file:/usr/lib/presto/lib/javax.ws.rs-api-2.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/data/presto/plugin/presto-event-listener/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class
java.lang.LinkageError: ClassCastException: attempting to castjar:file:/usr/lib/presto/lib/javax.ws.rs-api-2.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/data/presto/plugin/presto-event-listener/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class
at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:146)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:120)
at javax.ws.rs.core.MediaType.valueOf(MediaType.java:179)
at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:64)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:176)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
at com.sun.jersey.api.client.Client.init(Client.java:343)
at com.sun.jersey.api.client.Client.access$000(Client.java:119)
at com.sun.jersey.api.client.Client$1.f(Client.java:192)
at com.sun.jersey.api.client.Client$1.f(Client.java:188)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:188)
at com.sun.jersey.api.client.Client.<init>(Client.java:171)
at org.apache.atlas.AtlasBaseClient.getClient(AtlasBaseClient.java:273)
at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:413)
at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:148)
at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:106)
at com.eventbrite.integral.atlas.ApacheAtlasClient.setupAtlasClient(ApacheAtlasClient.java:43)
at com.eventbrite.integral.atlas.ApacheAtlasClient.getInstance(ApacheAtlasClient.java:23)
at com.eventbrite.integral.presto.handlers.HandleSelect.<clinit>(HandleSelect.java:33)
at com.eventbrite.integral.presto.PrestoQueryHandler.createDispatcher(PrestoQueryHandler.java:80)
at com.eventbrite.integral.presto.PrestoQueryHandler.<init>(PrestoQueryHandler.java:49)
at com.eventbrite.integral.presto.PrestoEventListener.<init>(PrestoEventListener.java:42)
at com.eventbrite.integral.presto.PrestoEventListenerFactory.create(PrestoEventListenerFactory.java:19)
at com.facebook.presto.eventlistener.EventListenerManager.setConfiguredEventListener(EventListenerManager.java:82)
at com.facebook.presto.eventlistener.EventListenerManager.loadConfiguredEventListener(EventListenerManager.java:67)
at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)
at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)
'''

I was intially directed to fix it by using a classloader safe implementation of EventListener, which I have done. The EventListenerFactory instantiates an instance of my original EventListener implementation, and passes it to a ClassLoaderSafeEventListener along with the thread classloader. The ClassLoaderSafeEventListener uses the instance as a delegate, and calls the delegate's methods after wrapping each as below:

'''
public class ClassLoaderSafeEventListener
implements EventListener
{
private final EventListener delegate;
private final ClassLoader classLoader;

public ClassLoaderSafeEventListener(EventListener delegate, ClassLoader classLoader)
{
this.delegate = requireNonNull(delegate, "delegate is null");
this.classLoader = requireNonNull(classLoader, "classLoader is null");
}

@Override
public void queryCreated(QueryCreatedEvent queryCreatedEvent)
{
try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(classLoader)) {
delegate.queryCreated(queryCreatedEvent);
}
}

@Override
public void queryCompleted(QueryCompletedEvent queryCompletedEvent)
{
try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(classLoader)) {
delegate.queryCompleted(queryCompletedEvent);
}
}

@Override
public void splitCompleted(SplitCompletedEvent splitCompletedEvent)
{
try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(classLoader)) {
delegate.splitCompleted(splitCompletedEvent);
}
}
}

'''

I'm still getting the above error, and i do not know what i should do next. Any help is greatly appreciated.

[Dain Sundstrom]

> On Nov 7, 2018, at 12:35 PM, gray <gr...@eventbrite.com> wrote:
>
>
> I'm still getting the above error, and i do not know what i should do next. Any help is greatly appreciated.

The root cause of the exception is EventListenerManager calling EventListenerFactory.create without setting a thread context class loader:

> at com.eventbrite.integral.presto.PrestoEventListenerFactory.create(PrestoEventListenerFactory.java:19)
> at com.facebook.presto.eventlistener.EventListenerManager.setConfiguredEventListener(EventListenerManager.java:82)
> at com.facebook.presto.eventlistener.EventListenerManager.loadConfiguredEventListener(EventListenerManager.java:67)
> at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)

> at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)f

I would update the code in EventListenerManager to use the same technique from ConnectorManager:

https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/connector/ConnectorManager.java#L318

If possible, please send a PR (or file an issue) for this.

-dain

[David Phillips]

The error comes from the EventListenerFactory.create() call, as Dain pointed out. It would be best to handle this in Presto, but for now, you can workaround it by adding your own ThreadContextClassLoader wrapper for that.

Note that this is really a bug in the library you are using — libraries should not be loading from the system class loader.

> On Nov 7, 2018, at 12:35 PM, gray <gr...@eventbrite.com> wrote:
>

> --
> You received this message because you are subscribed to the Google Groups "Presto" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[gr...@eventbrite.com]

On Wednesday, November 7, 2018 at 10:13:03 PM UTC-6, David Phillips wrote:
> The error comes from the EventListenerFactory.create() call, as Dain pointed out. It would be best to handle this in Presto, but for now, you can workaround it by adding your own ThreadContextClassLoader wrapper for that.
>
> Note that this is really a bug in the library you are using — libraries should not be loading from the system class loader.

Thanks David and Dain. Do you mean wrapping the .create() call within EventListenerManager.setConfiguredEventListener() like so:

try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(eventListenerFactory.getClass().getClassLoader())) {
EventListener eventListener = eventListenerFactory.create(ImmutableMap.copyOf(properties));
this.configuredEventListener.set(Optional.of(eventListener));
}

I've tried this, and I'm still running into the same issue. Still getting AbstractMethodError before including jersey server core and after adding, still getting the ClassCastException.

[David Phillips]

Can you put up the code for the listener, or a stripped down version that reproduces the issue?

Alternatively, maybe try a different HTTP client. We use the Jetty client in Presto (via the Airlift http-client wrapper) and OkHttp for the CLI and JDBC driver. Both work well. I’d start with OkHttp.

[gray]

Entry point looks like this:

public class PrestoEventListenerFactory
implements EventListenerFactory
{
public String getName()
{
return "integral-presto-event-listener";
}

public EventListener create(Map<String, String> config)
{
PrestoEventListener eventListener = new PrestoEventListener(config);
return new ClassLoaderSafeEventListener(eventListener, getClassLoader());
}

private static ClassLoader getClassLoader()
{
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
if (classLoader == null) {
classLoader = IntegralPlugin.class.getClassLoader();
}
return classLoader;
}
}

The ClassLoaderSafeEventListener simply wraps all methods in a try-with-resources just like the ClassLoaderSafeConnectorMetadata class.

From the queryCompletedEvent implementation in the delegate, PrestoEventListener eventListener, I simply call another class's function which will handle a given query based on its presto.sql.tree type (Query, CreateTable, CreateTableAsSelect, etc.) after parsing.

From that handler I'm testing connectivity with an AtlasClientV2 instance with the following code:
AtlasEntity inputTable = atlasClientV2.getEntityByAttribute(AppConstants.TABLE_TYPE, attributeMap).getEntity();

[Dain Sundstrom]

> On Nov 8, 2018, at 1:55 PM, gray <gr...@eventbrite.com> wrote:
>
> public class PrestoEventListenerFactory
> implements EventListenerFactory
> {
> public String getName()
> {
> return "integral-presto-event-listener";
> }
>
> public EventListener create(Map<String, String> config)
> {
> PrestoEventListener eventListener = new PrestoEventListener(config);
> return new ClassLoaderSafeEventListener(eventListener, getClassLoader());
> }
>
> private static ClassLoader getClassLoader()
> {
> ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
> if (classLoader == null) {
> classLoader = IntegralPlugin.class.getClassLoader();
> }
> return classLoader;
> }
> }

I’m pretty sure you are selecting the wrong class loader. Since, Presto is not currently setting the thread context class loader when calling `EventListenerFactory.create`, you should not be using that. Instead, you should either capture the thread context class loader when the plugin is created, or take the much simpler approach and always choose `PrestoEventListenerFactory.class.getClassLoader()`.

-dain

[gray]

Thanks Dain. That appears to be the case. It's working locally now... just have to see if it will work once deployed.

David -- is this something I should submit a PR for? Even if its technically a bug in the plugin's library, it could help some other people work around those issues. Thoughts?

[gr...@eventbrite.com]

Same to you Dain -- submit the PR with just this change in setConfiguredEventListener?