Invalid cross-device link

[Mario Giuffrida]

Hello CDAP team,

We have added our customized AuthZ Extension code (our Custom Authorizer) in the cdap `security-extensions` folder. On building the CDAP image, our Custom Authorizer jar is copied correctly into `/opt/cdap/master/ext/security/` inside the image - as expected.

When referencing the jar (from cdap-site.xml) to inject the Authorizer we face this issue:

---

2021-05-05 15:51:54,777 - ERROR [main:i.c.c.c.r.DaemonMain@41] - Exception raised when calling init
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: io.cdap.cdap.security.authorization.InvalidAuthorizerException: Failed to load authorization extension from /opt/cdap/master/ext/security/dop-cdap-security-authz-extension-1.0-SNAPSHOT.jar.
  at io.cdap.cdap.security.authorization.DelegatingPrivilegeManager.<init>(DelegatingPrivilegeManager.java:39)
  [...]

Caused by: io.cdap.cdap.security.authorization.InvalidAuthorizerException: Failed to load authorization extension from /opt/cdap/master/ext/security/dop-cdap-security-authz-extension-1.0-SNAPSHOT.jar.
at io.cdap.cdap.security.authorization.AuthorizerInstantiator.createAuthorizerClassLoader(AuthorizerInstantiator.java:212) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.security.authorization.AuthorizerInstantiator.get(AuthorizerInstantiator.java:138) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
... 44 common frames omitted
Caused by: java.nio.file.FileSystemException: /data/tmp/1620229914536-0/artifact2069794942698759533.jar -> /opt/cdap/master/ext/security/dop-cdap-security-authz-extension-1.0-SNAPSHOT.jar: Invalid cross-device link

---

Have you seen this issue before?

Thanks

--

Mario Giuffrida

LiveRamp | Engineering

[Terence Yim]

Hi Mario,

Do you have the full stacktrace of the error?

Terence

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAEOf%3DiJDUFGTyhLNbaNRsPYoyXhw9Y7J-Eoqmipz3XNvSzV2LA%40mail.gmail.com.

--

Terence Yim | Staff Software Engineer | tere...@google.com | 

[Terence Yim]

Hi Mario,

Nevermind, I think it is caused by a bug introduced in 6.4. See https://cdap.atlassian.net/browse/CDAP-17963

Terence

[Mario Giuffrida]

Hi Terence,

Thank you, this has fixed the `Invalid cross-device link` issue using 6.5.0-SNAPSHOT cdap core image.

We are now facing another issue on the same `appfabric` pod. I am posting the stack trace here but please let me know if you want me to open a new thread.

Thank you

M

////

2021-05-07 14:00:35,795 - ERROR [main:i.c.c.c.r.DaemonMain@41] - Exception raised when calling init

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.IllegalStateException: no 'this' pointer within static method

  at io.cdap.cdap.security.authorization.DelegatingPrivilegeManager.<init>(DelegatingPrivilegeManager.java:39)

  while locating io.cdap.cdap.security.authorization.DelegatingPrivilegeManager
  while locating io.cdap.cdap.security.spi.authorization.PrivilegesManager
    for parameter 0 at io.cdap.cdap.gateway.handlers.AuthorizationHandler.<init>(AuthorizationHandler.java:84)
  while locating io.cdap.cdap.gateway.handlers.AuthorizationHandler
  while locating io.cdap.http.HttpHandler annotated with @com.google.inject.multibindings.Element(setName=@com.google.inject.name.Named(value=appfabric.http.handler),uniqueId=44, type=MULTIBINDER, keyType=)
  at io.cdap.cdap.app.guice.MonitorHandlerModule.configure(MonitorHandlerModule.java:94) (via modules: io.cdap.cdap.app.guice.MonitorHandlerModule -> com.google.inject.multibindings.Multibinder$RealMultibinder)
  while locating java.util.Set<io.cdap.http.HttpHandler> annotated with @com.google.inject.name.Named(value=appfabric.http.handler)
    for parameter 4 at io.cdap.cdap.internal.app.services.AppFabricServer.<init>(AppFabricServer.java:109)
  while locating io.cdap.cdap.internal.app.services.AppFabricServer

1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051) ~[com.google.inject.guice-4.0.jar:na]
at io.cdap.cdap.master.environment.k8s.AppFabricServiceMain.addServices(AppFabricServiceMain.java:143) [io.cdap.cdap.cdap-master-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.master.environment.k8s.AbstractServiceMain.init(AbstractServiceMain.java:184) [io.cdap.cdap.cdap-app-fabric-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.common.runtime.DaemonMain.doMain(DaemonMain.java:39) ~[io.cdap.cdap.cdap-common-6.5.0-SNAPSHOT.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
at io.cdap.cdap.master.environment.k8s.AbstractServiceMain.main(AbstractServiceMain.java:121) [io.cdap.cdap.cdap-app-fabric-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.master.environment.k8s.AppFabricServiceMain.main(AppFabricServiceMain.java:80) [io.cdap.cdap.cdap-master-6.5.0-SNAPSHOT.jar:na]
Caused by: java.lang.IllegalStateException: no 'this' pointer within static method
at org.objectweb.asm.commons.GeneratorAdapter.loadThis(GeneratorAdapter.java:497) ~[org.ow2.asm.asm-commons-7.1.jar:7.1]
at io.cdap.cdap.security.authorization.AuthorizerClassLoader$2$1.onMethodEnter(AuthorizerClassLoader.java:229) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at org.objectweb.asm.commons.AdviceAdapter.visitCode(AdviceAdapter.java:127) ~[org.ow2.asm.asm-commons-7.1.jar:7.1]
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1282) ~[org.ow2.asm.asm-7.1.jar:7.1]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:688) ~[org.ow2.asm.asm-7.1.jar:7.1]
at org.objectweb.asm.ClassReader.accept(ClassReader.java:400) ~[org.ow2.asm.asm-7.1.jar:7.1]
at io.cdap.cdap.security.authorization.AuthorizerClassLoader.rewriteClass(AuthorizerClassLoader.java:148) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:54) ~[io.cdap.cdap.cdap-common-6.5.0-SNAPSHOT.jar:na]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_292]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_292]
at io.cdap.cdap.security.authorization.AuthorizerInstantiator.loadAuthorizerClass(AuthorizerInstantiator.java:223) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.security.authorization.AuthorizerInstantiator.createAuthorizer(AuthorizerInstantiator.java:154) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.security.authorization.AuthorizerInstantiator.get(AuthorizerInstantiator.java:139) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at io.cdap.cdap.security.authorization.DelegatingPrivilegeManager.<init>(DelegatingPrivilegeManager.java:40) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_292]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_292]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_292]
at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:86) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ExposedKeyFactory.get(ExposedKeyFactory.java:54) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:375) ~[com.google.inject.extensions.guice-multibindings-4.0.jar:na]
at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:258) ~[com.google.inject.extensions.guice-multibindings-4.0.jar:na]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[com.google.inject.guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) ~[com.google.inject.guice-4.0.jar:na]
... 10 common frames omitted
2021-05-07 14:00:35,796 - DEBUG [main:i.c.c.k.c.AbstractWatcherThread@196] - Stop watching for kubernetes of resource type class io.kubernetes.client.models.V1Service
2021-05-07 14:00:35,798 - INFO  [main:i.c.c.m.e.k.KubeMasterEnvironment@164] - Kubernetes environment destroyed
2021-05-07 14:00:35,800 - INFO  [kube-discovery-service:i.c.c.k.c.AbstractWatcherThread@191] - Stop watching for changes in kubernetes resource type class io.kubernetes.client.models.V1Service

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAFmsUY0c_5KgWFDU5oTvrpUkGBG_sVy7RzvEnXP%2Bgkmc6Mus9g%40mail.gmail.com.

[Terence Yim]

Hi,

It was due to a bug in the Authorizer class rewrite logic introduced by https://cdap.atlassian.net/browse/CDAP-17888 . The fix is in https://github.com/cdapio/cdap/pull/13370

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAEOf%3Di%2BR%2B0EUGVYktTnGyb%3DrhG4naUtydztwTYQAGJxKQBvyVw%40mail.gmail.com.

[Kushil Dodhia]

Hi Terence, 

Thanks for the quick fix again ! We have now progressed to a different error in the AppFabric for the same issue. Posting the stack trace below, let me know if you require more information or a separate thread. 

I believe it may be related to some injection via RX.

Thank you again for the support, 

 

Kush

2021-05-10 17:33:14,115 - INFO  [main:i.c.c.s.d.s.j.DataSourceProvider@155] - Successfully loaded org.postgresql.Driver from /opt/cdap/master/ext/jdbc
2021-05-10 17:33:14,120 - INFO  [main:i.c.c.s.d.s.j.DataSourceProvider@93] - Creating the DataSource with jdbc url: jdbc:postgresql://cdap-sqlproxy:5432/cdap
2021-05-10 17:33:14,223 - DEBUG [main:i.c.c.s.d.c.CachedStructuredTableRegistry@46] - Enabling cache for structured table registry lookups.
2021-05-10 17:33:14,311 - DEBUG [main:i.c.c.d.d.InMemoryDatasetFramework@134] - Adding Default module io.cdap.cdap.data2.dataset2.lib.kv.InMemoryKVTableDefinition$Module to system namespace
2021-05-10 17:33:14,409 - WARN  [main:i.c.c.e.c.ExploreFacade@55] - Explore functionality for datasets is disabled. All calls to enable explore will be no-ops
2021-05-10 17:33:14,499 - INFO  [main:i.c.c.s.a.AuthorizerInstantiator@200] - Creating authorization extension using jar /opt/cdap/master/ext/security/dop-cdap-security-authz-extension-1.0-SNAPSHOT.jar.
2021-05-10 17:33:16,565 - ERROR [main:i.c.c.c.r.DaemonMain@41] - Exception raised when calling init
com.google.inject.ProvisionException: Unable to provision, see the following errors:1) Error injecting constructor, java.lang.NoClassDefFoundError: javax/ws/rs/client/RxInvokerProvider

  at io.cdap.cdap.security.authorization.DelegatingPrivilegeManager.<init>(DelegatingPrivilegeManager.java:39)
  while locating io.cdap.cdap.security.authorization.DelegatingPrivilegeManager
  while locating io.cdap.cdap.security.spi.authorization.PrivilegesManager
    for parameter 0 at io.cdap.cdap.gateway.handlers.AuthorizationHandler.<init>(AuthorizationHandler.java:84)
  while locating io.cdap.cdap.gateway.handlers.AuthorizationHandler
  while locating io.cdap.http.HttpHandler annotated with @com.google.inject.multibindings.Element(setName=@com.google.inject.name.Named(value=appfabric.http.handler),uniqueId=44, type=MULTIBINDER, keyType=)
  at io.cdap.cdap.app.guice.MonitorHandlerModule.configure(MonitorHandlerModule.java:94) (via modules: io.cdap.cdap.app.guice.MonitorHandlerModule -> com.google.inject.multibindings.Multibinder$RealMultibinder)
  while locating java.util.Set<io.cdap.http.HttpHandler> annotated with @com.google.inject.name.Named(value=appfabric.http.handler)
    for parameter 4 at io.cdap.cdap.internal.app.services.AppFabricServer.<init>(AppFabricServer.java:109)

  while locating io.cdap.cdap.internal.app.services.AppFabricServer1 error

	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025) ~[com.google.inject.guice-4.0.jar:na]
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051) ~[com.google.inject.guice-4.0.jar:na]
	at io.cdap.cdap.master.environment.k8s.AppFabricServiceMain.addServices(AppFabricServiceMain.java:143) [io.cdap.cdap.cdap-master-6.5.0-SNAPSHOT.jar:na]
	at io.cdap.cdap.master.environment.k8s.AbstractServiceMain.init(AbstractServiceMain.java:184) [io.cdap.cdap.cdap-app-fabric-6.5.0-SNAPSHOT.jar:na]
	at io.cdap.cdap.common.runtime.DaemonMain.doMain(DaemonMain.java:39) ~[io.cdap.cdap.cdap-common-6.5.0-SNAPSHOT.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
	at io.cdap.cdap.master.environment.k8s.AbstractServiceMain.main(AbstractServiceMain.java:121) [io.cdap.cdap.cdap-app-fabric-6.5.0-SNAPSHOT.jar:na]
	at io.cdap.cdap.master.environment.k8s.AppFabricServiceMain.main(AppFabricServiceMain.java:80) [io.cdap.cdap.cdap-master-6.5.0-SNAPSHOT.jar:na]

Caused by: java.lang.NoClassDefFoundError: javax/ws/rs/client/RxInvokerProvider
	at org.glassfish.jersey.internal.inject.Providers.getJaxRsProviderInterfaces(Providers.java:117) ~[na:na]
	at org.glassfish.jersey.internal.inject.Providers.<clinit>(Providers.java:90) ~[na:na]
	at org.glassfish.jersey.model.internal.ComponentBag.modelFor(ComponentBag.java:518) ~[na:na]
	at org.glassfish.jersey.model.internal.ComponentBag.lambda$registerModel$10(ComponentBag.java:477) ~[na:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[na:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[na:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:229) ~[na:na]
	at org.glassfish.jersey.model.internal.ComponentBag.registerModel(ComponentBag.java:469) ~[na:na]
	at org.glassfish.jersey.model.internal.ComponentBag.register(ComponentBag.java:306) ~[na:na]
	at org.glassfish.jersey.model.internal.CommonConfig.register(CommonConfig.java:408) ~[na:na]
	at org.glassfish.jersey.client.ClientConfig$State.register(ClientConfig.java:235) ~[na:na]
	at org.glassfish.jersey.client.ClientConfig.register(ClientConfig.java:613) ~[na:na]
	at com.liveramp.asset_permissioning_service.v0.ApiClient.buildHttpClient(ApiClient.java:761) ~[na:na]
	at com.liveramp.asset_permissioning_service.v0.ApiClient.<init>(ApiClient.java:72) ~[na:na]
	at com.liveramp.asset_permissioning_service.v0.Configuration.<clinit>(Configuration.java:18) ~[na:na]
	at com.liveramp.asset_permissioning_service.v0.client.AssetGrantApi.<init>(AssetGrantApi.java:29) ~[na:na]
	at com.liveramp.cdap.security.auth.aps.ApsClient.<init>(ApsClient.java:22) ~[na:na]
	at com.liveramp.cdap.security.auth.DopAuthorizer.initialize(DopAuthorizer.java:51) ~[na:na]
	at io.cdap.cdap.security.authorization.AuthorizerInstantiator.createAuthorizer(AuthorizerInstantiator.java:170) ~[io.cdap.cdap.cdap-security-6.5.0-SNAPSHOT.jar:na]

Caused by: java.lang.ClassNotFoundException: javax.ws.rs.client.RxInvokerProvider
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_292]
	at io.cdap.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:44) ~[io.cdap.cdap.cdap-common-6.5.0-SNAPSHOT.jar:na]

	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_292]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_292]

	... 64 common frames omitted

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAFmsUY0nWcCRJh8gE-eDoUnFMpCDyN3RyvbY3hVThJWfBDB_4w%40mail.gmail.com.

[Terence Yim]

Hi,

Is the RxInvokerProvider class / jar file included in the authorizer extension jar? Can you share your jar file layout?

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/B765A723-CCE1-4E39-8837-DAAD826D459C%40liveramp.com.

[Kushil Dodhia]

Hi Terrence, 

Thanks for your email, we were indeed missing a dependency on our jar. We have solved the issue now thanks for your support !

Regards,

Kush

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAFmsUY28U%3D5WUHH76mRVDwn1z%2B--cdBoMCnJgOVb6%2ByVCKiP0g%40mail.gmail.com.