Spark job server refuses to start after enabling cassandra authenticating

[nilanjan sarkar]

Hi,

We are using DSE 5.0.1. Initially the spark job sever worked fine for us. But after we enabled authentication in cassandra, the spark job server just refuses to start.

Does anybody know where I need to specify the cassandra credentials in spark job server?

Thanks,
Nilanjan Sarkar

[Russell Spitzer]

The Spark Jobserver is running through dse spark-submit so the same configuration for any spark-submit can be used. For example you could do dse -u -p spark-jobserver ... if you wanted. 

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

Russell Spitzer
Software Engineer

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
http://spark-packages.org/package/datastax/spark-cassandra-connector

[nilanjan sarkar]

On Tuesday, 30 August 2016 21:21:05 UTC+4, Russell Spitzer wrote:
> The Spark Jobserver is running through dse spark-submit so the same configuration for any spark-submit can be used. For example you could do dse -u -p spark-jobserver ... if you wanted. 
>
>
>
> On Tue, Aug 30, 2016 at 12:28 AM nilanjan sarkar <nilanjan....@gmail.com> wrote:
> Hi,
>
>
>
> We are using DSE 5.0.1. Initially the spark job sever worked fine for us. But after we enabled authentication in cassandra, the spark job server just refuses to start.
>
>
>
> Does anybody know where I need to specify the cassandra credentials in spark job server?
>
>
>
> Thanks,
>
> Nilanjan Sarkar
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.
>
>
> --
>
>
>
>

> Russell Spitzer
> Software Engineer
>
>
>
>
>
>
>
> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
> http://spark-packages.org/package/datastax/spark-cassandra-connector

Russell you are a life saver :)

[Simone]

Hi all,

That works for me too. I then create a context on the jobserver with:
POST <jobserver_url>/contexts/MyContext?spark.cassandra.auth.username=<username>&spark.cassandra.auth.password=<password>.
which works as well.

When I try to start a job in that context, I get:

{
"status": "ERROR",
"ERROR": {
"message": "Authentication error on host /X.Y.Z.W:9042: Host /X.Y.Z.W:9042 requires authentication, but no authenticator found in Cluster configuration",
"errorClass": "com.datastax.driver.core.exceptions.AuthenticationException",
"stack": [
"com.datastax.driver.core.AuthProvider$1.newAuthenticator(AuthProvider.java:38)",
"com.datastax.driver.core.Connection$5.apply(Connection.java:251)",
"com.datastax.driver.core.Connection$5.apply(Connection.java:238)",
"com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)",
"com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)",
"com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)",
"com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)",
"com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)",
"com.datastax.driver.core.Connection$Future.onSet(Connection.java:1161)",
"com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:991)",
"com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:913)",
"io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)",
"io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)",
"io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)",
"io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)",
"io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)",
"io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)",
"io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)",
"io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)",
"io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)",
"io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)",
"io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)",
"io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)",
"io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293)",
"io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:840)",
"io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:830)",
"io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:348)",
"io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)",
"io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)",
"java.lang.Thread.run(Thread.java:745)"
]
}
}

I can authenticate just fine with cqlsh, dse spark and dse spark-submit, as well as accessing Cassandra data in Spark shell and regular Spark jobs, but I can't get it working with the jobserver.
There does not seem to be any reference either on DataStax or jobserver website about how to configure this, but looking at the code it seems that passing credentials to the context should be the way to go.
Any suggestions?

[Russell Spitzer]

I'm a little confused by your email, you talk about lots of things working but then not working. Could you specifically comment on 

How you are starting the jobserver?
How you are making the contexts?
What is the code of the job you are running?

--
[image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>

[Simone]

Sorry for the confusion, I will try to clarify.

1. Basic authentication is working, e.g. the following works
cqlsh -u <username> -p <password>
dse -u <username> -p <password>

2. Spark shell (i.e. dse spark) and regular Spark jobs (i.e. dse spark-submit) work, when adding the following parameters:
--conf spark.cassandra.auth.username=<username>
--conf spark.cassandra.auth.password=<password>

3. I am having problems running jobs in the jobserver (not starting the jobserver).

- I am starting the jobserver with: "dse -u <username> -p <password> spark-jobserver start". This works.

- I am creating the context via a regular call to the jobserver REST API to create the context, additionally passing the username and password. I assume this is how you are supposed to pass credentials to Spark jobs started with the jobserver:

POST <jobserver_url>/contexts/MyContext?spark.cassandra.auth.username=<username>&spark.cassandra.auth.password=<password>.

- I am then submitting a job via a regular call to the jobserver REST API (forgot to post this in my original message):
POST <jobserverUrl>/jobs?appName=<MyApp>&classPath=<myAppClassPath>&context=MyContext
When I do this, the job fails almost instantly with the exception posted in my original message (that you can see with a GET <jobserverUrl>/jobs/<jobId> as displayed in the jobserver UI).

[Russell Spitzer]

And what could is being run in the job? The trace seems to indicate a raw driver connection but it could be one of the internal SCC calls. Hard to tell

[Russell Spitzer]

Could-> Code

[Simone Franzini]

I think you are right.

The code is using some Spark driver stuff as well as a raw Cassandra connection that is currently missing a .withCredentials statements.

I will try adding that in. I assume that is likely going to resolve the issue.

Unfortunately the stack trace did not have any reference to application code, so it didn't help us debugging this one.

I will report back once I can test this (likely on Monday). Thanks for your help so far.

Could-> Code

> > > To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsub...@lists.datastax.com.

>
> > >
>
> > >
>
> > > --
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Russell Spitzer
>
> > > Software Engineer
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
>
> > > http://spark-packages.org/package/datastax/spark-cassandra-connector
>
> >
>
> > Russell you are a life saver :)
>
>
>
>
>
> --
>
> [image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsub...@lists.datastax.com.

--
[image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsub...@lists.datastax.com.

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsub...@lists.datastax.com.

--

Simone Franzini, PhD

Senior Platform Engineer
SpringCM

sfra...@springcm.com 

+1.312.881.2615

www.springcm.com

LinkedIn | Facebook | Twitter

[Russell Spitzer]

The "CassandraConnector" class can be used to create normal driver sessions from the passed Spark Conf if that is useful to you, see

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/1_connecting.md#connecting-manually-to-cassandra

Could-> Code

> > > To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

>
> > >
>
> > >
>
> > > --
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Russell Spitzer
>
> > > Software Engineer
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
>
> > > http://spark-packages.org/package/datastax/spark-cassandra-connector
>
> >
>
> > Russell you are a life saver :)
>
>
>
>
>
> --
>
> [image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--
[image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

Simone Franzini, PhD

Senior Platform Engineer
SpringCM

sfra...@springcm.com 

+1.312.881.2615

www.springcm.com

LinkedIn | Facebook | Twitter

--

You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

[nilanjan sarkar]

Hi,

I had a similar problem. Russell helped me out back then. As per his suggestion, I simply started the job server liker this :

$DSE_HOME/bin/dse -u <user> -p <pass> spark-jobserver start

This solved the issue for me. Try and see if this works for you.

Regards,

Nilanjan Sarkar

Could-> Code

> > > To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsubscrib...@lists.datastax.com.

>
> > >
>
> > >
>
> > > --
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Russell Spitzer
>
> > > Software Engineer
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
>
> > > http://spark-packages.org/package/datastax/spark-cassandra-connector
>
> >
>
> > Russell you are a life saver :)
>
>
>
>
>
> --
>
> [image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsubscrib...@lists.datastax.com.

--
[image: P7LdPGG7gGSnBkiXjbODbxbg8 (230×41)] <https://www.springcm.com/>

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsubscrib...@lists.datastax.com.

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-user+unsubscrib...@lists.datastax.com.

--

Simone Franzini, PhD

Senior Platform Engineer
SpringCM

sfra...@springcm.com 

+1.312.881.2615

www.springcm.com

LinkedIn | Facebook | Twitter

--