Streaming API Error: 403::Handshake denied

[Justin Saul]

Hi,

 I have a very simple pipeline running that has an origin of Salesforce and a destination of ElasticSearch.  For the most part, it works very well. However, I noticed records missing and looking in the logs, I see this error occur every couple hours, requiring the stream to restart:

 com.streamsets.pipeline.api.StageException: FORCE_09 - Streaming API Error: 403::Handshake denied

  Version I'm running: 

StreamSets Data Collector 2.7.0.0 ( built by jenkins on 2017-08-18T18:09Z git: 791f327170ea7a43e4f92e935ad279ddfa4fbd18)

 I am not hitting any API limits on Salesforce.  Any thoughts as to how to resolve this error? 

Thanks.

[Jonathan Natkins]

Pat Patterson is the expert here, but a little bit of research seems to suggest that this might be the Salesforce API barfing a little bit and invalidating the authentication token. It seems like the Apache Camel community noted something similar: https://issues.apache.org/jira/browse/CAMEL-11980

Is there a full stack trace in the logs that you can share?

Jonathan "Natty" Natkins
StreamSets | Field Engineering Director

mobile: 609.577.1600 | linkedin

--
You received this message because you are subscribed to the Google Groups "sdc-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+unsubscribe@streamsets.com.
Visit this group at https://groups.google.com/a/streamsets.com/group/sdc-user/.

[Pat Patterson]

Hi Justin,

It does look similar to the issue in Apache Camel (nice find, Natty!). If you can pull the stack trace from sdc.log, that should give us a good idea of what exactly is happening.

One other piece of information that might be relevant - what is the session timeout in your Salesforce org?

Cheers,

Pat

--

Pat Patterson | Community Champion | http://about.me/patpatterson

[Justin Saul]

Thanks for the reply.

Session timeout is set at 2 hours. 

Trace:

ProductionPipeline - Error while running: com.streamsets.pipeline.api.StageException: FORCE_09 - Streaming API Error: 403::Handshake denied

com.streamsets.pipeline.api.StageException: FORCE_09 - Streaming API Error: 403::Handshake denied

at com.streamsets.pipeline.stage.origin.salesforce.ForceSource.processMetaMessage(ForceSource.java:671)

at com.streamsets.pipeline.stage.origin.salesforce.ForceSource.streamingProduce(ForceSource.java:786)

at com.streamsets.pipeline.stage.origin.salesforce.ForceSource.produce(ForceSource.java:332)

at com.streamsets.pipeline.configurablestage.DSource.produce(DSource.java:38)

at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:228)

at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:222)

at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)

at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:180)

at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:249)

at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:231)

at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:718)

at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:518)

at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:361)

at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:498)

at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)

at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:74)

at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:740)

at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:173)

at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:250)

at java.security.AccessController.doPrivileged(Native Method)

To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+u...@streamsets.com.

Visit this group at https://groups.google.com/a/streamsets.com/group/sdc-user/.

--
You received this message because you are subscribed to the Google Groups "sdc-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+u...@streamsets.com.

[Pat Patterson]

Hi Justin,

It looks exactly like the Camel issue. We expect, and handle, 401 errors, but, for some reason, the server is returning 403. I'll try to reproduce and add code to handle 403.

Looking back up the log before the stack trace, do you see any lines with Bayeux error message in them? I'm wondering if we're getting a 401, reconnecting, but the handshake is failing at that point.

Cheers,

Pat

--

Pat Patterson | Community Champion | http://about.me/patpatterson

To unsubscribe from this group and stop receiving emails from it, send an email to sdc-user+unsubscribe@streamsets.com.

[Justin Saul]

Three lines before 403 error:

INFO  ForceStreamConsumer - Bayeux client disconnected: true

INFO  ForceSource - Queue was empty at shutdown. No data lost.

INFO  AbstractRunner - Scheduling retry in '59991' milliseconds