Jsch Exception (channel is not opened) while importing files from sftp server to hdfs

[Saurabh Paliwal]

Hi!
I am trying to import files from sftp server to hdfs. Everything was working fine when I used 0.6.2 version. Since I wanted better distcp conf and some bugs in state-store/watermarks, tried shifting to 0.8.0 but facing this issue.

While the datasets are found correctly when it starts, since I get such logs 

gobblin.data.management.retention.profile.ConfigurableGlobDatasetFinder  - Found dataset at <files>

But it can't open channel to create work unit.

STACKTRACE 

java.lang.RuntimeException: Failed to generate work units for dataset <some-file>

at gobblin.data.management.copy.CopySource$DatasetWorkUnitGenerator.call(CopySource.java:287)

at gobblin.data.management.copy.CopySource$DatasetWorkUnitGenerator.call(CopySource.java:233)

Caused by: java.io.IOException: 0: Cannot open a channel to SFTP server

at gobblin.source.extractor.extract.sftp.SftpLightWeightFileSystem.getFileStatus

Caused by: 0: Cannot open a channel to SFTP server

at gobblin.source.extractor.extract.sftp.SftpFsHelper.getExecChannel

Caused by: com.jcraft.jsch.JSchException: channel is not opened.

at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:768)

at com.jcraft.jsch.Channel.connect(Channel.java:151)

at com.jcraft.jsch.Channel.connect(Channel.java:145)

at gobblin.source.extractor.extract.sftp.SftpFsHelper.getExecChannel

Please help.

Thanks!

[Saurabh Paliwal]

and this is my .pull

job.name=Test

job.group=test

job.description=Copy from SFTP to HDFS

job.commit.policy=full

job.lock.enabled=true

source.class=gobblin.data.management.copy.CloseableFsCopySource

source.filebased.fs.uri=sftp:///

gobblin.dataset.pattern=/a/b/c/d/*.e

gobblin.retention.dataset.blacklist=/a/b/c/d/(black(.)*)

# this key is put on hdfs so that every mapper has it

source.conn.private.key=/etc/gobblin/test/test.pem

source.conn.username=root

source.conn.host=test.host

writer.fs.uri=hdfs://hadoop

writer.builder.class=gobblin.data.management.copy.writer.FileAwareInputStreamDataWriterBuilder

data.publisher.type=gobblin.data.management.copy.publisher.CopyDataPublisher

data.publisher.final.dir=/tmp/gobblin-test