Teradata table loading via CDAP pipeline

179 views
Skip to first unread message

nirav.l...@gmail.com

unread,
Feb 2, 2017, 7:14:11 PM2/2/17
to CDAP User


Hello,

we are trying to load csv file to teradata table via CDAP pipe but we are getting below error messages. could you please help us identify issue?

1. we have loaded teradata jdbc driver as external database plugin using teradata jar file "terajdbc4.jar" with plugin name teradatasql_terajdbc4  via CLI 

+======================================================================================================+
| type | name                  | classname                    | descripti | artifact                   |
|      |                       |                              | on        |                            |
+======================================================================================================+
| jdbc | teradatasql_terajdbc4 | com.teradata.jdbc.TeraDriver | Plugin fo | USER:terajdbc4-16.00.00.23 |
|      |                       |                              | r TERADAT |                            |
|      |                       |                              | A SQL JDB |                            |
|      |                       |                              | C driver  |                            |
|      |                       |                              | 16.00.00. |                            |
|      |                       |                              | 23        |                            |
+======================================================================================================+

+==========================================================================================================+
| type | name                    | classname                    | descripti | artifact                     |
|      |                         |                              | on        |                              |
+==========================================================================================================+
| jdbc | teradatasql_tdgssconfig | com.teradata.jdbc.TeraDriver | Plugin fo | USER:tdgssconfig-16.00.00.23 |
|      |                         |                              | r TERADAT |                              |
|      |                         |                              | A SQL JDB |                              |
|      |                         |                              | C driver  |                              |
|      |                         |                              | 16.00.00. |                              |
|      |                         |                              | 23        |                              |
+==========================================================================================================+

2. created pipeline which have source as file --> csv  parser --> sink as database

3. in database sink ,
we have defined connection string as

"jdbc:teradata://host.url.com/database=DATABASENAME,tmode=ANSI,charset=UTF8"

table name as tbl_nm and also provided user name and password.


Error Log: 


org.apache.tephra.TransactionFailureException: Failed to execute method initialize inside a transaction
	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:177) ~[co.cask.cdap.cdap-data-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.beforeSubmit(MapReduceRuntimeService.java:504) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:215) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:422) [co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.lang.ExceptionInInitializerError: null
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:577) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.<init>(GenericTeraEncrypt.java:116) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeradataConnection.<init>(GenericTeradataConnection.java:103) ~[na:na]
	at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:185) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:36) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169) ~[na:na]
	at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:235) ~[1486077273608-0/:na]
	at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:161) ~[1486077273608-0/:na]
	at co.cask.hydrator.plugin.JDBCDriverShim.connect(JDBCDriverShim.java:60) ~[1486077272352-0/:na]
	at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_66]
	at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_66]
	at co.cask.hydrator.plugin.DBManager.tableExists(DBManager.java:74) ~[1486077272352-0/:na]
	at co.cask.hydrator.plugin.db.batch.sink.DBSink.prepareRun(DBSink.java:104) ~[1486077272352-0/:na]
	at co.cask.hydrator.plugin.db.batch.sink.DBSink.prepareRun(DBSink.java:66) ~[1486077272352-0/:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:44) ~[cdap-etl-batch-3.5.0.jar:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:41) ~[cdap-etl-batch-3.5.0.jar:na]
	at co.cask.cdap.etl.log.LogContext.run(LogContext.java:59) ~[cdap-etl-core-3.5.0.jar:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable.prepareRun(LoggedBatchConfigurable.java:41) ~[cdap-etl-batch-3.5.0.jar:na]
	at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce.initialize(ETLMapReduce.java:206) ~[cdap-etl-batch-3.5.0.jar:na]
	at co.cask.cdap.api.mapreduce.AbstractMapReduce.initialize(AbstractMapReduce.java:171) ~[co.cask.cdap.cdap-api-3.5.0.jar:na]
	at co.cask.cdap.api.mapreduce.AbstractMapReduce.initialize(AbstractMapReduce.java:33) ~[co.cask.cdap.cdap-api-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:511) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:504) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:174) ~[co.cask.cdap.cdap-data-fabric-3.5.0.jar:na]
	... 5 common frames omitted
Caused by: java.lang.NullPointerException: null
	at com.teradata.tdgss.jtdgss.TdgssConfigApi.GetMechanisms(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<init>(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<clinit>(Unknown Source) ~[na:na]
	... 31 common frames omitted



java.lang.NullPointerException: null
	at com.teradata.tdgss.jtdgss.TdgssConfigApi.GetMechanisms(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<init>(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<clinit>(Unknown Source) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:577) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.<init>(GenericTeraEncrypt.java:116) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeradataConnection.<init>(GenericTeradataConnection.java:103) ~[na:na]
	at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:185) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:36) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169) ~[na:na]
	at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:235) ~[na:na]
	at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:161) ~[na:na]
	at co.cask.hydrator.plugin.JDBCDriverShim.connect(JDBCDriverShim.java:60) ~[na:na]
	at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_66]
	at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_66]
	at co.cask.hydrator.plugin.DBManager.tableExists(DBManager.java:74) ~[na:na]
	at co.cask.hydrator.plugin.db.batch.sink.DBSink.prepareRun(DBSink.java:104) ~[na:na]
	at co.cask.hydrator.plugin.db.batch.sink.DBSink.prepareRun(DBSink.java:66) ~[na:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:44) ~[na:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable$1.call(LoggedBatchConfigurable.java:41) ~[na:na]
	at co.cask.cdap.etl.log.LogContext.run(LogContext.java:59) ~[na:na]
	at co.cask.cdap.etl.batch.LoggedBatchConfigurable.prepareRun(LoggedBatchConfigurable.java:41) ~[na:na]
	at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce.initialize(ETLMapReduce.java:206) ~[na:na]
	at co.cask.cdap.api.mapreduce.AbstractMapReduce.initialize(AbstractMapReduce.java:171) ~[co.cask.cdap.cdap-api-3.5.0.jar:na]
	at co.cask.cdap.api.mapreduce.AbstractMapReduce.initialize(AbstractMapReduce.java:33) ~[co.cask.cdap.cdap-api-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:511) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:504) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:174) ~[co.cask.cdap.cdap-data-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.beforeSubmit(MapReduceRuntimeService.java:504) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:215) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:422) ~[co.cask.cdap.cdap-app-fabric-3.5.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

leo...@cask.co

unread,
Feb 2, 2017, 7:25:33 PM2/2/17
to CDAP User, nirav.l...@gmail.com
As far as I can see, JDBC can not establish connectivity to database:
Caused by: java.lang.ExceptionInInitializerError: null
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:577) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.<init>(GenericTeraEncrypt.java:116) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeradataConnection.<init>(GenericTeradataConnection.java:103) ~[na:na]
	at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:185) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:36) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169) ~[na:na]
	at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:235) ~[1486077273608-0/:na]
	at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:161) ~[1486077273608-0/:na]
and 

java.lang.NullPointerException: null
	at com.teradata.tdgss.jtdgss.TdgssConfigApi.GetMechanisms(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<init>(Unknown Source) ~[na:na]
	at com.teradata.tdgss.jtdgss.TdgssManager.<clinit>(Unknown Source) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:577) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeraEncrypt.<init>(GenericTeraEncrypt.java:116) ~[na:na]
	at com.teradata.jdbc.jdbc.GenericTeradataConnection.<init>(GenericTeradataConnection.java:103) ~[na:na]
	at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:185) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:36) ~[na:na]
	at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179) ~[na:na]
	at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169) ~[na:na]
	at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:235) ~[na:na]
	at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:161) ~[na:na]

Please, make sure you can communicate to database using this driver not from the CDAP.

peter....@gmail.com

unread,
Feb 3, 2017, 7:32:03 PM2/3/17
to CDAP User, nirav.l...@gmail.com
According to Teradata JDBC doc:
Your classpath must include:

  terajdbc4.jar
  tdgssconfig.jar

Not sure how you would specify both of these in the Database plugin since there is only space for one JDBC artifact

leo...@cask.co

unread,
Feb 15, 2017, 8:04:55 PM2/15/17
to CDAP User, nirav.l...@gmail.com, peter....@gmail.com
In addition to said, there is a way to use this type of rivers in CDAP.
In order to do this, uber JAR with both driver and configuration jars included needs to be created.
Here is the steps for creating it:

1. put both teradata jar files to the same directory.
2. Extract both jar files (jar -xvf xxx.jar) starting from configuration jar.
3. remove original JAR files from the folder
4. package all extracted data to single jar file (jat -cvf terajdbc4.jar META-INF co lib org). List all folders you have in the directory after jars extraction.

JSON configuration file should be same as usually used, something like this:

{ 
"parents": [ 
"system:cdap-etl-batch[3.5.0,3.6.0)", 
"system:cdap-etl-realtime[3.5.0,3.6.0)", 
"system:cdap-data-pipeline[3.5.0,3.6.0)" 
], 
"plugins": [ 
{ 
"name": "teradatasql_terajdbc4", 
"type": "jdbc", 
"className": "com.teradata.jdbc.TeraDriver", 
"description" : "Plugin for TERADATA SQL JDBC driver" 
} 
] 
}

Hope this help.

Leonid

leo...@cask.co

unread,
Feb 15, 2017, 8:18:05 PM2/15/17
to CDAP User, nirav.l...@gmail.com, peter....@gmail.com
Meant to say " this type of drivers "...
Reply all
Reply to author
Forward
0 new messages