Redshift issue

164 views
Skip to first unread message

jnkrish

unread,
Feb 23, 2016, 11:09:49 PM2/23/16
to cascading-user
Hello,

 We are trying our first POC for connecting to Redshift. I am running the below code on EMR and reading data from S3 to load to Redshift. I am getting the below error

 I have Postgresql JDBC jar and Redshift jars in my POM. 

NOt sure which driver i need to add as per the error message below. Please let me know how to resolve this.

POM

                                                           <dependency>
  <groupId>postgresql</groupId>
  <artifactId>jdbc.postgresql</artifactId>
  <version>9.3.1102</version>
  </dependency>
  
<dependency>
  <groupId>redshift</groupId>
  <artifactId>jdbc.connection</artifactId>
  <version>1.1.9</version>
  </dependency>




Java:

public static void main( String[] args ) {
       Properties properties = new Properties();
       String accessKey = "<>";
       String secretKey = "<>";
        
       AppProps.setApplicationJarClass(properties, App.class);
       Hadoop2MR1FlowConnector flowConnector = new Hadoop2MR1FlowConnector(   
                                             properties );
       String input = "s3://<>";
       Tap inTap = new Hfs( new TextDelimited( false, "," ), input);
       Pipe copyPipe = new Pipe( "copy" );
      // Tap outTap = new Hfs( new TextDelimited( false, ";" ),"/tmp/output");
       
       
       AWSCredentials awsCredentials = new AWSCredentials( accessKey, secretKey );
       
       Fields rsFields = new Fields( "id", "val" ).applyTypes( String.class, String.class );
       // Redshift table descriptor
       RedshiftTableDesc redshiftTableDescriptor = new RedshiftTableDesc( "TEST", new String[]{"id", "val"}, new String[]{"varchar(100)", "varchar(100)"}, null, null );
       // instantiate RedshiftTap - setting useDirectInsert (COPY) to true
       Tap sinkTap = new RedshiftTap( "jdbc:redshift://cloudanal.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev", "<>", "<>", "/tmp/staging", awsCredentials, redshiftTableDescriptor, new RedshiftScheme( rsFields, redshiftTableDescriptor ), SinkMode.UPDATE , false, true );
        
       FlowDef flowDef = FlowDef.flowDef()
       .addSource( copyPipe, inTap )
       .addTailSink( copyPipe, sinkTap );
       flowConnector.connect( flowDef ).complete();
        

Error:

{"type":"MAP_ATTEMPT_FAILED","event":{"org.apache.hadoop.mapreduce.jobhistory.TaskAttemptUnsuccessfulCompletion":{"taskid":"task_1456259383224_0001_m_000000","taskType":"MAP","attemptId":"attempt_1456259383224_0001_m_000000_0","finishTime":1456260106329,"hostname":"ip-10-206-40-223.dqa.capitalone.com","port":8041,"rackname":"/default-rack","status":"FAILED","error":"Error: cascading.tap.TapException: No suitable driver found for jdbc:redshift://cloudanal.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev (SQL error code: 0) opening connection: jdbc:redshift://cloudanal.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev\n\tat jdbc.JDBCTap.createConnection(JDBCTap.java:428)\n\tat jdbc.JDBCTap.resourceExists(JDBCTap.java:540)\n\tat jdbc.JDBCTap.createResource(JDBCTap.java:492)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:145)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:24)\n\tat cascading.tap.Tap.createResource(Tap.java:428)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:119)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:24)\n\tat cascading.flow.stream.SinkStage.prepare(SinkStage.java:60)\n\tat cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:167)\n\tat cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:110)\n\tat org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:172)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)\nCaused by: java.sql.SQLException: No suitable driver found for jdbc:redshift://cloudanal.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev\n\tat java.sql.DriverManager.getConnection(DriverManager.java:596)\n\tat java.sql.DriverManager.getConnection(DriverManager.java:215)\n\tat jdbc.JDBCTap.createConnection(JDBCTap.java:410)\n\t... 17 more\n","counters":{"org.apache.hadoop.mapreduce.jobhistory.JhCounters":{"name":"COUNTERS","groups":[{"name":"org.apache.hadoop.mapreduce.FileSystemCounter","displayName":"File System Counters","counts":[{"name":"FILE_BYTES_READ","displayName":"FILE: Number of bytes read","value":0},{"name":"FILE_BYTES_WRITTEN","displayName":"FILE: Number of bytes written","value":130669},{"name":"FILE_READ_OPS","displayName":"FILE: Number of read operations","value":0},{"name":"FILE_LARGE_READ_OPS","displayName":"FILE: Number of large read operations","value":0},{"name":"FILE_WRITE_OPS","displayName":"FILE: Number of write operations","value":0},{"name":"HDFS_BYTES_READ","displayName":"HDFS: Number of bytes read","value":276},{"name":"HDFS_BYTES_WRITTEN","displayName":"HDFS: Number of bytes written","value":0},{"name":"HDFS_READ_OPS","displayName":"HDFS: Number of read operations","value":1},{"name":"HDFS_LARGE_READ_OPS","displayName":"HDFS: Number of large read operations","value":0},{"name":"HDFS_WRITE_OPS","displayName":"HDFS: Number of write operations","value":0},{"name":"S3_BYTES_READ","displayName":"S3: Number of bytes read","value":0},{"name":"S3_BYTES_WRITTEN","displayName":"S3: Number of bytes written","value":0},{"name":"S3_READ_OPS","displayName":"S3: Number of read operations","value":0},{"name":"S3_LARGE_READ_OPS","displayName":"S3: Number of large read operations","value":0},{"name":"S3_WRITE_OPS","displayName":"S3: Number of write operations","value":0}]},{"name":"org.apache.hadoop.mapreduce.TaskCounter","displayName":"Map-Reduce Framework","counts":[{"name":"MAP_INPUT_RECORDS","displayName":"Map input records","value":0},{"name":"MAP_OUTPUT_RECORDS","displayName":"Map output records","value":0},{"name":"SPLIT_RAW_BYTES","displayName":"Input split bytes","value":276},{"name":"SPILLED_RECORDS","displayName":"Spilled Records","value":0},{"name":"FAILED_SHUFFLE","displayName":"Failed Shuffles","value":0},{"name":"MERGED_MAP_OUTPUTS","displayName":"Merged Map outputs","value":0},{"name":"GC_TIME_MILLIS","displayName":"GC time elapsed (ms)","value":225},{"name":"CPU_MILLISECONDS","displayName":"CPU time spent (ms)","value":3920},{"name":"PHYSICAL_MEMORY_BYTES","displayName":"Physical memory (bytes) snapshot","value":281309184},{"name":"VIRTUAL_MEMORY_BYTES","displayName":"Virtual memory (bytes) snapshot","value":2055344128},{"name":"COMMITTED_HEAP_BYTES","displayName":"Total committed heap usage (bytes)","value":298844160}]},{"name":"cascading.flow.SliceCounters","displayName":"cascading.flow.SliceCounters","counts":[{"name":"Process_Begin_Time","displayName":"Process_Begin_Time","value":1456260106012}]},{"name":"org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter","displayName":"File Input Format Counters ","counts":[{"name":"BYTES_READ","displayName":"Bytes Read","value":0}]},{"name":"org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter","displayName":"File Output Format Counters ","counts":[{"name":"BYTES_WRITTEN","displayName":"Bytes Written","value":0}]}]}},"clockSplits":[13393,2,1,2,1,2,2,1,2,1,2,2],"cpuUsages":[3920,0,0,0,0,0,0,0,0,0,0,0],"vMemKbytes":[2007171,2007171,2007172,2007171,2007172,2007171,2007172,2007171,2007172,2007172,2007171,2007172],"physMemKbytes":[274716,274716,274716,274715,274716,274715,274716,274715,274716,274716,274715,274716]}}}

 



Ken Krugler

unread,
Feb 23, 2016, 11:13:11 PM2/23/16
to cascadi...@googlegroups.com
I assume you packaged up the dependent jars properly, i.e. they're in the job jar you're submitting to EMR (either as an uber jar, or in the lib subdir)

-- Ken

From: jnkrish

Sent: February 23, 2016 8:09:48pm PST

To: cascading-user

Subject: Redshift issue


--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/f31fdd2b-2c72-4283-bb4a-d196d3dc1012%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--------------------------
Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr







--------------------------
Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr





Andre Kelpe

unread,
Feb 24, 2016, 5:39:15 AM2/24/16
to cascading-user
If you use the postgresql driver, you have to use the postgresql
namespace in your JDBC connection. Replace jdbc:redshit with
jdbc:postgresql and it should work.

- André
> --
> You received this message because you are subscribed to the Google Groups
> "cascading-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cascading-use...@googlegroups.com.
> To post to this group, send email to cascadi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/cascading-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cascading-user/f31fdd2b-2c72-4283-bb4a-d196d3dc1012%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

jnkrish

unread,
Feb 24, 2016, 11:52:43 PM2/24/16
to cascading-user
Hi Andre,

  Thanks,yes this seems to help us move past the below issue. I am getting a new issue where no matter whether the table was available or dropped, i keep getting message such as Relation \"test123\" already exists and job fails. I tried with various table names, still same issue whether they existed or not.

 I also tried with various Sink mode options Replace, Update but no luck.

Can  you please let me know what the issue is and how to resolve? Thanks.


Error: cascading.flow.stream.DuctException: failed opening sink\n\tat cascading.flow.stream.SinkStage.prepare(SinkStage.java:70)\n\tat cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:167)\n\tat cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:110)\n\tat org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:172)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)\nCaused by: java.io.IOException: SQL error code: 0 executing update statement: CREATE TABLE TEST123 ( id varchar(100), val varchar(100) ) \n\tat jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:112)\n\tat jdbc.JDBCUtil.createTableIfNotExists(JDBCUtil.java:90)\n\tat jdbc.JDBCTap.createResource(JDBCTap.java:499)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:145)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:24)\n\tat cascading.tap.Tap.createResource(Tap.java:428)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:119)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:24)\n\tat cascading.flow.stream.SinkStage.prepare(SinkStage.java:60)\n\t... 9 more\nCaused by: org.postgresql.util.PSQLException: ERROR: Relation \"test123\" already exists\n\tat org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)\n\tat org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)\n\tat org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)\n\tat jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:105)\n\t... 17 more\n"

Andre Kelpe

unread,
Feb 25, 2016, 5:54:25 AM2/25/16
to cascading-user
Can you run the tests in cascading-jdbc against a redshift instance of
yours? I want to confirm that the code and your env are in a state
where things work.

Check out the project (https://github.com/cascading/cascading-jdbc)
and then run this (with your URL)

gradle build -Dcascading.jdbc.url.redshift='jdbc:postgresql://some-host/somedb?user=some-user&password=somepw'
-i

You will need gradle 2.x.

- André
> https://groups.google.com/d/msgid/cascading-user/76fe905f-1864-4c94-adc7-540a7ca35250%40googlegroups.com.

jnkrish

unread,
Mar 3, 2016, 12:50:01 AM3/3/16
to cascading-user
Hi Andre,

 We have some issues running gradle, we are using Maven for all our projects. Regardless, we dropped the table test123 manually using SQL workbench and when the job runs first it creates the table test123 and then failed saying ERROR: Relation \"test123\" already exists. Not sure why it is checking again for the presence of the table.

 We encountered the same issue in Teradata tap also. For resolving we modified JDBCUtil.tableExists to return "true" and it worked, but same does not work for Redshift.

Do you know of any any other possible issue that is causing this?

{"type":"TASK_FAILED","event":{"org.apache.hadoop.mapreduce.jobhistory.TaskFailed":{"taskid":"task_1456784336418_0003_m_000001","taskType":"MAP","finishTime":1456787411041,"error":", Error: cascading.flow.stream.DuctException: failed opening sink\n\tat cascading.flow.stream.SinkStage.prepare(SinkStage.java:70)\n\tat cascading.flow.stream.StreamGraph.prepare(StreamGraph.java:167)\n\tat cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:110)\n\tat org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:172)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)\nCaused by: java.io.IOException: SQL error code: 0 executing update statement: CREATE TABLE TEST123 ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id) \n\tat jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:115)\n\tat jdbc.JDBCUtil.createTableIfNotExists(JDBCUtil.java:93)\n\tat jdbc.JDBCTap.createResource(JDBCTap.java:499)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:145)\n\tat jdbc.RedshiftTap.createResource(RedshiftTap.java:24)\n\tat cascading.tap.Tap.createResource(Tap.java:428)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:119)\n\tat jdbc.RedshiftTap.openForWrite(RedshiftTap.java:24)\n\tat cascading.flow.stream.SinkStage.prepare(SinkStage.java:60)\n\t... 9 more\nCaused by: org.postgresql.util.PSQLException: ERROR: Relation \"test123\" already exists\n\tat org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)\n\tat org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)\n\tat org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)\n\tat org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)\n\tat jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:108)\n\t... 17 more\n","failedDueToAttempt":{"string":"attempt_1456784336418_0003_m_000001_3"},"status":"FAILED","counters":{"org.apache.hadoop.mapreduce.jobhistory.JhCounters":{"name":"COUNTERS","groups":[{"name":"org.apache.hadoop.mapreduce.TaskCounter","displayName":"Map-Reduce Framework","counts":[{"name":"CPU_MILLISECONDS","displayName":"CPU time spent (ms)","value":0},{"name":"PHYSICAL_MEMORY_BYTES","displayName":"Physical memory (bytes) snapshot","value":0},{"name":"VIRTUAL_MEMORY_BYTES","displayName":"Virtual memory (bytes) snapshot","value":0}]}]}}}}}

Andre Kelpe

unread,
Mar 3, 2016, 5:54:12 AM3/3/16
to cascading-user
Which version of Cascading and Cascading-JDBC are you using?

- André
> https://groups.google.com/d/msgid/cascading-user/57b22bc0-acd1-47e6-a8db-ee5b85417adf%40googlegroups.com.

jnkrish

unread,
Mar 4, 2016, 12:19:47 AM3/4/16
to cascading-user
Hi Andre,

Earlier i was using Cascading version = 2.5.1  and Cascading-JDBC=3.0.0.

Now i just updated Cascading to version =3.0.1 and Cascading-JDBC=3.0.0. and tried again. It gives me a new error

ERROR flow.Flow: [] unable to prepare tap for write: jdbc and original issue ERROR: Relation "test123" already exists

Can you please check below log and let me know.


16/03/04 05:10:31 INFO util.Version: Concurrent, Inc - Cascading 3.0.1
16/03/04 05:10:31 INFO flow.Flow: [] starting
16/03/04 05:10:31 INFO flow.Flow: []  source: Hfs["TextDelimited[['id', 'val' | String, String]]"]["s3://sherlock-holmes-test-data/my-log.csv"]
16/03/04 05:10:31 INFO flow.Flow: []  sink: RedshiftTap["RedshiftScheme[['id', 'val' | String, String]]"][->"s3://sherlock-holmes-test-data/tmp/edd3ec87-e939-4898-810a-1721bac6984e"->"jdbc://jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev&id=4700f1db-2c22-4b51-b76c-07c520ba1c7f"]
16/03/04 05:10:31 INFO flow.Flow: []  parallel execution of steps is enabled: true
16/03/04 05:10:31 INFO flow.Flow: []  executing total steps: 1
16/03/04 05:10:31 INFO flow.Flow: []  allocating management threads: 1
16/03/04 05:10:31 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/04 05:10:31 INFO jdbc.RedshiftTap: creating resources
16/03/04 05:10:31 INFO jdbc.RedshiftTap: creating DB table: jdbc://jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev&id=4700f1db-2c22-4b51-b76c-07c520ba1c7f
16/03/04 05:10:31 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/04 05:10:31 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/04 05:10:31 INFO jdbc.RedshiftTap: Creating table: CREATE TABLE TEST123 ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id)
16/03/04 05:10:31 INFO jdbc.JDBCUtil: executing update: CREATE TABLE TEST123 ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id)
16/03/04 05:10:31 ERROR flow.Flow: [] unable to prepare tap for write: jdbc://jdbc:postgresql://cloudanl.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev&id=4700f1db-2c22-4b51-b76c-07c520ba1c7f
java.io.IOException: SQL error code: 0 executing update statement: CREATE TABLE TEST123 ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id)
        at jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:114)
        at jdbc.JDBCUtil.createTableIfNotExists(JDBCUtil.java:92)
        at jdbc.JDBCTap.createResource(JDBCTap.java:500)
        at jdbc.RedshiftTap.createResource(RedshiftTap.java:145)
        at jdbc.JDBCTap.prepareResourceForWrite(JDBCTap.java:487)
        at jdbc.JDBCTap.prepareResourceForWrite(JDBCTap.java:79)
        at cascading.flow.planner.BaseFlowStep.prepareResource(BaseFlowStep.java:533)
        at cascading.flow.planner.BaseFlowStep.prepareResources(BaseFlowStep.java:515)
        at cascading.flow.planner.BaseFlowStep.prepareResources(BaseFlowStep.java:501)
        at cascading.flow.planner.FlowStepJob.prepareResources(FlowStepJob.java:200)
        at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:168)
        at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:134)
        at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:45)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: Relation "test123" already exists
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)
        at jdbc.JDBCUtil.executeUpdate(JDBCUtil.java:107)

Andre Kelpe

unread,
Mar 4, 2016, 5:05:33 AM3/4/16
to cascading-user
On Fri, Mar 4, 2016 at 6:19 AM, jnkrish <jnkr...@gmail.com> wrote:
> Hi Andre,
>
> Earlier i was using Cascading version = 2.5.1 and Cascading-JDBC=3.0.0.

That will not work. Cascading-JDBC 3.x requires Cascading 3.x to work correctly.

>
> Now i just updated Cascading to version =3.0.1 and Cascading-JDBC=3.0.0. and
> tried again. It gives me a new error
>
> ERROR flow.Flow: [] unable to prepare tap for write: jdbc and original issue
> ERROR: Relation "test123" already exists
>
> Can you please check below log and let me know.


Can you please try Cascading 3.0.4 with Cascading JDBC 3.0.0?
> https://groups.google.com/d/msgid/cascading-user/112762c2-5d37-4afc-a5ef-d9ba0780776d%40googlegroups.com.

jnkrish

unread,
Mar 8, 2016, 11:26:01 PM3/8/16
to cascading-user
I used Cascading 3.0.4 with Cascading JDBC 3.0.0. This time it doesnt give the error "relation already exists"  but still getting the issue   " unable to prepare tap for write".   Also i could see that it did create the table in the process which did not exist before running the job. So connection seems to be fine but not sure why it is unable to write after creating the table. 

Are there any more compatible versions? Is anything wrong in the code which is in my first mail chain?

 See below:

16/03/08 20:11:07 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://ferraridb1.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/08 20:11:07 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://ferraridb1.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/08 20:11:07 INFO jdbc.RedshiftTap: Creating table: CREATE TABLE Finl_Ferrari ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id)
16/03/08 20:11:07 INFO jdbc.JDBCUtil: executing update: CREATE TABLE Finl_Ferrari ( id varchar(30), val varchar(30) )  DISTKEY (id)  SORTKEY (id)
16/03/08 20:11:07 INFO jdbc.JDBCTap: creating connection: jdbc:postgresql://ferraridb1.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev
16/03/08 20:11:07 ERROR flow.Flow: [] unable to prepare tap for write: jdbc://jdbc:postgresql://ferraridb1.cbljilhgawom.us-east-1.redshift.amazonaws.com:8080/dev&id=f                       2ec03ee-c233-4b6b-89eb-1c3176bece4b


Thanks..

jnkrish

unread,
Mar 11, 2016, 7:45:21 AM3/11/16
to cascading-user
Hi Andre,

 Can you please let me know if you got chance to check this issue below. The versions suggested were not compatible or some other issue going on. I also checked with Core 3.0.3 still same issue.

Thanks
...

Andre Kelpe

unread,
Mar 11, 2016, 8:46:26 AM3/11/16
to cascading-user
I am really out of ideas what might be wrong. Does the user have
enough rights to interact with the database?

- André
> --
> You received this message because you are subscribed to the Google Groups
> "cascading-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cascading-use...@googlegroups.com.
> To post to this group, send email to cascadi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/cascading-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cascading-user/16868b0e-6964-452b-827c-92f27307f100%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages