[Mifos-developer] Mifos BI ETL job failure

102 views

Skip to first unread message

Ramesh Nayak

unread,

Sep 8, 2011, 11:56:29 AM9/8/11

to John Woodlock, Mifos software development

Hi John,

I am using Mifos BI on ubuntu system, i am having problems running the ETL job,
etl_build_prod.sh . I also run the healthy scripts (mifosPreCheck and mifosUnsuccessfulETL)
both are fine and etl jobs are running successfully in my local ubuntu system.

When reading customer_attendance table error is coming.

I am pasting the error here, could you please help me to resolve this issue.

ERROR 08-09 21:19:06,146 - Read customer_attendance - Unexpected error :

ERROR 08-09 21:19:06,171 - Read customer_attendance - org.pentaho.di.core.exception.KettleDatabaseException:

Couldn't get row from result set

Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException

at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1934)

at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2433)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798)

at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316)

at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370)

at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360)

at com.mysql.jdbc.ResultSet.next(ResultSet.java:5970)

at org.pentaho.di.core.database.Database.getRow(Database.java:2714)

at org.pentaho.di.core.database.Database.getRow(Database.java:2699)

at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:140)

at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)

at java.lang.Thread.run(Thread.java:662)

** END NESTED EXCEPTION **

Last packet sent to the server was 572914 ms ago.

org.pentaho.di.core.database.Database.getRow(Database.java:2784)

org.pentaho.di.core.database.Database.getRow(Database.java:2699)

org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:140)

org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)

java.lang.Thread.run(Thread.java:662)

INFO 08-09 21:19:06,172 - Read customer_attendance - Finished reading query, closing connection.

ERROR 08-09 21:19:06,172 - Read customer_attendance - Unexpected error closing query : org.pentaho.di.core.exception.KettleDatabaseException:

Couldn't close query: resultset or prepared statements

Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException

at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1934)

at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2380)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798)

at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316)

at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370)

at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360)

at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)

at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6314)

at com.mysql.jdbc.ResultSet.close(ResultSet.java:749)

at org.pentaho.di.core.database.Database.closeQuery(Database.java:2327)

at org.pentaho.di.trans.steps.tableinput.TableInput.closePreviousQuery(TableInput.java:206)

at org.pentaho.di.trans.steps.tableinput.TableInput.dispose(TableInput.java:262)

at org.pentaho.di.trans.step.RunThread.run(RunThread.java:69)

at java.lang.Thread.run(Thread.java:662)

** END NESTED EXCEPTION **

Last packet sent to the server was 572940 ms ago.

ERROR 08-09 21:19:06,172 - SourceDB - Error disconnecting from database:

Error comitting connection

Streaming result set com.mysql.jdbc.RowDataDynamic@cc4419 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.

ERROR 08-09 21:19:06,172 - SourceDB - org.pentaho.di.core.exception.KettleDatabaseException:

Error comitting connection

org.pentaho.di.core.database.Database.commit(Database.java:686)

org.pentaho.di.core.database.Database.commit(Database.java:656)

org.pentaho.di.core.database.Database.disconnect(Database.java:547)

org.pentaho.di.trans.steps.tableinput.TableInput.dispose(TableInput.java:273)

org.pentaho.di.trans.step.RunThread.run(RunThread.java:69)

java.lang.Thread.run(Thread.java:662)

INFO 08-09 21:19:06,173 - Read customer_attendance - Finished processing (I=124578, O=0, R=0, W=124577, U=0, E=1)

INFO 08-09 21:19:06,173 - Load_ClientAttendances - Load_ClientAttendances

INFO 08-09 21:19:06,175 - Client lookup based on meeting date - Finished processing (I=104779, O=0, R=104779, W=104778, U=0, E=0)

ERROR 08-09 21:19:06,180 - Load_ClientAttendances - Errors detected!

INFO 08-09 21:19:06,181 - Calculate late Column - Finished processing (I=0, O=0, R=55166, W=55166, U=0, E=0)

INFO 08-09 21:19:06,181 - Calculate absent Column - Finished processing (I=0, O=0, R=74976, W=74976, U=0, E=0)

INFO 08-09 21:19:06,181 - Calculate present Column - Finished processing (I=0, O=0, R=84877, W=84877, U=0, E=0)

INFO 08-09 21:19:06,181 - Calculate approved_leave Column - Finished processing (I=0, O=0, R=65075, W=65075, U=0, E=0)

INFO 08-09 21:19:06,192 - Meeting Date Lookup - Finished processing (I=774, O=0, R=114679, W=114679, U=0, E=0)

INFO 08-09 21:19:06,222 - Create fact_client_attendances - Finished processing (I=0, O=45224, R=45224, W=45223, U=0, E=0)

INFO 08-09 21:19:06,230 - Client Found? - Finished processing (I=0, O=0, R=94878, W=94878, U=0, E=0)

INFO 08-09 21:19:11,400 - Read customer_schedule - linenr 250000

INFO 08-09 21:19:15,888 - Create dw_loan_schedules - Finished processing (I=0, O=215679, R=215679, W=215679, U=0, E=0)

INFO 08-09 21:19:23,139 - Due Date Lookup - linenr 250000

INFO 08-09 21:19:31,265 - Read customer_schedule - Finished reading query, closing connection.

INFO 08-09 21:19:31,265 - Read customer_schedule - Finished processing (I=266606, O=0, R=0, W=266606, U=0, E=0)

INFO 08-09 21:19:35,551 - Create dw_customer_schedules - linenr 250000

INFO 08-09 21:19:40,558 - Due Date Lookup - Finished processing (I=1589, O=0, R=266606, W=266606, U=0, E=0)

INFO 08-09 21:19:50,423 - Create dw_customer_schedules - Finished processing (I=0, O=266606, R=266606, W=266606, U=0, E=0)

INFO 08-09 21:19:50,454 - Load DW Trxn Tables - Finished job entry [No Schedules Audit Found] (result=[false])

INFO 08-09 21:19:50,454 - Load DW Trxn Tables - Finished job entry [Simple evaluation] (result=[false])

INFO 08-09 21:19:50,454 - Load DW Trxn Tables - Finished job entry [Check For Schedules Audit] (result=[false])

INFO 08-09 21:19:50,454 - Load DW Trxn Tables - Job execution finished

INFO 08-09 21:19:50,468 - Create Data Warehouse - Finished job entry [Load DW Trxn Tables] (result=[false])

INFO 08-09 21:19:50,468 - Create Data Warehouse - Finished job entry [Apply Loan Adjustments and Reversals] (result=[false])

INFO 08-09 21:19:50,468 - Create Data Warehouse - Finished job entry [Load Fact Tables] (result=[false])

INFO 08-09 21:19:50,469 - Create Data Warehouse - Finished job entry [Adjust Customer Status Values] (result=[false])

INFO 08-09 21:19:50,469 - Create Data Warehouse - Finished job entry [Load Customer And Loan And Savings Dimensions] (result=[false])

INFO 08-09 21:19:50,469 - Create Data Warehouse - Finished job entry [Load Staging Tables] (result=[false])

INFO 08-09 21:19:50,469 - Create Data Warehouse - Finished job entry [Load Personnel Dimension] (result=[false])

INFO 08-09 21:19:50,469 - Create Data Warehouse - Job execution finished

INFO 08-09 21:19:50,715 - Data Warehouse Initial Load - Finished job entry [Create Data Warehouse] (result=[false])

INFO 08-09 21:19:50,715 - Data Warehouse Initial Load - Finished job entry [Set Date Parameters for Initial Load] (result=[false])

INFO 08-09 21:19:50,715 - Data Warehouse Initial Load - Finished job entry [Log Start Entry for Initial Load] (result=[false])

INFO 08-09 21:19:50,715 - Data Warehouse Initial Load - Job execution finished

INFO 08-09 21:19:50,735 - Kitchen - Kitchen

ERROR 08-09 21:19:50,735 - Kitchen - Kitchen

INFO 08-09 21:19:50,735 - Kitchen - Kitchen

ETL Has Errors

--
Thanks and Regards,
Ramesh.A.Nayak| Software Engineer @ ConfluxTechnologies | Address :#412, 20th Main,Chord Road 1st Block Rajajinagar Bengaluru,Karnataka-560010 | Email :ramesh....@confluxtechnologies.com | website: WWW.Confluxtechnologies.com |
Contact Number - Mob : +919739317276.

John Woodlock

unread,

Sep 8, 2011, 1:29:05 PM9/8/11

to Ramesh Nayak, Mifos software development

Ramesh,

what BI version are you running?

I have seen this type of error before but can't remember if it was down to mysql settings or data based or something else or multiple reasons (so I'm not sure).

Is it repeatable and the same error every time? If so, you could just disable the hop to "Load fact_client_attendances" in "LoadDWTrxnTables.kjb" (using Spoon or directly in an editor) as its not that important.

John

Ramesh Nayak

unread,

Sep 8, 2011, 10:58:56 PM9/8/11

to John Woodlock, Mifos software development

Thanks John,

I am using data integration 4.0.0 and apache tomcat 7.0.12.

This error will come some time and some time running successfully.

Thanks and Regards

Ramesh.A.N

Vishwas Babu A J

unread,

Oct 27, 2011, 12:36:35 AM10/27/11

to Mifos software development

Hi Ramesh,

Increasing "net_write_timeout" for your MySQL server should solve the problem.

ref:http://wiki.pentaho.com/display/EAI/MySQL

Thanks and Regards,
Vishwas

------------------------------------------------------------------------------
Why Cloud-Based Security and Archiving Make Sense
Osterman Research conducted this study that outlines how and why cloud
computing security and archiving is rapidly being adopted across the IT 
space for its ease of implementation, lower cost, and increased 
reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/

Mifos-developer mailing list
mifos-d...@lists.sourceforge.net
Unsubscribe or change settings at:
https://lists.sourceforge.net/lists/listinfo/mifos-developer

Reply all

Reply to author

Forward

0 new messages