out of memory error

[jrc]

Hello all, just started using diffkit to see if it will be useful for
our company.

I'm running under windows xp against postgres 8.3.5

I've been able to get diffkit working but only against a small table
of 1000 rows.

When i try to compare 2 identical tables with ~200,000 rows each, I
get a java.lang.OutOfMemoryError: Java heap space error.

The tables are defined as follows...
CREATE TABLE lhs
(
id integer NOT NULL,
adclaim integer NOT NULL,
adnumber integer NOT NULL,
adevent integer,
adclaimed numeric(15,2),
adsmsapproved numeric(15,2),
adapproved numeric(15,2),
adpaid numeric(15,2),
adbalance numeric(15,2),
addatepaid date,
adprogram character varying(15),
adsize double precision,
aduom character varying(15),
adunits double precision,
adfree1 character varying(20),
adfree2 character varying(20),
adfree3 character varying(20),
adfree4 character varying(20),
adfree5 character varying(20),
adfree6 double precision,
adfree7 double precision,
adfree8 date,
adfree9 date,
admediaseq integer,
adinvoice character varying(20),
adinsertdate date,
addays double precision,
adspots double precision,
admediaid character varying(15),
admedianame character varying(40),
admediacode character varying(15),
admediazip character varying(10),
admediaproduction numeric(15,2),
admediaprinting numeric(15,2),
admediaapvpct double precision,
admediareimpct double precision,
admediaadjpct double precision,
admediacommentcode character varying(15),
admediainvoiceamount numeric(15,2),
admediaineligamount numeric(15,2),
adprodseq integer,
adproduct character varying(15),
adprodheight double precision,
adprodwidth double precision,
adprodsize double precision,
adprodapvpct double precision,
adprodreimpct double precision,
adprodadjpct double precision,
adprodpricepoint numeric(15,2),
adprodcommentcode character varying(15),
adslick character varying(15),
adpromotion character varying(22),
adfun1 double precision,
adfundedby character varying(20),
adfundedbyid integer,
adonhold character varying(3),
adunpaid numeric(15,2),
priorapvid integer,
CONSTRAINT pkey_ad PRIMARY KEY (id, adclaim, adnumber)
)
WITH (
OIDS=FALSE
);

[joe]

Hi,

Can you please post:

The plan file
The complete command line invocation you used; java -jar...
The complete java stack trace from when it bombs out

thanks

Joe

[joe]

Sorry, one other thing. Which JRE/JVM are you using?

On Nov 19, 2:03 pm, jrc <john.can...@gmail.com> wrote:

[jrc]

plan file
<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

<bean id="plan" class="org.diffkit.diff.conf.DKMagicPlan">
<property name="lhsDBTableName" value="lhs" />
<property name="rhsDBTableName" value="rhs" />
<property name="dbConnectionInfo"
ref="connectionInfo" />
<property name="keyColumnNames">
<list>
<value>id</value>
<value>adclaim</value>
<value>adnumber</value>
</list>
</property>
<property name="sinkFilePath" value="./test10.sink.diff" />
</bean>

</beans>

command line I'm using (i can't cut and paste from the command line so
I hope i didnt miss type it here)
java -jar ..\diffkit-app.jar -planfiles
test10.plan.xml,dbconnectioninfo.postgres.xml


and the result (pipped into a file so i know i didnt miss type)

DiffKit home->C:\Program Files\DiffKit
planfile(s)->test10.plan.xml
lhsSource->DKDBSource@3ef810[lhs,jdbc:postgresql://smsdev1:5432/
jrctest]
rhsSource->DKDBSource@100363[rhs,jdbc:postgresql://smsdev1:5432/
jrctest]
sink->DKFileSink@bcda2d[.\test10.sink.diff]
tableComparison-
>org.diffkit.diff.conf.DKAutomaticTableComparison@97d01f
ERROR[14:45:24.278]{main}(DKApplication.main:130)-null
java.lang.OutOfMemoryError: Java heap space
at
com.jdotsoft.jarloader.JarClassLoader.getJarBytes(JarClassLoader.java:
681) [diffkit-app.jar:na]
at
com.jdotsoft.jarloader.JarClassLoader.findJarClass(JarClassLoader.java:
344) [diffkit-app.jar:na]
at
com.jdotsoft.jarloader.JarClassLoader.loadClass(JarClassLoader.java:
551) [diffkit-app.jar:na]
at java.lang.ClassLoader.loadClass(Unknown Source) [na:1.6.0_22]
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:
1817) [diffkit-app.jar:na]
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:
257) [diffkit-app.jar:na]
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:
500) [diffkit-app.jar:na]
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:
374) [diffkit-app.jar:na]
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:
254) [diffkit-app.jar:na]
at org.diffkit.util.DKSqlUtil.executeQuery(DKSqlUtil.java:400)
[diffkit-app.jar:na]
at org.diffkit.diff.sns.DKDBSource.createResultSet(DKDBSource.java:
219) [diffkit-app.jar:na]
at org.diffkit.diff.sns.DKDBSource.open(DKDBSource.java:134) [diffkit-
app.jar:na]
at org.diffkit.diff.engine.DKContext.open(DKContext.java:73) [diffkit-
app.jar:na]
at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:63)
[diffkit-app.jar:na]
at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:56)
[diffkit-app.jar:na]
at org.diffkit.diff.conf.DKApplication.doDiff(DKApplication.java:191)
[diffkit-app.jar:na]
at org.diffkit.diff.conf.DKApplication.runPlan(DKApplication.java:
172) [diffkit-app.jar:na]
at org.diffkit.diff.conf.DKApplication.main(DKApplication.java:110)
[diffkit-app.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:
1.6.0_22]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [na:
1.6.0_22]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
[na:1.6.0_22]
at java.lang.reflect.Method.invoke(Unknown Source) [na:1.6.0_22]
at
com.jdotsoft.jarloader.JarClassLoader.invokeMain(JarClassLoader.java:
520) [diffkit-app.jar:na]
at org.diffkit.diff.conf.DKLauncher.main(DKLauncher.java:26) [diffkit-
app.jar:na]

[jrc]

not sure how to check the JRE/JVM

[joe]

java -version

[jrc]

java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) Client VM (build 17.1-b03, mixed mode, sharing)

[joe]

The Postgres JDBC adapter has default behavior that's a little
different from most other big name JDBC drivers. The issue is
described here:

http://jdbc.postgresql.org/documentation/83/query.html#query-with-cursor

Here's someone who encountered the problem and got the same error you
did:

http://archives.postgresql.org/pgsql-jdbc/2009-01/msg00043.php

If this is indeed your issue (very likely), the fix is quite simple.
I"ll reproduce the problem on my end and then put in a bug fix
tomorrow.

In the meantime, I'v entered the bug into the issue tracking system:

http://code.google.com/p/diffkit/issues/detail?id=50

cheers,

Joe

[jrc]

excellent! i look forward to trying again soon.

[joe]

I was able to reproduce the error. Fixed in 0.8.3, which was just
released,

cheers,

Joe