DB Solo3: Getting Java Heap Error when comparing more than 40-50 tables.

[Kuruba]

Hi,

I am using the trail version of DB Solo 3, where i am getting Java
Heap Error when comparing more than 40-50 tables.

DB Environments:
I am using Oracle10g and having more than 270 tables in each database.

Requirement:
We will be running the data migration from one database to other using
in house developed tool. Where after the data migration need to
compare the table datas to ensure the success of data migration.


My Questions:
1. Is this error getting because of trail version?
2. Is there any limitation on number of tables comparision?
3. Is there any plans to include sort of dash board reports?

Thanks,
Ramesh K

[DB Solo]

How many rows do you have in these tables? I would recommend dividing
your comparison into several projects, so that you will not run out of
memory.

The only limitation is the amount of memory available, there is no
limitations in the trial version.

Marko

[Kuruba]

Hi Marko,

Some of the table are having more than 1 lakh of records.

I have divided 270 tables and tried for 40-50 range of tables and
getting the Java Heap error. Please see the attachment for more
information.

I have few other queries like
1. Is Execution time tracking facility is available?
2. Is there any memory setting options?
3. During fields selection for the data compare (first time), its
taking more time. Once the tables structure are loaded, i think it
should not take so much of time during fields selection. Please
correct me if i am wrong.

Thanks,
Ramesh K

> >   Ramesh K- Hide quoted text -
>
> - Show quoted text -

[DB Solo Admin]

Do both sides (source & destination) contain most of the rows, i.e.
the number of different/missing/extra rows is fairly small?

If so, have you checked the 'Discard Matching Rows' checkbox on the
settings page? This will greatly reduce the amount of RAM required to
run the comparisons, since
rows that match on both sides will not be kept in memory.

Marko

[Kuruba]

Hi Marko,

Yes, most of the rows will be matching and around 50,000 rows will
be different and something around 20,000 might be some extra.

I tried using "Discard Matching Rows" check box, with that i am
finding some improvement in the performance, but the same Java Heap
Error is generating for more than 30-40 tables.

Other Issues:
1. Finding some discrepancies in the HTML report generation: Total
number of tables and the results are not reflected properly.
2. Java Heap space Error log

As there is no option for attachements, i will be sending the
attachements to mhan...@gmail.com from my gmail:
ramram...@gmail.com

Machine Details:
VM Ware Image: XP- Service Pack2 Version2002
1 GB RAM, 30GB Hard Disk

Thanks,
Ramesh K

On Apr 10, 7:12 am, DB Solo Admin <mhant...@gmail.com> wrote:
> Do both sides (source & destination) contain most of the rows, i.e.
> the number of different/missing/extra rows is fairly small?
>
> If so, have you checked the 'Discard Matching Rows' checkbox on the
> settings page? This will greatly reduce the amount of RAM required to
> run the comparisons, since
> rows that match on both sides will not be kept in memory.
>
> Marko
>
> On Apr 9, 1:30 am, Kuruba <ramramesh...@gmail.com> wrote:
>
>
>
> > Hi Marko,
>
> >   Some of the table are having more than 1 lakh of records.
>
> >   I have divided 270 tables and tried for 40-50 range of tables and

> > getting theJavaHeaperror. Please see the attachment for more

> > information.
>
> >  I have few other queries like
> > 1. Is Execution time tracking facility is available?
> > 2. Is there any memory setting options?
> > 3. During fields selection for the data compare (first time), its
> > taking more time. Once the tables structure are loaded, i think it
> > should not take so much of time during fields selection. Please
> > correct me if i am wrong.
>
> > Thanks,
> >  Ramesh K
>
> > On Apr 4, 4:31 pm, "DB Solo" <mhant...@gmail.com> wrote:
>
> > > How many rows do you have in these tables? I would recommend dividing
> > > your comparison into several projects, so that you will not run out of
> > > memory.
>
> > > The only limitation is the amount of memory available, there is no
> > > limitations in the trial version.
>
> > > Marko
>
> > > On Fri, Apr 4, 2008 at 4:28 AM, Kuruba <ramramesh...@gmail.com> wrote:
>
> > > >  Hi,
>
> > > >   I am using the trail version of DB Solo 3, where i am gettingJava

> > > >  HeapErrorwhen comparing more than 40-50 tables.

>
> > > >  DB Environments:
> > > >  I am using Oracle10g and having more than 270 tables in each database.
>
> > > >  Requirement:
> > > >  We will be running the data migration from one database to other using
> > > >  in house developed tool. Where after the data migration need to
> > > >  compare the table datas to ensure the success of data migration.
>
> > > >  My Questions:

> > > >  1. Is thiserrorgetting because of trail version?

> > > >  2. Is there any limitation on number of tables comparision?
> > > >  3. Is there any plans to include sort of dash board reports?
>
> > > >  Thanks,
> > > >   Ramesh K- Hide quoted text -
>

> > > - Show quoted text -- Hide quoted text -

[DB Solo]

You can increase the heap size by starting DB Solo as follows

dbsolo.exe -J-Xmx1024m

This will use 1024m for heap which is double the standard 512m. You
may even try a bit more than 1024m if you have enough system memory,
maybe 1250m.
Let me know if that helps.

Marko

[Kuruba]

Hi Marko,

The option you specified is working fine upto 1500m, i beilive it
can be increased depending on the size of RAM memory available in the
machine. Correct me if i am wrong.

To some extent i am able to split the number of tables into sets of
tables and trying to run.
Its working fine and for some tables still i am getting the Java Heap
Error. I just wonder how i can get rid off this error? please suggest
me tips if any.

Also i am able to generate the Summary report, where as some times
the sub reports will not be generated. Will you please look into the
issue?

Thanks,
Ramesh K

On Apr 15, 5:10 pm, "DB Solo" <mhant...@gmail.com> wrote:
> You can increase the heap size by starting DB Solo as follows
>
> dbsolo.exe -J-Xmx1024m
>
> This will use 1024m for heap which is double the standard 512m. You
> may even try a bit more than 1024m if you have enough system memory,
> maybe 1250m.
> Let me know if that helps.
>
> Marko
>
>
>
> On Tue, Apr 15, 2008 at 1:05 AM, Kuruba <ramramesh...@gmail.com> wrote:
>
> >  Hi Marko,
>
> >   Yes, most of the rows will be matching and around 50,000 rows will
> >  be different and something around 20,000 might be some extra.
>
> >    I tried using "Discard Matching Rows" check box, with that i am
> >  finding some improvement in the performance, but the same Java Heap
> >  Error is generating for more than 30-40 tables.
>
> >   Other Issues:

> >   1. Finding some discrepancies in the HTMLreportgeneration: Total

> >  number of tables and the results are not reflected properly.
> >   2. Java Heap space Error log
>
> >   As there is no option for attachements, i will be sending the

> >  attachements to mhant...@gmail.com from my gmail:
> >  ramramesh...@gmail.com

[DB Solo]

The maximum amount of RAM you can specify depends on the platform. I
think on Windows it's hard to go above 1500m no matter how much RAM
you have available. On MacOS you should be able
to allocate much more if your system has enough memory.

If you have one big table, you can split it into multiple comparisons
(and therefore decreasing the required RAM) by specifying a WHERE
statement for the table. E.g. the first part could be compared by
saying

WHERE PK BETWEEN 0 AND 10000

and the second comparison by saying

WHERE PK >= 10000

You need to specify the same WHERE statement in both text boxes (i.e.
also for the mapped side). Naturally you it doesn't need to be in two
parts, you can divide the comparison into as many parts as you want
to.

I have not seen a case where the sub-reports were not generated
(unless the row count for that sub report is zero), so it's hard to
figure what's going on without a concrete example.

Marko

[Marko Hantula]

You want to use the disk-based comparison for this as you will have 30m missing/extra rows in the results. You might also want to split it into multiple chunks by adding a where statement to each comparison.

On Mac you can make a copy of the commandline.sh script and add -Xmx4096m after 'java' in your new script. Change 4096 to whatever amount of memory your system can handle. You also want to remove these first two arguments from the script if you want to run the GUI

-Djava.awt.headless=true -DdbsoloCommandLine=true

Marko

Hi Marko,

I'm having the same issue on Mac OS trying to compare TableA_columnPhone (31million rows) against TableB_columnPhone (250.000 rows)

Both Tables share the same column structure, so my questions are: 

- How should I handle this comparison? 

- How can I increase the memory allocated to DB solo on macOS?

Thanks in advance,

--
You received this message because you are subscribed to the Google Groups "DB Solo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to db-solo+unsubscribe@googlegroups.com.
To post to this group, send email to db-...@googlegroups.com.
Visit this group at https://groups.google.com/group/db-solo.
For more options, visit https://groups.google.com/d/optout.

[Zafer]

Hi there,

I have been doing data comparison on Ubuntu 16.04 with DBSolo and Oracle Java 1.8 - for quite some time.  I always do check to ignore the "same" records. My max heap size is currently set to 8Gb and using "memory" option to compare. Had similar issues with lower max heap size like yours years ago.  Not having any issues with the latest version, with this heap size max setting - comparing ~ 10 million records. Note: on a Unix/Linux system you can edit the dbsolo script directly to edit the max heap size.

Cheers

Zaf.

-Djava.awt.headless=true -DdbsoloCommandLine=true

Marko

To unsubscribe from this group and stop receiving emails from it, send an email to db-solo+u...@googlegroups.com.