copy from one datastore into another.

[Detlef Brendle]

Hi ,
I've got a question about copying data from one datastore(datawindow) to
another.

I am looking for the most performant and least memory use - version.

the test dataObject consists of 1700 rows and 17 columns.

The two datawindows(datastores) does not have the same dataObject. So I
first tried to copy each field from the source to the destination. the
code is show below.

FOR i=1 TO lds_dstore_source.rowcount()
FOR j=1 TO integer(lds_cust_dstore.Object.DataWindow.Column.Count)
lds_dstore_dest.Object.Data.Primary[i,j] =
ds_dstore_source.Object.data.primary[i,j]
NEXT
NEXT

the result is that it takes about 51 seconds to copy all fields to its
destination and about 18Mb of ram. !!! thats ways over the best case
!!!!

so the next attempt was to copy it row after row (just to see how this
will work.)

FOR i=1 TO lds_dstore_source.rowcount()
lds_dstore_source.rowscopy(i,i,Primary!,lds_dstore_dest,0,Primary!)
NEXT

this takes about 0.5 of a second and only 600kb of ram.
Can somebody explain this to me .?
The other thing is :
I need to use the first solution( because I dont want to copy each row
into another (because the dataObject of source and dest. is not the
same!!!).
The strange thing is that I cant reallocate this huge amount of memory
(18 Mb)
In the second try I destroy both datastores after they have been filled
and saved back. This mem is reallocate .....hm.... does anyone have the
key to this problem. ?

detlef

[Anne Pruitt]

From the help on Datawindow data expressions, subtopic "a block of rows
and columns"

Description

You can access data in a range of rows and columns by specifying the
starting and ending row and column numbers.

Syntax

dwcontrol.Object.Data {.buffer } {.datasource } [ startrownum,
startcolnum, endrownum, endcolnum ]

[Simon Caldwell]

How about

ll_max = lds_dstore_source.rowcount()
li_count = integer(lds_cust_dstore.Object.DataWindow.Column.Count)
lds_dstore_dest.Object.Data.Primary[1,1, ll_max, li_count] =
ds_dstore_source.Object.Data.Primary[1,1, ll_max, li_count]

HTH

Simon

Detlef Brendle wrote in message <377F7DBB...@fh-konstanz.de>...

[Detlef Brendle]

good advice, but the source and the destination ds arent form the same
dataObject.
So some destination columns may be different to the source columns.

detlef

[Simon Caldwell]

I think I'm missing something here. I was going to say that my code
achieved exactly the same as yours. But then I noticed that you are copying
from dstore_source to dstore_dest, but getting the column count from
cust_dstore. Is this right?

Simon

Detlef Brendle wrote in message <3780B891...@fh-konstanz.de>...

[Detlef Brendle]

hi Simon,

you are right - I should be dstore_source instead cust_dstore.
that was my fault .sorry - but the problem is the same...
detlef

[Ken Drendel]

Why don't you, take the seelct statmant from the datastore (source)
and instead of putting it into a datastore or datawindow, put it into a
temp table.
But change the select statment to put the coulmns in the order the
destanation table wants.
Then just do a

insert into destanation_table values (col_1,col_2) (select col_1,col_2
from source_table);
?

[Simon Caldwell]

In that case, my code appears to be doing exactly the same as yours - it
finds the number of columns in the source, and copies these columns into the
first columns of the dest. You could even simplify it further:

ll_max = lds_dstore_source.rowcount()
li_count = integer(lds_cust_dstore.Object.DataWindow.Column.Count)

lds_dstore_dest.Object.Data = ds_dstore_source.Object.Data[1,1, ll_max,
li_count]

Or was your code an example only, is your actual requirement different?

Simon

Detlef Brendle wrote in message <3780DD8E...@fh-konstanz.de>...

>hi Simon,
>
>you are right - I should be dstore_source instead cust_dstore.
>that was my fault .sorry - but the problem is the same...
>detlef
>
>Simon Caldwell wrote:
>
>> I think I'm missing something here. I was going to say that my code
>> achieved exactly the same as yours. But then I noticed that you are
copying
>> from dstore_source to dstore_dest, but getting the column count from
>> cust_dstore. Is this right?
>>

>> Simon
>>
>> Detlef Brendle wrote in message <3780B891...@fh-konstanz.de>...

[Detlef Brendle]

ok Simon, you are right , but thats only a demo. The more important thing for me
is - why is the other syntax (with copy each field to the dest field) that slow
and that ram-killer.
copy from one ds to another it is neccessary to have the same dataobject. But I
need to convert (maybe) some columns (from string to integer). So it will crash
when I do it straight.
And when I want to copy the whole column I cant use the convertion anymore.Its
not working or I couldnt figure it out.
the syntax is now with field copy

adw_dest.Object.Data.Primary[ai_row_dest,ai_col_dest] =
string(adw_src.Object.data.primary[ai_row_src,ai_col_src])

and If I use columns copy

adw_dest.Object.Data.Primary[ai_row_dest,ai_col_dest,ai_maxrow,ai_maxcolumn] =
string(adw_src.Object.data.primary[ai_row_src,ai_col_src,ai_maxrow,ai_maxcolumn])

its not working because its not a single field to convert anymore.
Can I do it anyway_?

detlef

[Joe Parra]

Do you have the same number of columns, regardless of column type? If so,
how about saving as a text file from source, then using importfile into the
other datastore. This should implicitly handle conversions.
The other question is what is your ultimate goal? If you're trying to
handle a database conversion, a pipeline would be much easier.

HTH,
Joe Parra