Maple slow's down the computation when the stored amont of data grows in the memory !!!

[gpe...@gmail.com]

Can someone please explain me why is that and how to speed up my computations.
I tried successively to store the data in external files and call it back when is needed, but there is still large amount of the data stored and updated in the memory. It will be really painful to store it in files since I call it frequently. I think it's related to the GARBAGE COLLECTION but I am not sure.

[Axel Vogt]

On 11.08.2012 18:04, gpe...@gmail.com wrote:
> Can someone please explain me why is that and how to speed up my computations.
> I tried successively to store the data in external files and call it back when is needed, but there is still large amount of the data stored and updated in the memory. It will be really painful to store it in files since I call it frequently. I think it's related to the GARBAGE COLLECTION but I am not sure.

This is not very specific, you may say more (the essential
parts of your program). Usually Maple cares for that, but
not in all situations, of course. It does not try to use
minimal memory and speed may have other reasons.

For example if you use Vectors (or similar) they still may
be stored in your worksheet.

http://www.mapleprimes.com/recent/all is a place, where you
may ask as well and can upload your sheet.

[Dante]

> This is not very specific, you may say more (the essential
>
> parts of your program). Usually Maple cares for that, but
>
> not in all situations, of course. It does not try to use
>
> minimal memory and speed may have other reasons.
>
>
>
> For example if you use Vectors (or similar) they still may
>
> be stored in your worksheet.
>
>
>
> http://www.mapleprimes.com/recent/all is a place, where you
>
> may ask as well and can upload your sheet.

Check this profiling information as an example:

> LP2(3.5+7*I, 1000000);

> PrintProfiles(LP2);

LP2
LP2 := proc(XIn, NC)
local i, C11;
global A11;
|Calls Seconds Words|
PROC | 1 30.125 140395652|
1 | 1 0.031 1000312| A11 := Array(1 .. NC);
2 | 1 2.293 0| for i to NC do
3 |1000000 25.060 132240861| C11 := ln(1/2*XIn);
4 |1000000 2.741 7154479| A11(i) := C11
end do
end proc

> LP3(3.5+7*I, 1000000);
print(??); # input placeholder
> PrintProfiles(LP3);

LP3
LP3 := proc(XIn, NC)
local i, C11;
|Calls Seconds Words|
PROC | 1 22.063 132230867|
1 | 1 1.652 0| for i to NC do
2 |1000000 20.411 132230867| C11 := ln(1/2*XIn)
end do
end proc

The process LP2 calls the ln() function 1000000 times and the calculation time for it is 25.060 seconds.
On the other hand the process LP3 calls the ln() function 1000000 times and it does it for less time - 20.411 seconds.
In my original process after the ln() function I have several local variables calculated and several global arrays re-sized with new data so the difference in time with just calculating the ln() functions grows rappidly. I need to know why the calculation of the ln() function lacks behind.

[Dante]

Check out what happens when I apply two more arrays:

> PrintProfiles(LP2);

LP2
LP2 := proc(XIn, NC)

local A13, i, C11;
global A11, A12;
|Calls Seconds Words|
PROC | 1 46.828 156565473|
1 | 1 0.062 1000311| A11 := Array(1 .. NC);
2 | 1 0.063 1000294| A12 := Array(1 .. NC);
3 | 1 0.062 1000294| A13 := Array(1 .. NC);
4 | 1 4.762 0| for i to NC do
5 |1000000 32.927 132259600| C11 := ln(1/2*XIn);
6 |1000000 4.103 7144518| A11(i) := C11;
7 |1000000 2.723 7078484| A12(i) := C11;
8 |1000000 2.126 7081972| A13(i) := C11
end do
end proc

[Axel Vogt]

On 12.08.2012 18:45, Dante wrote:
..

I used (for both of course)
N:=1000000; Profile(LP2);
LP2(3.5+7*I, N);
PrintProfiles(LP2);
UnProfile(LP2);

Using that for LP3 first and then LP2 I get with Maple 15:

LP3
2 |1000000 14,315 132037945| C11 := ln(1/2*XIn)

LP2
3 |1000000 15,028 132025102| C11 := ln(1/2*XIn);

With Maple 16 it was similar. Though my difference in timings
for log is much smaller I can not explain it (it also exists,
if line #4 in LP2 is de-activated).

[Dante]

Thanks for the help Vogt...anyway I think it's closely related to the way Maple uses physical memory and the swapping between physical and virtual memory. I know that using physical memory leads to faster computations and I forced the calculations to hardware floats with evalhf(), the speed did increase slightly but there is still significant difference between computations with free and filled memory pool.

[Axel Vogt]

On 12.08.2012 21:30, Dante wrote:
> Thanks for the help Vogt...anyway I think it's closely related to the way Maple uses physical memory and the swapping between physical and virtual memory. I know that using physical memory leads to faster computations and I forced the calculations to hardware floats with evalhf(), the speed did increase slightly but there is still significant difference between computations with free and filled memory pool.
>

Yes, may be. At least it is related to create/modify Arrays.
And I am not sure that 'Profile' reports correctly (may be?).

Concerning your other post ( = more Arrays increases timings
in a linear way):

LP4:=proc(XIn, NC)
global A11, A12, A13;
A11 := Array(1 .. NC, n -> ln(1/2*XIn), datatype=complex[8]);
A12 := Array(1 .. NC, n -> A11[n], datatype=complex[8]);
A13 := Array(1 .. NC, n -> A11[n], datatype=complex[8]);
end proc;

For N:=100000 = 10^7 I get time ~ 15 sec for A11 and for the
subsequent A12, A13 (I avoided 'copy') I get 2.5 sec each.

Which does not increase linear in the number of Arrays.

[Dante]

Thanks again...can we help me with this process:

###### Define global variables
DGV:=proc()
global CI0P,CI1P,CK0P,CK1P,CK0G,CK1G;

CI0P:=Array([1,6.25,9.765625,6.781684,2.6490953,0.66227383,0.1149781,0.014665573,0.0014321849,0.11050809e-3,0.69067559e-5,0.35675392e-6,0.15484111e-7,0.57263725e-9,0.18260116e-10]);
CI1P:=Array([0.5,1.5625,1.6276042,0.8477105,0.26490953,0.055189486,0.82127211e-2,0.91659834e-3,0.79565828e-4,0.55254047e-5,0.31394345e-6,0.14864747e-7,0.59554274e-9,0.2045133e-10,0.60867054e-12]);
CK0P:=Array([-0.57721566,2.6424021,9.0115658,8.5185931,3.9898493,1.1299171,0.21532918,0.029560538,0.30657944e-2,0.24883689e-3,0.16242981e-4,0.87142913e-6,0.39112788e-7,0.14905278e-8,0.48833881e-10]);
CK1P:=Array([1,0.96519581,-26.280638,-44.329875,-29.269699,-10.636897,-2.468972,-0.39918196,-0.047620526,-0.43685559e-2,-0.31795287e-3,-0.18814687e-4,-0.92322279e-6,-0.38181087e-7,-0.13490886e-8]);
CK0G:=Array([1.2533141,-0.031332853,0.35249460e-2,-0.73436375e-3,0.2248989e-3,-0.91084054e-4,0.45921544e-4,-0.27716932e-4,0.19488468e-4,-0.15644909e-4,0.1411953e-4,-0.1415162e-4]);
CK1G:=Array([1.2533141,0.09399856,-0.587491e-2,0.10281093e-2,-0.28915573e-3,0.11132496e-3,-0.54270916e-4,0.31981075e-4,-0.2208693e-4,0.17485486e-4,-0.15605797e-4,0.15499393e-4]);

end proc:

DGV();

###### Generate the complex valued functions
GBIKF012:=proc(XIn)
global CI0P,CI1P,CK0P,CK1P,CK0G,CK1G;

X11:=evalhf(XIn/2);XAbs:=evalhf(abs(XIn));
XVal:=evalhf(XIn*XIn/25);
C11:=evalhf(ln(5/XAbs));C12:=ln(X11);
if XAbs<5 then
NST:=ceil(max(2,min(15,24.177/C11)));
FI0M:=evalhf(add(CI0P[i]*XVal^(i-1),i=2..NST));
FI1M:=evalhf(XIn*add(CI1P[i]*XVal^(i-1),i=2..NST));
FK0M:=evalhf(-FI0M*C12+add(CK0P[i]*XVal^(i-1),i=2..NST));
FK0:=evalhf(FK0M-C12+CK0P[1]);
FK1M:=evalhf((FI1M*XIn*C12+add(CK1P[i]*XVal^(i-1),i=2..NST))/XIn+(CK0P[1]+0.5)*X11);
FK1:=evalhf(FK1M+1/XIn+XIn*(C12-CK0P[1]-0.5)/2);
else
C13:=evalhf(exp(-XIn)/sqrt(XIn));
NST:=ceil(max(2,min(12,12.088/C11)));
FK0:=evalhf((CK0G[1]+add(CK0G[i]*XVal^(i-1),i=2..NST))*exp(-XIn)/sqrt(XIn));
FK0M:=evalhf(FK0+ln(XIn/2)-CK0P[1]);
FK1:=evalhf((CK1G[1]+add(CK1G[i]*XVal^(i-1),i=2..NST))*C13);
FK1M:=evalhf(FK1-1/XIn-XIn*(C12-CK0P[1]-0.5)/2) fi;

FK2M:=evalhf(FK0M+FK1M*2/XIn);FK2:=evalhf(FK0+FK1*2/XIn);
return Array([FK0,FK1,FK2]);
end proc:

I need to improve the process, because when I ran it 250000 times it took at about 160 seconds which is much...I'll be grateful to your help.
This is the actual problem...in contrast the same amount of calculation of the ln() alone takes at about 3 second.

[Axel Vogt]

On 13.08.2012 20:01, Dante wrote:
> Thanks again...can we help me with this process:

... (snipped)

Hm .. tough. Difficult to guess from code what can be improved, if ever.
Is it simulation? What is the range and type for XIn ? What is the range
for the resulting 'NST' ? May some compiled version is the answer - I do
not see that any symbolics of Maple is needed ... in that form it looks
like a purely numerical task (to be done in classical coding).

[Roman Pearce]

On Saturday, August 11, 2012 9:04:35 AM UTC-7, Dante wrote:
> Can someone please explain me why is that and how to speed up my computations.
>
> I tried successively to store the data in external files and call it back when is needed, but there is still large amount of the data stored and updated in the memory. It will be really painful to store it in files since I call it frequently. I think it's related to the GARBAGE COLLECTION but I am not sure.

The computation time in Maple grows with the amount of memory used because garbage collection scans that memory. If possible, use hardware datatypes. For example, instead of a generic matrix use double precision:

A := Matrix(1000,1000,datatype=float[8]);

If you use these hardware data structures, Maple won't scan them during garbage collection and you can get an even bigger speedup from compiled routines. Most routines in the LinearAlgebra package are compiled for double precision, and for your own routines you can try "option autocompile". Here is a simple example, but you can have loops and what not.

A := Matrix(1000,1000,rand(-100..100),datatype=float[8]);
# evaluate polynomial using horner form
f := proc(x::float) option autocompile; ((((-7*x+22)*x-55)*x-94)*x+87)*x-56; end proc:
# this call should compile f
f(2.0);
# this runs the compiled function
B := map(f,A);
# newer versions of Maple also
# have elementwise arithmetic
C := B *~ B;

[Dante]

> The computation time in Maple grows with the amount of memory used because garbage collection scans that memory. If possible, use hardware datatypes. For example, instead of a generic matrix use double precision:
>
>
>
> A := Matrix(1000,1000,datatype=float[8]);

Thanks Roman this really increased the computation time.
Can you tell me if garbage collection scans read only arrays?
I mean if some of the arrays cannot be generatetd as float[8] type if I make them read onle would garbage collection scan them?

[Roman Pearce]

> Thanks Roman this really increased the computation time.

I hope you mean decrease :)

> Can you tell me if garbage collection scans read only arrays?

Yes, it has to scan everything except hardware datatypes.