standard deviation and vDSP_normalize()

[Alexandru Ciobanu]

Hi everyone,

I am using vDSP_normalize() to compute the standard deviation and mean of an array of floats.

I am seeing that there are cases when the standard deviation returned by vDSP_normalize() is incorrect or nan even though the data is reasonable:
- several hundreds of values
- well behaved data (finite, non-constant)

I attach a small C program that reproduces the problem, along with two files with sample data.
(The program includes a naive stddev computation for comparison purposes.)

Here is how I run the program:

$ clang -framework Accelerate main.c

$ ./a.out < shorter.txt
num data points: 554
vdsp stddev = nan // INCORRECT
naive stddev = 0.4816624

$ ./a.out < longer.txt
num data points: 1178
vdsp stddev = 0.8304628 // INCORRECT
naive stddev = 0.4815856

The naive implementation produces the correct results even though it uses a float accumulator for the whole data.


Does anyone have an idea of what is the problem here?

Also I would be interested to know of alternative, better ways of computing the standard deviation and the mean in one pass, if possible.

sincerely,
Alex Ciobanu

Rogue Research Inc

[Stephen Canon]

Hi Alex —

The problem you’re running into is actually precisely that vDSP uses the single-pass algorithm for mean and standard deviation:

vDSP_sve_svesq(data, 1, &Sum, &SumSquares, numDataPoints);
mean = Sum / numDataPoints;
deviation = sqrt(SumSquares / numDataPoints - mean*mean);

Your data is sufficiently ill-conditioned that the single-pass algorithm cannot produce an accurate result. One good option is to handle this by trying to use the single pass algorithm as sketched above, and falling back onto the two-pass algorithm if the computed deviation is either NaN or very small compared to the average squared value (vDSP does not use this two-pass algorithm itself because one of the requirements of vDSP is that it deliver *predictable* performance whenever possible; the time required should not vary depending on the data).

– Steve
_______________________________________________
Do not post admin requests to the list. They will be ignored.
PerfOptimization-dev mailing list (PerfOptimi...@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/perfoptimization-dev/perfoptimization-dev-garchive-8409%40googlegroups.com

This email sent to perfoptimization-...@googlegroups.com