Nan being converted in mex files
Andreas Nielsen
From a mex file I am returning a double matrix. Since I have a need for returning 64 bit integers I am typecasting the integers into the doubles in the matrix. For integers being translated to NaNs some of them , but when they are transferred to matlab the bits have been changed. They are still NaNs but with other bit patterns, which makes it impossible to convert to the relevant 64 bit number.
So my question is if it is possible to turn off the "translation" of the doubles before returning the values. It is not possible for me to change the type of matrix I return, since it is only some of the rows that will be integers while others are doubles.
Thanks,
Andreas
Andreas Nielsen
A C mex function returns a double matrix.
Each row represents a different variable in my function.
Some of these variables are in 64 bit integer format.
I can typecast these variables to fit into the double (64 bit) matrix, with all the bits intact.
When debugging the C program the bits are there, but when looking at the variable in matlab, the bitpattern has changed for some of the numbers.
All of these numbers are NaN in double domain, so I am concluding that matlab "interpretes" the matrix instead of just copying it.
Can this be turned off?
Bruno Luong
> All of these numbers are NaN in double domain, so I am concluding that matlab "interpretes" the matrix instead of just copying it.
I'm very surprised if it does. Can you post a code to illustrate it?
Bruno
Andreas Nielsen
Four numbers are created by the mex file (2x2). The first two are two different NaNs which seems to be converted to the same number in matlab: bit 13 should be 0 in the first but returns as 1 for both numbers. The other two are just numbers to illustrate the conversion in matlab.
/* dummy.c */
#include <stdio.h>
#include <mex.h>
/* Output Arguments */
#define OUT plhs[0]
//---------------------------------------------
// MEX-Function : Gateway-function to matlab.
//---------------------------------------------
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
double *out_buf;
long long nanvar1 = 0xfff6ac007ffead00;
long long nanvar2 = 0xfffeac007ffead00;
long long var1 = 0x0000000000000037;
long long var2 = 0x3333333333333333;
OUT = mxCreateDoubleMatrix(2,2,mxREAL);
out_buf = mxGetPr(OUT);
out_buf[0] = *(double*) &nanvar1;
out_buf[1] = *(double*) &nanvar2;
out_buf[2] = *(double*) &var1;
out_buf[3] = *(double*) &var2;
}
/* End of file */
% testdummy.c
clear all
a = dummy();
b = zeros(32,2);
c = typecast(a(3),'uint32'); % chop into 32 bit variables since dec2bin does not support 64
c = dec2bin(c)'-48;
c = c(:,[2 1]); % little-endian conversion
b(end-size(c,1)+1:end,:) = c; % if the last bits are zero they are not returned in the string by dec2bin
b = reshape(b,64,1);
% end of file
Andreas Nielsen
memcpy(&out_buf[0], &nanvar1, sizeof(double));
memcpy(&out_buf[1], &nanvar2, sizeof(double));
memcpy(&out_buf[2], &var1, sizeof(double));
memcpy(&out_buf[3], &var2, sizeof(double));
This assures that it is not the casting in C that changes the numbers. I think I saw a VS compiler having problems with that.
Bruno Luong
>> mex testdummy.c
>> d=testdummy;
>> i64=typecast(d(:),'int64');
>> format hex;
>> d
>> i64
d =
fff6ac007ffead00 0000000000000037
fffeac007ffead00 3333333333333333
i64 =
fff6ac007ffead00
fffeac007ffead00
0000000000000037
3333333333333333
>> format short
>> i64
i64 =
-2625631619732224
-373831806046976
55
3689348814741910323
>>
Andreas Nielsen
>> a = dummy();
>> i64 = typecast(a(:), 'int64');
>> i64
i64 =
fffeac007ffead00
fffeac007ffead00
0000000000000037
3333333333333333
This is on another computer, but the result is the same. I tried using the matlab compiler aswell as MS Visual C++ 2005 with the same result. Are you on a windows pc?
Andreas Nielsen
Bruno Luong
>
> This is on another computer, but the result is the same. I tried using the matlab compiler aswell as MS Visual C++ 2005 with the same result. Are you on a windows pc?
Yes. PC Vista 64 bits, Matlab 2009A 64 bits, MSVS C++ 2008.
Bruno
James Tursa
>
> >
> > This is on another computer, but the result is the same. I tried using the matlab compiler aswell as MS Visual C++ 2005 with the same result. Are you on a windows pc?
>
> Yes. PC Vista 64 bits, Matlab 2009A 64 bits, MSVS C++ 2008.
>
> Bruno
I also tried some tests on WinXP, lcc, R2008a and R2007a. Got the leading bit change. It appears to be a compiler/processor issue unrelated to MATLAB, however. e.g.,
#include <string.h>
#include "mex.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
long long nan1 = 0xfff0000000000001;
long long nan2;
double d;
union {long long i; double d;} x, y;
d = *(double *)&nan1;
mexPrintf("Using typecast assignment method:\n");
mexPrintf("nan1 = %llx\n",nan1);
mexPrintf("d = %f\n",d);
nan2 = *(long long *)&d;
mexPrintf("nan2 = %llx\n",nan2);
mexPrintf("Using memcpy method:\n");
mexPrintf("nan1 = %llx\n",nan1);
memcpy(&d,&nan1,sizeof(nan1));
mexPrintf("d = %f\n",d);
memcpy(&nan2,&d,sizeof(d));
mexPrintf("nan2 = %llx\n",nan2);
mexPrintf("Using union method:\n");
mexPrintf("nan1 = %llx\n",nan1);
x.i = nan1;
mexPrintf("x.d = %f\n",x.d);
mexPrintf("x.i = %llx\n",x.i);
y.d = x.d;
mexPrintf("y.d = %f\n",y.d);
mexPrintf("y.i = %llx\n",y.i);
}
>> mex nantest.c
>> nantest
Using typecast assignment method:
nan1 = fff0000000000001
d = -1.#QNAN0
nan2 = fff8000000000001
Using memcpy method:
nan1 = fff0000000000001
d = -1.#QNAN0
nan2 = fff0000000000001
Using union method:
nan1 = fff0000000000001
x.d = -1.#QNAN0
x.i = fff0000000000001
y.d = -1.#QNAN0
y.i = fff8000000000001
You can see that the original typecast method produces the problem (i.e., the leading mantissa bit in the NaN pattern is forced to be set in the assigned value). Using memcpy does not force this. Using unions I was able to set the double NaN value directly with a long long assignment, and then do a double assignment. This seems to confirm that it is the assignment operator that is forcing the leading mantissa bit to be set in the assigned NaN value.
The leading mantissa bit in a IEEE NaN value is the quiet/signaling bit. If the bit is set, the NaN is quiet and is allowed to propagate throughout calculations without any exceptions. If the bit is clear, then the NaN is signaling and may cause an exception in the program to occur, depending on compiler settings, whenever that value gets used in an operation. Once used, the signaling NaN is turned into a quiet NaN (leading mantissa bit gets set) for all downstream operations.
It appears what is happening is that your 0xfff6ac007ffead00 pattern is a signaling NaN because the leading mantissa bit is clear. It is then used in an operation, namely the assignment operation. That act causes the system to act on the signaling NaN (in this case compiler settings are set up to ignore signaling NaN values) and then convert it into a quiet NaN (i.e., set the leading mantissa bit) for downstream calculations.
So for your case, if you want the bit patterns to propagate exactly, you *must* avoid all double operations on the values ... even something seemingly innocuous as a simple assignment, as this can change the leading mantissa bit. You can use either a direct memory copy using memcpy or the like, or you can use unions to get at the bit patterns with integers to copy the data, but you *cannot* use the data as doubles in any operations as this may change the bit pattern.
Finally, I suppose I must point out that all of this is non-conforming. Any attempt to "get at the bits" of a double with type punning, unions, etc. is all compiler/processor dependent.
James Tursa
James Tursa
btw, you also need to be careful when using MATLAB functions with NaN values. MATLAB tends to detect NaN patterns in some situations and then force the use of a "standard" NaN pattern for subsequent calculations. e.g.,
>> A = int64(-1)
A =
-1
>> B = typecast(A,'double')
B =
NaN
>> format hex
>> A
A =
ffffffffffffffff
>> B
B =
ffffffffffffffff
>> format long
>> num2hex(B)
ans =
fff8000000000000
You can see that the int64 variable A has the underlying bit pattern of all bits set, and the double variable B also has all bits set. But the num2hex function displays B as a "standard" NaN pattern, not the original underlying bit pattern.
James Tursa
Rune Allnor
Then do things the oposite way: Type-cast the doubles to integers
and cast back in the matlab workspace.
Or use a struct or cell array tpo return mixed-type data.
Rune
Andreas Nielsen
It sounds right to avoid all operations involving the doubles, but as far as I can see I dont do any operations and still get the change. It seems that the interface between the C mex file and matlab does an operation on the doubles that changes the bit.
@ Rune Alnor
Changing the output format would obviously be a solution, but since this is part of a larger system with 20+ users, they would probably complain if I were to change the interface, so that is not an option.
Instead I will be saving the numbers in two rows of the matrix each with a 32 bit integer converted to double format.
Bruno Luong
> @James Tursa
> It sounds right to avoid all operations involving the doubles, but as far as I can see I dont do any operations and still get the change. It seems that the interface between the C mex file and matlab does an operation on the doubles that changes the bit.
I rather agree with James : I don't think Matlab has anything to do with it. To sum up, the "issue" is from the fact that statement in C
a = b;
where a, b are double are not warranty to preserve bits. It happens at the C-compiler level. It seems still odd, but not non-sense.
I wonder if you compile in plain C or C++ ?
Bruno
James Tursa
> It sounds right to avoid all operations involving the doubles, but as far as I can see I dont do any operations and still get the change.
Yes you do. As I pointed out (or tried to point out) in my earlier post, you do a double assignment operation. The = is an operation. e.g., from your own code, here are two = operations on doubles:
out_buf[0] = *(double*) &nanvar1;
out_buf[1] = *(double*) &nanvar2;
out_buf[0], which is assigned the value from the original nanvar1 bit pattern, a signaling NaN, will have the bit change immediately after the above line gets executed, *before* it gets returned to MATLAB.
> It seems that the interface between the C mex file and matlab does an operation on the doubles that changes the bit.
No. The example I provided showed that the change happens inside the mex routine with the = operations, *before* the mex routine returns back to MATLAB. The change is *entirely* the result of how signaling NaN bit patterns are changed to quiet NaN bit patterns as the result of any double operation with them. It has nothing to do with MATLAB.
James Tursa
Andreas Nielsen
Im afraid I have to disagree. As I stated in a previous post, I have also implemented the mex function using the memcpy instead, but with the same result. I have also compiled a larger project in Visual C++ giving me the opportunity to debug the program line by line. When doing the double assignment, I could observe the change within the code, but using memcpy the bits were unchanged within the C part, but was still changed after returning to matlab. Forgive if I am wroing, but as I understand your post my code should work simply changing
out_buf[0] = *(double*) &nanvar1;
memcpy(&out_buf[0], &nanvar1, sizeof(double));
@ Bruno Luong
By default I use the MS Visual C++ 2005 compiler, but I have also tried the matlab one, which I believe is C. They both return the same results.
Bruno Luong
> out_buf[0] = *(double*) &nanvar1;
> to
> memcpy(&out_buf[0], &nanvar1, sizeof(double));
> This, unfortunately, does not solve the problem.
With LCC, memcpy works. MSVS 2005 probably is too smart ? Could you try.
int sz;
sz = sizeof(double);
// ...
memcpy((void*)&out_buf[0], (void*)&nanvar1, sz);
Bruno
Andreas Nielsen
>> mex -setup
Please choose your compiler for building external interface (MEX) files:
Would you like mex to locate installed compilers [y]/n? y
Select a compiler:
[1] Lcc-win32 C 2.4.1 in C:\PROGRA~1\MATLAB\R2007b\sys\lcc
[2] Microsoft Visual C++ 2005 Express Edition in C:\Program Files\Microsoft Visual Studio 8
[3] Microsoft Visual C++ .NET 2003 in C:\Program Files\Microsoft Visual Studio .NET 2003
[0] None
Compiler: 1
Please verify your choices:
Compiler: Lcc-win32 C 2.4.1
Location: C:\PROGRA~1\MATLAB\R2007b\sys\lcc
Are these correct?([y]/n): y
Trying to update options file: C:\Documents and Settings\abn\Application Data\MathWorks\MATLAB\R2007b\mexopts.bat
From template: C:\PROGRA~1\MATLAB\R2007b\bin\win32\mexopts\lccopts.bat
Done . . .
>> mex dummy.c
>> format hex
>> dummy()
ans =
fffeac007ffead00 0000000000000037
fffeac007ffead00 3333333333333333
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
double *out_buf;
int sz;
long long nanvar1 = 0xfff6ac007ffead00;
long long nanvar2 = 0xfffeac007ffead00;
long long var1 = 0x0000000000000037;
long long var2 = 0x3333333333333333;
sz = sizeof(double);
OUT = mxCreateDoubleMatrix(2,2,mxREAL);
out_buf = mxGetPr(OUT);
memcpy((void*)&out_buf[0], (void*)&nanvar1, sz);
memcpy((void*)&out_buf[1], (void*)&nanvar2, sz);
memcpy((void*)&out_buf[2], (void*)&var1, sz);
memcpy((void*)&out_buf[3], (void*)&var2, sz);
}
Bruno Luong
> >> mex dummy.c
> >> format hex
> >> dummy()
>
> ans =
>
> fffeac007ffead00 0000000000000037
> fffeac007ffead00 3333333333333333
>
No. don't test like the above. Test like this:
mex dummy.c
format hex
d=dummy()
i64=typecast(d(:),'int64')
% Bruno
James Tursa
> @ James Tursa
> Im afraid I have to disagree. As I stated in a previous post, I have also implemented the mex function using the memcpy instead, but with the same result. I have also compiled a larger project in Visual C++ giving me the opportunity to debug the program line by line. When doing the double assignment, I could observe the change within the code, but using memcpy the bits were unchanged within the C part, but was still changed after returning to matlab. Forgive if I am wrong, but as I understand your post my code should work simply changing
> out_buf[0] = *(double*) &nanvar1;
> to
> memcpy(&out_buf[0], &nanvar1, sizeof(double));
> This, unfortunately, does not solve the problem.
How are you determining that this does not solve the problem? Are you displaying the data as a double in MATLAB and looking at the bit pattern? This is *not* the way to test this. As I pointed out in an earlier post, you cannot trust all of the MATLAB display functions, conversion functions, and operations to deal with NaN bit patterns and retain the original bit pattern. The NaN values get detected as NaN and sometimes get used/displayed differently. For example, using your exact nanvar1 bit pattern:
// dummy2.c
#include <string.h>
#include "mex.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
long long nanvar1 = 0xfff6ac007ffead00;
plhs[0] = mxCreateDoubleMatrix(1, 2, mxREAL);
memcpy(mxGetPr(plhs[0]),&nanvar1,sizeof(double));
}
>> format hex
>> mex dummy2.c
>> A = dummy2
A =
fffeac007ffead00 0000000000000000
>> B = typecast(A,'int64')
B =
fff6ac007ffead00 0000000000000000
Here you can see that the original bit pattern, when displayed as a double in MATLAB, has the signaling bit set, and *apparently* this has been changed just by the act of returning the variable to MATLAB. *But* when typecast to an int64 and displayed it is seen that the underlying bit pattern of the double in fact *does* have the signaling bit cleared, just like the original bit pattern in the C mex routine. So the original bit pattern was in fact preserved all along, you just couldn't tell by examining the bit pattern as a double.
Having said all that, I don't disagree that you still may have a problem. If you could post an example of your code using memcpy that still exhibits the problem I would love to look at it. To be honest, I don't even know if the typecast function in MATLAB that I used above can always be trusted to retain the original bit pattern. That is because the typecast function does not make a shared data copy, but actually copies the data. I wouldn't be surprised to learn that in some implementations the copy is basically a loop with double assignments in the background (that would change the signaling NaN bit), whereas in other implementations the copy is basically a memcpy in the background (that would not change the signaling NaN bit). And I would generalize this concern: *any* operation on the MATLAB side that results in a data copy may be implemented in the background as a double assignment
in a loop, and thus be subject to the signaling NaN bit changing. i.e., doing C = A in the above example and then doing C(2) = 1 will cause a data copy to take place for C, and thus may in fact change the C(1) bit pattern depending on how the background data copy is implemented.
As an aside, I don't understand why the typecast function does an actual data copy instead of a shared data copy. There is no need to do an actual data copy for this function, so I don't know why MATLAB implements it this way. I will soon be posting a replacement C-mex version of typecast to the FEX that *does* do a shared data copy (I recently figured out how to do it using some undocumented functions).
James Tursa
Andreas Nielsen
Thank you all for your enthusiastic and skilled replies.
Andreas
Rune Allnor
> @ Rune Alnor
> Changing the output format would obviously be a solution, but since this is part of a larger system with 20+ users, they would probably complain if I were to change the interface, so that is not an option.
You have already recieved ample explanations why what
you do is bad.
Changing the output fromat is your only option. If users
are gicen the choice between changing an interface or
getting software that behaves unpredictably - well, it's
not much of a choise, is it?
> Instead I will be saving the numbers in two rows of the matrix each with a 32 bit integer converted to double format.
You are asking for trouble.
Rune
Andreas Nielsen
In matlab this typecast is fine:
typecast(s(:),'uint64')
whereas this changes the bit:
typecast(s(4,:), 'uint64')
I guess the s(4,:) copies the variable, which changes the bit, whereas s(:) simply returns a pointer to the existing variable.
This does not change the bit:
a=s;
typecast(a(:),'uint64')
so I wouldnt say Matlab is completely consistent in the processing of NaNs. I guess many of you already pointed this out :)
Steven Lord
"Rune Allnor" <all...@tele.ntnu.no> wrote in message
news:88c9d27b-ca45-46d7...@f41g2000pra.googlegroups.com...
> On 30 Apr, 08:30, "Andreas Nielsen" <muaddibs...@gmail.com> wrote:
>
>> @ Rune Alnor
>> Changing the output format would obviously be a solution, but since this
>> is part of a larger system with 20+ users, they would probably complain
>> if I were to change the interface, so that is not an option.
>
> You have already recieved ample explanations why what
> you do is bad.
>
> Changing the output fromat is your only option. If users
> are gicen the choice between changing an interface or
> getting software that behaves unpredictably - well, it's
> not much of a choise, is it?
I agree with Rune. You're doing something "sneaky" that sounds like it
could be very fragile. Changing the interface now will cause some pain now,
but will likely prevent pain in the future, and changing the interface is
easiest when there are few dependencies on it.
As a concrete example: last release we introduced a new 2D integration
function called QUAD2D. If for whatever reason we felt we needed to modify
all our 2D integration routines, I guarantee you modifying QUAD2D (which
doesn't have as many people depending on its behavior) would cause less pain
than modifying the older 2D integration function DBLQUAD, which has been
part of MATLAB for Quite Some Time and is likely used in many, many more
functions than QUAD2D.
>> Instead I will be saving the numbers in two rows of the matrix each with
>> a 32 bit integer converted to double format.
>
> You are asking for trouble.
I agree.
--
Steve Lord
sl...@mathworks.com