Usage of malloc/free in mex *.c files
Gunnar Selke
the documentation is a bit unclear about this.. when inside
mexFunction(), is it allowed to allocate memory using the c malloc
library function?
I am trying to do so and I am am getting weird heap corruption errors
when mexFunction is called the 2nd time. I am using the malloc'ed blocks
only within the mexFunction(), and the pointers are never returned to
Matlab.
The documentation says I *should* use the Matlab equivalents, but I
can't easily do so because I am using a 3rd party library.
Thanks in advance,
Gunnar
Praetorian
Are you freeing the memory after you're done using it? mxCalloc,
mxMalloc etc. free all allocated memory automatically when the MEX
file exits so you don't HAVE to call mxFree (unless you made the
memory persistent by calling mexMakeMemoryPersistent). Also, if you're
using the mexFunction interface you should have access to all those
functions, what do you mean when you say you can't use the mex
equivalents 'easily'?
Ashish.
James Tursa
wrote:
Using malloc (or calloc or realloc) without freeing the memory will
not in and of itself result in memory corruption errors, it will
simply result in an out of memory error when the heap is exhausted
because MATLAB no longer has access to that memory. This is
particularly true if you repeatedly call the function that leaks
memory over-and-over as you indicate. If you are getting memory
corruption errors then I would suspect you are overwriting memory
array bounds somewhere in your code, or misusing a MATLAT mx___
function call. A 3rd party library should free up any temporary memory
it allocates, or it isn't written very well. Are you getting a pointer
returned from a function you are calling from this 3rd party library,
but not freeing this memory later on? Can you post some code for us to
look at?
And like the other poster already replied, code that you write should
always use mxMalloc, mxCalloc, and mxRealloc to avoid potential memory
leaks.
James Tursa
Gunnar Selke
thank you very much for your replies.
>> the documentation is a bit unclear about this.. when inside
>> mexFunction(), is it allowed to allocate memory using the c malloc
>> library function?
>
> Using malloc (or calloc or realloc) without freeing the memory will
> not in and of itself result in memory corruption errors, it will
> simply result in an out of memory error when the heap is exhausted
> because MATLAB no longer has access to that memory.
I am aware of how memory allocation (and deallocation) in C works. My
problem is heap corruption, not memory leaking.
If it is okay to use malloc and free interally inside mexFunction, and
malloc'd (vs. mxAlloc) memory is not garbage collected when Matlab
regains control after mexFunction returns, then it must be a bug in my
code (or in the library's). I guess I have some more debugging to do..
Thanks again,
Gunnar
Ralph Schleicher
> If it is okay to use malloc and free interally inside mexFunction, and
> malloc'd (vs. mxAlloc) memory is not garbage collected when Matlab
> regains control after mexFunction returns, then it must be a bug in my
> code (or in the library's). I guess I have some more debugging to do..
You can use malloc/free without limitations. If your memory gets
corrupted, your code is buggy.
--
Ralph
5539 days of Linux experience.
James Tursa
<r...@mueller-schleicher.de> wrote:
>Gunnar Selke <1se...@informatik.uni-hamburg.de> writes:
>
>> If it is okay to use malloc and free interally inside mexFunction, and
>> malloc'd (vs. mxAlloc) memory is not garbage collected when Matlab
>> regains control after mexFunction returns, then it must be a bug in my
>> code (or in the library's). I guess I have some more debugging to do..
>
>You can use malloc/free without limitations.
No. You can use them, but there *are* limitations. Namely the memory
will leak if you execute a mexErrMsgTxt call inbetween the malloc and
the free calls. That's what I had pointed out in an earlier post.
James Tursa
Ralph Schleicher
> No. You can use them, but there *are* limitations. Namely the memory
> will leak if you execute a mexErrMsgTxt call inbetween the malloc and
> the free calls. That's what I had pointed out in an earlier post.
If that can happen, then the code is buggy.
--
Ralph
James Tursa
message <87ir1nw...@bravo.mueller-schleicher.i>...
You are still missing the point, which is this:
It is *impossible* to write mex code that mixes
malloc/calloc/realloc with any of the mx routines that
allocate memory (e.g., mxCreate...) or with the
mexErrMsgTxt routine without risking a memory leak unless
*all* of the malloc/calloc/realloc memory is freed before
you make any of the mx calls. That is why this statement
you made in an earlier post does not make any sense to me:
>You can use malloc/free without limitations.
I would consider the restriction to never mix the calls in
the way I described a very real and important limitation.
For instance, consider the following example:
1) mxArray *mx;
2) double *dp;
3) dp = (double *) calloc(100*200, sizeof(double));
4) if( dp == NULL )
5) do something to handle error here;
6) mx = mxCreateDoubleMatrix(100, 200, mxREAL);
7) if( mx == NULL ) {
8) free(dp);
9) mexErrMsgTxt("Unable to create mx");
10) }
11) // some code here to use mx and dp
12) mxDestroyArray(mx);
13) free(dp);
14) // rest of your code
Looks like line 8 will cover me and prevent a memory leak
if the mxCreateDoubleMatrix allocation doesn't work,
right? Wrong. Line 8 and 9 will never be executed under
any conditions in a mex routine. In a mex routine, if the
mxCreateDoubleMatrix call fails then control is returned
directly to MATLAB after the memory allocated from
previous MATLAB mx routines such as mxMalloc, mxCalloc,
mxCreate..., etc. is freed. Line 7 will never be reached
if mx is NULL. The memory that dp points to is leaked in
this case.
Now, I don't have a particular problem with calling my
example code above buggy, since it can produce a memory
leak. But I do have a problem with saying
malloc/calloc/realloc can be used without limitations.
They can't.
James Tursa
Ralph Schleicher
> You are still missing the point, which is this:
I don't miss the point. Instead, I'm fully aware of it. But this is
all very well explained in the Matlab API documentation. Your example
is the typical code of an inexperienced MEX-file programmer who (maybe)
has read the Matlab API documentation but neither does understand it
nor is able to transliterate it into an error-free program. Again,
utilizing malloc/free within a MEX-file is totally legal and they can
be used without limitations. C gives you enough rope to hang yourself,
but nobody said that you shall actually do it.
--
Ralph
Kris Thielemans
> "James Tursa" <aclassyguy...@hotmail.com> writes:
>
> > You are still missing the point, which is this:
>
> I don't miss the point. Instead, I'm fully aware of it. But this is
> all very well explained in the Matlab API documentation.
Hi
question: what happens when the user tries presses Ctrl-C (or equivalent)? Can your mex code be interrupted at any point (e.g. between malloc and free calls)? If so, you'll have a memory leak. But maybe matlab only breaks when you call a mx function?
Thanks
Kris
Ralph Schleicher
> question: what happens when the user tries presses Ctrl-C (or
> equivalent)? Can your mex code be interrupted at any point
> (e.g. between malloc and free calls)? If so, you'll have a memory
> leak. But maybe matlab only breaks when you call a mx function?
Hello Kris,
For such a case, you depend on how Matlab handles the signal.
I've only once investigated the behavior of a C-MEX S-function
in case the user interrupts a running simulation with Ctrl-C.
The C-MEX S-function under investigation did extensive data
logging of Simulink signals. More than 500 instances of the
C-MEX S-function wrote approx. 10000 Simulink signals at each
major simulation time step into a HDF5 file. All the memory
for the internal data structures was allocated with malloc.
My tests showed that it was not possible to generate a memory
leak as long as Matlab continued to run.
Here is the general plan for how to use malloc/free
successfully in Matlab:
1. Initialize all pointers
2. Install a cleanup procedure (and make sure
it is called in case of an error)
3. Allocate memory
4. Work real hard
5. Call the cleanup procedure unless Matlab
does it for you automatically
In the cleanup procedure
1. Free allocated memory
2. Reinitialize pointers
--
Ralph Schleicher <http://ralph-schleicher.de>
Development * Consulting * Training
Mathematical Modeling and Simulation
Software Tools
Ajay
Recently I also faced this problem when I failed to use malloc.
I was trying to pass an array of structure, for example
//c-style structure
typedef struct A {
int a1,
int a2
};
Now I pass Matlab structure from matlab to mex function(c) through this way
a(1).a1 = 1;
a(1).a1 = 2;
a(2).a1 = 3;
a(4).a1 = 4;
I wanted use malloc to dynamically allocate memory for my structure A.
And it told me undeclared identified with this statement
A *temp; //A is already defined as struct above
temp = (A*)malloc(sizeof(A);
Above syntax did not work out, but if you write in this way
struct A *temp;
temp = (struct A*)malloc (sizeof(struct A));
Using struct infront of A, worked out.
I hope this could be useful for others people who search for Mex Malloc Memory allocation issue.
Ajay Pratap Singh apts
ps:. please let me know if it was helpful
Gunnar Selke <gse...@gunnar.info> wrote in message <478a9107$1...@news.arcor-ip.de>...
Val
I am trying to read a binary file using an external library and pass the results to MATLAB. The external library allocates memory, reads the file and returns a data structure and it's type with each read call.
My own code then creates a MATLAB structure with fields to match the record that was read, and then creates mxArray's of type double for each of the fields and inserts these into the structure. Multiple records are read (the user specifies how many as an argument) and each resulting MATLAB structure is inserted into a cell array which is ultimately passed back to the MATLAB user space. (All of this is happening in a mex file with mxCreateCell, mxCreateStruct, mxSetField, mxSetCell, mxCreateDoubleMatrix, etc.).
I have several questions.
From what I have understood you to say, because I am interleaving calls to malloc and mxMalloc (via the external library and calls to create the structure and each field), I am corrupting my memory space. Is that correct?
I think I understand this discussion here to imply that one can use malloc/free within a mex file as along as all memory is freed before calls to mxMalloc. I cannot see a way to read the data using the library (which will call a malloc) and then free this memory space before calling mxMalloc as there's no place to put the data yet. Is there a standard obvious way to do this that I'm missing?
Finally, when I call the library's function call to free memory allocated when reading a record, MATLAB immediately crashes with an abort(). I suspect this results because the malloc and mxMalloc calls have allocated overlapping memory spaces and that call is attempting to free memory allocated to another pointer. Do you think I'm getting that right?
-Val
James Tursa
>
> I am trying to read a binary file using an external library and pass the results to MATLAB. The external library allocates memory, reads the file and returns a data structure and it's type with each read call.
>
> My own code then creates a MATLAB structure with fields to match the record that was read, and then creates mxArray's of type double for each of the fields and inserts these into the structure. Multiple records are read (the user specifies how many as an argument) and each resulting MATLAB structure is inserted into a cell array which is ultimately passed back to the MATLAB user space. (All of this is happening in a mex file with mxCreateCell, mxCreateStruct, mxSetField, mxSetCell, mxCreateDoubleMatrix, etc.).
>
> I have several questions.
>
> From what I have understood you to say, because I am interleaving calls to malloc and mxMalloc (via the external library and calls to create the structure and each field), I am corrupting my memory space. Is that correct?
> I think I understand this discussion here to imply that one can use malloc/free within a mex file as along as all memory is freed before calls to mxMalloc.
> I cannot see a way to read the data using the library (which will call a malloc) and then free this memory space before calling mxMalloc as there's no place to put the data yet. Is there a standard obvious way to do this that I'm missing?
> Finally, when I call the library's function call to free memory allocated when reading a record, MATLAB immediately crashes with an abort(). I suspect this results because the malloc and mxMalloc calls have allocated overlapping memory spaces and that call is attempting to free memory allocated to another pointer. Do you think I'm getting that right?
James Tursa
Val
One detail: you'll see below that I'm calling an external MATLAB function with "mxCallMATLAB" to create an empty data structure. The m-file that is called creates an empty data structure of the type of record passed to it in which the fields for the type of structure are defined something like:
function rec = createemtpyrecord(id)
switch id
case 2:
rec.latitude = [];
rec.longitude = [];
end
Then I populate these fields with values from the record read from the binary file. I simply found this method to be easier than coding this up within the MEX file.
First I'm defining this utility function which accepts a pointer to a mxArray structure, a field name and the field value (as a double) and creates a mxArray double matrix, assigns the value to the mxArray and inserts it into the structure.
#include "gsf.h" /* this is the external library for reading and writing the files */
#include <mex.h>
#include <string.h>
#include <stdio.h>
#include <math.h>
#define PI 3.141592653589793
/* A function to reset a value of type double in a structure by name */
void setMxDoubleField( mxArray *record, const char *fieldname, double value, int DEBUG ) {
mxArray *thisfield;
double *pvalue;
if (mxGetField(record, (mwIndex) 0, fieldname) != NULL){
mxDestroyArray(mxGetField(record,(mwIndex) 0, fieldname));
}
thisfield = mxCreateDoubleMatrix( (mwSize) 1, (mwSize) 1, mxREAL);
pvalue = mxGetPr(thisfield);
*pvalue = value;
if (DEBUG > 0) {
mexPrintf("DEBUG %s:\t%f\n",fieldname, *pvalue);
}
/* Then push the whole thing back into the structure */
mxSetField(record, (mwIndex) 0, fieldname, thisfield);
}
--------------------
Ok, here is the mex function with some of the details removed for clarity:
/* MEX FUNCTION */
/*************************************************************************/
/* INPUT: */
/* filename: text string containing the path and gsf filename */
/* startidx: index of first record to read from file */
/* endidx: index of last record to read from file */
/* rtype: type of record to read 0=all, see gsf.h for types */
/* DEBUG: 0 or for none, > 0 for increased debugging output */
/* OUTPUT: */
/* rec: cell array of structures containing the records read*/
/*************************************************************************/
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
/* Variables to handle reading GSF files */
gsfRecords gsf_record;
gsfDataID id;
/* Utility Variables */
int i = 0, j = 0, p = 0, q = 0, N = 0, status = 0, recordSize = 0;
int count = 0;
int FID;
/* Variables to handle input arguments */
char* filename;
double *pstartidx, *pendidx, *prtype, *pDEBUG;
int startidx = 0, endidx = 0, rtype = 0, DEBUG = 0;
/* Variables to create output cell array */
mxArray *rec;
mwSize ndim;
mwSize dims[2];
ndim = 2;
dims[0] = 1;
dims[1] = 1;
/* Variables to handle output data structures */
mxArray *oplhs[1], *oprhs[1];
mxArray *thisrecord, *thisfield;
double *orecordid;
int Nfields;
double *value;
struct timespec gsftime; // tv_sec (time_t); tv_nsec (long); (seconds and nanoseconds)
double tmp;
const char **fnames;
/* SKIPPING DETAILS REGARDING HANDING OF INPUT ARGUMENTS... */
/* CREATE OUTPUT ARRAY */
/***********************/
/* Create cell array to hold results whose size is endidx-startidx. */
dims[1] = endidx - startidx + 1;
rec = mxCreateCellArray(ndim,dims);
/* OPEN AND READ THE FILE */
/**************************/
/* Open the file */
if ( gsfOpen(filename, GSF_READONLY, &FID) != 0 ) {
mexPrintf("Failed to open GSF file: %s\n", filename);
return;
}
/* BEGIN READING THE FILE*/
/* Here count is counting from 0, as it's used to index into the output */
/* array, but endidx is counting from 1 since it makes more cenceptual */
/* sense to say I want records 1 to 3 and get the first 3 records back. */
while (count + 1 <= endidx) {
/* read a record */
recordSize = gsfRead(FID, 0, &id, &gsf_record, NULL, 0);
/*NOTE: IF I COMMENT EVERYTHING ELSE OUT AND CALL gsfFree(&gsf_record); HERE */
/*MATLAB IMMEDIATELY CRASHES WITH AN ABORT(). */
/* Quit if we're at the end of the file. */
if (recordSize == GSF_READ_TO_END_OF_FILE) {
mexPrintf("Reached EOF.");
break;
}
if (recordSize < 0) {
mexPrintf("Failed to read GSF Record.\n");
continue;
}else{
if (DEBUG > 0) {
mexPrintf("DEBUG Read record, %d bytes.\n",recordSize);
}
}
if (DEBUG > 0 ) {
mexPrintf("\n");
mexPrintf("DEBUG record ID: %d\n", id.recordID);
mexPrintf("DEBUG record number: %d\n", id.record_number);
mexPrintf("\n");
}
/* If we haven't reached the first desired record yet, or */
/* if we aren't asking for all records (0) and the record we've */
/* got isn't the one we want (rtype), then go again. */
/* Remember that count starts from 0 but startidx starts from 1. */
if (count + 1 < startidx | ( (rtype > 0) & (id.recordID != rtype) ) ) {
//gsfFree(&gsf_record); /* COMMENTED OUT BECAUSE IT CAUSES A CRASH*/
continue;
}
/* Set up for creating empty data structure to hold the record */
/* Create a mwArray to hold our record id argument */
oprhs[0] = mxCreateDoubleMatrix( (mwSize)1, (mwSize)1, mxREAL);
/* Get a pointer to where the data goes */
orecordid = mxGetPr(oprhs[0]);
/* switch on recordid */
switch (id.recordID)
{
case 1:
break;
case 2:
/* Assign the record id to our mxArray that is our argument list */
orecordid[0] = 2.0;
/* Call gsf_create_emptyrecord with our record id as our argument list */
/* This should return a pointer to a 1x1 structure array in oplhs */
if ( mexCallMATLAB(1,oplhs,1,oprhs,"gsf_createemptyrecord") != 0) {
mexPrintf("Failed to create empty record structure for gsf data.\n");
return;
}
thisrecord = oplhs[0];
Nfields = mxGetNumberOfFields(thisrecord);
if (DEBUG > 0) {
mexPrintf("Number of Fields in empty structure: %d\n",Nfields);
}
/* allocate memory for storing pointers for field names */
fnames = mxCalloc(Nfields, sizeof(*fnames));
/* get field name pointers */
for (i=0; i< Nfields; i++){
fnames[i] = mxGetFieldNameByNumber(thisrecord,i);
}
/* Loop through the fields, assigning each one. */
for (i=0;i<Nfields;i++) {
if (DEBUG > 2) {
mexPrintf("DEBUG fieldname: %s\n",fnames[i]);
}
/* latitude */
if (strcmp(fnames[i],"latitude") == 0) {
setMxDoubleField( thisrecord, fnames[i], (double) gsf_record.mb_ping.latitude, DEBUG);
continue;
}
/* longitude */
if (strcmp(fnames[i],"longitude") == 0) {
setMxDoubleField( thisrecord, fnames[i], (double) gsf_record.mb_ping.longitude, DEBUG);
}
} /* end for loop */
/* Assign the record to the output cell array */
mxSetCell(rec, (mwIndex) count, thisrecord);
/* increment our output record counter */
count = count + 1;
mxFree(fnames);
//gsfFree(&gsf_record);
break;
default:
break;
}
}
/* End while */
/* Close file */
gsfClose(FID);
/* When finished using the string, deallocate it. */
mxFree(filename);
/* cleanup allocated memory */
mxDestroyArray(oprhs[0]);
mxDestroyArray(oplhs[0]);
/* Point output to our records */
plhs[0] = rec;
return;
}
After writing all this up I think the problem is within the library function that frees the memory. Without that i have a memory leak but it causes MATLAB to crash every time I invoke it. Maybe someone will see something else too.
Thanks,
Val
Bruno Luong
> void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
> char* filename;
...
> if ( gsfOpen(filename, GSF_READONLY, &FID) != 0 ) {
> mexPrintf("Failed to open GSF file: %s\n", filename);
> return;
> }
>
> /* When finished using the string, deallocate it. */
> mxFree(filename);
>
Bruno
Val
>
> May be I miss something obvious, but I can't see where *filename is allocated or initialized.
>
> Bruno
This is the snippet of code I omitted:
N = mxGetN(prhs[0])*sizeof(mxChar)+1;
filename = mxMalloc(N);
/* Copy the string data into buf. */
status = mxGetString(prhs[0], filename, (mwSize)N);
mexPrintf("Reading: %s\n", filename);
James Tursa
>
> /* Variables to handle reading GSF files */
> gsfRecords gsf_record;
> /* read a record */
> recordSize = gsfRead(FID, 0, &id, &gsf_record, NULL, 0);
>
> /*NOTE: IF I COMMENT EVERYTHING ELSE OUT AND CALL gsfFree(&gsf_record); HERE */
> /*MATLAB IMMEDIATELY CRASHES WITH AN ABORT(). */
James Tursa
Bruno Luong
> /* Assign the record id to our mxArray that is our argument list */
> orecordid[0] = 2.0;
> /* Call gsf_create_emptyrecord with our record id as our argument list */
> /* This should return a pointer to a 1x1 structure array in oplhs */
> if ( mexCallMATLAB(1,oplhs,1,oprhs,"gsf_createemptyrecord") != 0) {
> mexPrintf("Failed to create empty record structure for gsf data.\n");
> return;
> }
> thisrecord = oplhs[0];
> mxDestroyArray(oplhs[0]);
>
Can you explain why mxDestroyArray(oplhs[0]) is carried out outside the switch/case (all the time) but the oplhs[0] is filled only for "case 2:". Do you risk to mxDestroyArray with an invalid pointer?
Bruno
Val
I compiled some test code within a debugger and had a conversation with the author of the library and found that some of the pointers within gsfRecords are not set to null by default and may not be allocated on a call to gsfRead(). The subsequent call to gsfFree checks to see that they are non-Null and then deallocates them if they are, but in this case was deallocating unassigned memory. Hence the abort(). This statement after the variable declarations prevents the abort:
memset(&gsf_record, 0, sizeof(gsfRecords));
With that solved something is still amiss regarding my assignment of variables into the cell array. I'll try to post some specific questions as they come up.
-Val
Val
> Can you explain why mxDestroyArray(oplhs[0]) is carried out outside the switch/case (all the time) but the oplhs[0] is filled only for "case 2:". Do you risk to mxDestroyArray with an invalid pointer?
>
> Bruno