Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Using malloc/free in a tight loop
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  13 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Navaneeth  
View profile  
 More options Nov 17 2010, 4:02 am
Newsgroups: comp.lang.c
From: Navaneeth <navaneet...@gmail.com>
Date: Wed, 17 Nov 2010 01:02:17 -0800 (PST)
Local: Wed, Nov 17 2010 4:02 am
Subject: Using malloc/free in a tight loop
My application will be looping for 5000 times. On each iteration, it
has to create a string and pass that into other methods for
processing. This string length will only be known at the runtime.

So in each iteration, this is what happens.

ptr = malloc
process(ptr)
free(ptr)

I am wondering doing frequent malloc and free is ok? Will this
fragment the memory?

My development environment is GCC and Linux. But I'd be happy to know
about the implementation in windows also.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Malcolm McLean  
View profile  
 More options Nov 17 2010, 5:12 am
Newsgroups: comp.lang.c
From: Malcolm McLean <malcolm.mcle...@btinternet.com>
Date: Wed, 17 Nov 2010 02:12:45 -0800 (PST)
Local: Wed, Nov 17 2010 5:12 am
Subject: Re: Using malloc/free in a tight loop
On Nov 17, 11:02 am, Navaneeth <navaneet...@gmail.com> wrote:

Unlikely. When you create strings and delete strings at random you
might fragment the memory, but if you are matching allocations to
frees then the system will usually resuse the memory. (If you print
out the pointers with %p you'll probably find that you get the same
value time after time).

Unless the strings are extremely long then a modern computer isn't
going to notice allocating 5000 of them.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Navaneeth  
View profile  
 More options Nov 17 2010, 7:00 am
Newsgroups: comp.lang.c
From: Navaneeth <navaneet...@gmail.com>
Date: Wed, 17 Nov 2010 04:00:25 -0800 (PST)
Local: Wed, Nov 17 2010 7:00 am
Subject: Re: Using malloc/free in a tight loop

> Unless the strings are extremely long then a modern computer isn't
> going to notice allocating 5000 of them.

Thanks. My strings are not long. May be 20 characters.

So in general, can malloc fragment the memory? I was expecting the
modern implementations will be clever enough to reduce this problem.
Isn't that true?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mikko Rauhala  
View profile  
 More options Nov 17 2010, 7:25 am
Newsgroups: comp.lang.c
From: Mikko Rauhala <m...@iki.fi>
Date: Wed, 17 Nov 2010 12:25:39 +0000 (UTC)
Local: Wed, Nov 17 2010 7:25 am
Subject: Re: Using malloc/free in a tight loop
On Wed, 17 Nov 2010 04:00:25 -0800 (PST), Navaneeth <navaneet...@gmail.com>

 wrote:
>> Unless the strings are extremely long then a modern computer isn't
>> going to notice allocating 5000 of them.

> Thanks. My strings are not long. May be 20 characters.

You know, if you have a very short maximum cap for the length, you
could just allocate that and reuse the buffer. (You could also by
default reuse the buffer and only reallocate a larger one as it
becomes necessary.)

That said, the first response was correct in that it's not likely to
make much of a difference in this quite simple case. In general though,
I would avoid unnecessary repeated allocations and frees in a tight
loop.

> So in general, can malloc fragment the memory?

In general, it can indeed. Especially if your processing step here
involves allocating space that _isn't_ freed, it can happen in your
example too, but probably not to a large extent. (The malloc
implementation just needs to be smart enough to reuse the space
left by the previous freed buffer, in the common case that it's
sufficiently large).

--
Mikko Rauhala <m...@iki.fi>       - http://www.iki.fi/mjr/blog/  
The Finnish Pirate Party         - http://piraattipuolue.fi/  
World Transhumanist Association  - http://transhumanism.org/  
Singularity Institute            - http://singinst.org/  


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
JohnF  
View profile  
 More options Nov 17 2010, 8:01 am
Newsgroups: comp.lang.c
From: JohnF <j...@please.see.sig.for.email.com>
Date: Wed, 17 Nov 2010 13:01:17 +0000 (UTC)
Local: Wed, Nov 17 2010 8:01 am
Subject: Re: Using malloc/free in a tight loop

Navaneeth <navaneet...@gmail.com> wrote:
> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.
> So in each iteration, this is what happens.
>   ptr = malloc
>   process(ptr)
>   free(ptr)
> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?

As per MM's reply, not a problem. Nevertheless, I usually do
this kind of thing, when I very temporarily need lots of short
strings, with a short function that allocates memory from a
static array used as a wraparound buffer, something like
as follows,
   #define BUFFSZ 1000  /* any size you like */
   void *tempalloc(int nbytes) {
     static unsigned char buffer[BUFFSZ]; /* buffer */
     static int index = 0;      /*start at beginning of buffer*/
     void *p = NULL;            /* returned ptr */
     if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
     if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
     p = (void *)(buffer+index); /* ptr returned to caller */
     index += nbytes;           /* next ptr past returned nbytes */
     end_of_job: return(p); }   /* back to caller */
No need to free, or anything. You just have to be logically sure
you won't still be using the memory when it's "realloc'ed" after
the internal static buffer wraps around. (And the above code
doesn't align returned memory on anything.)
--
John Forkosh  ( mailto:  j...@f.com  where j=john and f=forkosh )

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Navaneeth  
View profile  
 More options Nov 17 2010, 8:07 am
Newsgroups: comp.lang.c
From: Navaneeth <navaneet...@gmail.com>
Date: Wed, 17 Nov 2010 05:07:18 -0800 (PST)
Local: Wed, Nov 17 2010 8:07 am
Subject: Re: Using malloc/free in a tight loop

>    #define BUFFSZ 1000  /* any size you like */
>    void *tempalloc(int nbytes) {
>      static unsigned char buffer[BUFFSZ]; /* buffer */
>      static int index = 0;      /*start at beginning of buffer*/
>      void *p = NULL;            /* returned ptr */
>      if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
>      if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
>      p = (void *)(buffer+index); /* ptr returned to caller */
>      index += nbytes;           /* next ptr past returned nbytes */
>      end_of_job: return(p); }   /* back to caller */

John,

Thanks. This seems to be a good idea. I will evaluate it.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
William Hughes  
View profile  
 More options Nov 17 2010, 9:26 am
Newsgroups: comp.lang.c
From: William Hughes <wpihug...@hotmail.com>
Date: Wed, 17 Nov 2010 06:26:17 -0800 (PST)
Local: Wed, Nov 17 2010 9:26 am
Subject: Re: Using malloc/free in a tight loop
On Nov 17, 9:01 am, JohnF <j...@please.see.sig.for.email.com> wrote:

A variant that will deal with long
strings  (this is the method described
by Mikko Rauhala).  The idea is malloc once and reuse
the buffer.   Only realloc if you have to.

(untested, no error checking)

    #define BUFFSZ 20  /* any size you like, 2 will do */
    #define MEM_INCREMENT 2 /* unless you have a good reason use 2 */

    void *tempalloc(int nbytes) {
       static void *p=malloc(BUFFSZ);
       static int currbufsize = BUFFSZ;
      

      while ( nbytes > currbufsize)  {
          curbuffsize = currbufsize*MEM_INCREMENT;
          p==remlloc(p,currbuufsize);
      }

      return p;

}

This has the advantage of portable alignment and
it will not fail if a large allocation is requested.
It has the disadvantage that it
requires the use of the library malloc.
You can only use tempalloc safely if you are
sure that you do not need the memory from
a previous call to tempalloc any more.

A


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Kenneth Brody  
View profile  
 More options Nov 17 2010, 11:51 am
Newsgroups: comp.lang.c
From: Kenneth Brody <kenbr...@spamcop.net>
Date: Wed, 17 Nov 2010 11:51:39 -0500
Local: Wed, Nov 17 2010 11:51 am
Subject: Re: Using malloc/free in a tight loop
On 11/17/2010 4:02 AM, Navaneeth wrote:
> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.

> So in each iteration, this is what happens.

> ptr = malloc
> process(ptr)
> free(ptr)

> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?

[...]

Depending on what sizes you malloc, and what order you do so, and what other
memory allocating/freeing might be going on, the answer is a definite "maybe".

I might suggest a tweak to your method, however.

Keep track of the size you allocated.  Then, rather than unconditionally
freeing it and allocating a new chunk, check the new size that you need.  If
the current allocation is at least that large, use the current allocation.
Otherwise, free/malloc, and keep track of the new size.  Then, after the
loop is finished, free the memory once and for all.

And, if you're feeling adventurous, start with a block of memory that
"should" be sufficient for "most" scenarios.

--
Kenneth Brody


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Chris M. Thomasson  
View profile  
 More options Nov 17 2010, 12:26 pm
Newsgroups: comp.lang.c
From: "Chris M. Thomasson" <cris...@charter.net>
Date: Wed, 17 Nov 2010 09:26:06 -0800
Local: Wed, Nov 17 2010 12:26 pm
Subject: Re: Using malloc/free in a tight loop
"Navaneeth" <navaneet...@gmail.com> wrote in message

news:8f23500d-4a21-4412-b799-9547e7e5714e@t35g2000yqj.googlegroups.com...

> My application will be looping for 5000 times. On each iteration, it
> has to create a string and pass that into other methods for
> processing. This string length will only be known at the runtime.

> So in each iteration, this is what happens.

> ptr = malloc
> process(ptr)
> free(ptr)

> I am wondering doing frequent malloc and free is ok? Will this
> fragment the memory?

> My development environment is GCC and Linux. But I'd be happy to know
> about the implementation in windows also.

Sounds like a region allocator might help you out a bit:

http://groups.google.com/group/comp.lang.c/browse_frm/thread/97a65e8d...


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
pete  
View profile  
 More options Nov 17 2010, 2:03 pm
Newsgroups: comp.lang.c
From: pete <pfil...@mindspring.com>
Date: Wed, 17 Nov 2010 14:03:55 -0500
Local: Wed, Nov 17 2010 2:03 pm
Subject: Re: Using malloc/free in a tight loop

Navaneeth wrote:

> > Unless the strings are extremely long then a modern computer isn't
> > going to notice allocating 5000 of them.

> Thanks. My strings are not long. May be 20 characters.

> So in general, can malloc fragment the memory? I was expecting the
> modern implementations will be clever enough to reduce this problem.
> Isn't that true?

I suspect that it might be true
of the ancient implementations as well.

Section 8.7 of the 1978 edition of The C Programming Language
describes a storage allocator system in which the freeing function,
merges the freed regions of memory, for the allocating function to use.

--
pete


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Chris M. Thomasson  
View profile  
 More options Nov 17 2010, 9:13 pm
Newsgroups: comp.lang.c
From: "Chris M. Thomasson" <cris...@charter.net>
Date: Wed, 17 Nov 2010 18:13:57 -0800
Local: Wed, Nov 17 2010 9:13 pm
Subject: Re: Using malloc/free in a tight loop
"Chris M. Thomasson" <cris...@charter.net> wrote in message
news:xeUEo.35278$qg3.27264@newsfe14.iad...

Or, you might be able to do something really simple like:

http://codepad.org/kTdySzAe
______________________________________________________________
#include <stdio.h>
#include <stdlib.h>

#define ITERS 128U

#define RAND() ((size_t)rand())

#define STRING_MAX_SIZE 64U

#define STRING_ALPHABET "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

#define STRING_ALPHABET_GET() \
    STRING_ALPHABET[RAND() % (sizeof(STRING_ALPHABET) - 1)]

char*
string_populate(
    char* string,
    size_t* psize
){
    size_t i;
    size_t size = RAND() % STRING_MAX_SIZE;
    *psize = size;

    string[size] = '\0';

    for (i = 0; i < size; ++i)
    {
        string[i] = STRING_ALPHABET_GET();
    }

    return string;

}

void
string_process(
    char const* string,
    size_t size
){
    size_t i;

    for (i = 0; i < size; ++i)
    {
        putchar(string[i]);
    }

    puts("\n_______________________________________________\n");

}

int
main(void)
{
    char buffer[STRING_MAX_SIZE];
    size_t i;

    for (i = 0; i < ITERS; ++i)
    {
        size_t size;
        char* string = string_populate(buffer, &size);
        string_process(string, size);
    }

    puts("\n\n___________________________________________\n"
         "The program has completed.\n");

    getchar();

    return 0;

}

______________________________________________________________

No need to call `malloc()/free()' and friends... ;^)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nobody  
View profile  
 More options Nov 19 2010, 1:54 pm
Newsgroups: comp.lang.c
From: Nobody <nob...@nowhere.com>
Date: Fri, 19 Nov 2010 18:54:42 +0000
Local: Fri, Nov 19 2010 1:54 pm
Subject: Re: Using malloc/free in a tight loop

On Wed, 17 Nov 2010 04:00:25 -0800, Navaneeth wrote:
> So in general, can malloc fragment the memory? I was expecting the
> modern implementations will be clever enough to reduce this problem.
> Isn't that true?

They will try to minimise it, but it isn't always possible.

E.g.:

        for (n = n0; n <= n1; n++) {
                void *p = malloc(n);
                void *q = malloc(1);
                ...
                free(p);
        }

Each new allocation will be 1 byte too large to fit into the space
for the previous allocation. Some of them will fit due to the previous
allocation having been oversized, but they can't all fit.

If you really need to prevent fragmentation, you need a relocating
allocator, which implies a different interface. With malloc(), any
returned pointer remains valid until free()d; the implementation can't
just move blocks as it chooses.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gordon Burditt  
View profile  
 More options Nov 20 2010, 6:15 am
Newsgroups: comp.lang.c
From: gordonb.b5...@burditt.org (Gordon Burditt)
Date: Sat, 20 Nov 2010 05:15:35 -0600
Local: Sat, Nov 20 2010 6:15 am
Subject: Re: Using malloc/free in a tight loop

>has to create a string and pass that into other methods for
>processing. This string length will only be known at the runtime.

>So in each iteration, this is what happens.

>ptr = malloc
>process(ptr)
>free(ptr)

>I am wondering doing frequent malloc and free is ok? Will this
>fragment the memory?

If you only hold on to one string at a time, it is unlikely you
will fragment memory much.  Each time through the loop, you will
go back to the state of everything freed.  A "reasonable" allocator
will recombine adjacent allocations when both are freed.

For programs like you describe, it might be less time-consuming to
keep re-using the single chunk of memory as long as it's big enough,
and realloc() it larger when needed.

Here's an example of (rather horribly) fragmenting memory:

You have only 1000TB of memory for string data.  You allocate
10,000,000,000,000 strings averaging 100 bytes each (including
malloc() bookkeeping overhead) and maybe ranging in size from 50-150
bytes.  (Perhaps these are DNA fragments from various species.)
This just about fills up the memory.  Now free() all but 10,000,000
of them, but which of the strings gets free()d and which gets kept
depends somewhat randomly on external data.  There is only about
0.001 TB of memory still allocated, but the strings are scattered
all over, so good luck trying to allocate 1TB of contiguous memory,
even though you've got 1,000 times that in free memory.

Oh, yes, malloc()/free() cannot move allocated memory unless there's
a sophisticated garbage collector that can track down all references
to memory in use, including in files.

Some allocators try to avoid fragmentation by taking large requests
from one pool, and tiny requests from another, perhaps with a whole
range of power-of-two-sized blocks.  That can be effective, but not
in the above case, since all of the strings were roughly the same
size.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »