malloc error

[denis.be...@gmail.com]

In Clips 6.3 or 6.4, I get the following malloc error message, several times:

Clips(17442,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

There's plenty of RAM still available and this doesn't prevent the process to continue.

What happens? And can the results be corrupted by this problem?

[CLIPS Support]

That message is coming from the malloc function, not CLIPS. If malloc returns a NULL pointer to CLIPS, CLIPS will retry the malloc call after releasing some cached memory. If the malloc call returns NULL again, CLIPS will release all cached memory and then retries the malloc call. If the call fails again, CLIPS will print an out of memory error message, stop execution, and exit. So I don't know why malloc would print an error message like that if it wasn't returning NULL back to CLIPS. If CLIPS didn't exit when malloc returned NULL, it would crash when it tried writing anything to that memory address. 

[denis.be...@gmail.com]

The fact is, the 1st malloc message appears when lots of RAM are available. After it, CLIPS continues to work. 

Then the same malloc message appears again when lots of RAM are still available, and CLIPS doesn’t exit.

I tried the same problem with a different setting, putting less stress on memory. A different malloc message appears:

Clips(17612,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

The error is perfectly reproducible (with a different 1st number - process number):

Clips(23275,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

It occurs at exactly the same point of processing, when CLIPS uses less than 9 GB of RAM and no virtual memory (64 available).

CLIPS continues to  run and to claim memory (18.8 GB right now)

I therefore asked the question to ChatGPT and I got the following answer, which seems quite relevant:

It’s Apple’s libmalloc telling you it ran out of virtual address space for a given allocation zone.

---

What it means

• On macOS, malloc doesn’t just grab memory linearly — it uses zones (like the nano allocator, scalable allocator, etc.).

• Large objects are carved into segments within those zones.

• The message

malloc: Failed to allocate segment from range group - out of space

• appears when the allocator couldn’t reserve virtual memory for a new segment in the requested zone.

So it’s not about CLIPS freeing something invalid — it’s genuinely an out-of-memory (or out-of-address-space) situation at the allocator level.

---

Why it shows up now in Tahoe

• Apple has changed the way zones are managed in recent macOS versions (especially with 64-bit nano allocator and address space randomization).

• Tahoe may use a different range group granularity (larger chunks reserved at once), so when CLIPS asks for more memory, the allocator can run out of contiguous space sooner.

• Even if physical RAM isn’t exhausted, the allocator’s range groups can hit a ceiling (fragmentation, per-zone limits).

---

Interaction with CLIPS’s error handling

• CLIPS only knows if malloc() returned NULL.

• In your run, before CLIPS can even see a NULL, libmalloc itself prints that diagnostic because the failure happened at the segment-reservation level.

• Then CLIPS likely does its retry logic — but you see Apple’s message first.

---

✅ So in plain terms:

On macOS Tahoe, CLIPS is asking for memory, but the allocator can’t extend its zone (segment reservation failed). That triggers Apple’s own malloc error message, before CLIPS does its retry/exit.

---

[denis.be...@gmail.com]

.

This is a follow-up to my previous message. I ran the same process on an old Mac and the malloc error doesn't appear.

This seems to confirm ChatGPT's  analysis about Tahoe.

[denis.be...@gmail.com]

Having received no response, I wonder whether you are working on the problem or whether you consider it's just a system message that can be discarded as irrelevant as it doesn't lead CLIPS to fail.

Partly for the fun of using ChatGPT (and knowing how unreliable it is on precise technical questions, even elementary ones), I had some "talk" with it (him? her?). After eliminating suggestions that would require giving up the default GENERIC setting or that would decrease overall performance or that wouldn't take into account the possibly very large numbers of tiny facts or partial matches in the RETE network, I finally got the following answer. 

ChatGPT also provided C code for implementing its proposal. I'll post it  if anyone is interested. I have some notions of C but I'm not a C programmer and I wonder whether this code is worth anything.

1. Short summary

On macOS 15 (Tahoe), CLIPS sometimes fails with:

Clips(17442,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

This does not happen on older macOS versions. The new allocator in Tahoe (libmalloc) manages “range groups” differently, and CLIPS’s heavy use of millions of small allocations (facts, RETE nodes) can exhaust these groups before malloc even returns NULL.

Workaround

Disabling the Nano allocator (MallocNanoZone=0) avoids the crash, but may reduce performance, since Nano is optimized for small objects (which CLIPS uses heavily).

Proposed solution

Add a custom pool allocator inside CLIPS for tiny objects (facts, RETE nodes, symbols):

• Reserve a large block of memory at startup.

• Partition into size classes (16B, 32B, 64B, …).

• Manage allocations via free lists.

• Use system malloc only for large allocations.

Benefits

• Works across all macOS versions.

• Avoids Tahoe allocator failures.

• Likely improves performance (O(1) alloc/free, better memory locality).

• Portable: same code on Windows, Linux, macOS.

  
  

  2. Detailed proposal

  Problem Analysis

  • Tahoe’s allocator imposes stricter limits on segment allocation (“range groups”).

  • CLIPS relies on system malloc for millions of tiny objects.

  • MallocNanoZone=0 avoids crashes but disables the Nano allocator, which reduces allocation efficiency for small objects.

  Option A – Custom Pool Allocator (Recommended)

  • At startup, reserve a large contiguous block (e.g., 256 MB).

  • Divide it into size classes, maintain free lists per class.

  • Provide clips_pool_alloc(size) / clips_pool_free(ptr) wrappers for hot-path allocations (facts, RETE nodes, symbols).

  • Fallback to system malloc for large objects.

  Advantages

  • Eliminates Tahoe-specific allocation failures.

  • Maintains high performance for millions of tiny allocations.

  • Portable and fully under CLIPS control.

  • Predictable memory usage and low fragmentation.

  Implementation Sketch

void *clips_pool_alloc(size_t size) { /* allocate from pool or fallback malloc */ }
void clips_pool_free(void *ptr) { /* return to pool or free */ }
void clips_pool_init(size_t reserve_bytes); /* call at startup */

Testing Recommendations

• Micro-benchmark: 1M allocations/frees at 16–64B sizes.

• Compare pool allocator vs. system malloc vs. Nano disabled (MallocNanoZone=0).

• Validate that previous Tahoe “range group” failures no longer occur.

Optional Refinements

• Make the pool thread-safe if needed (pthread_mutex), or keep single-threaded for minimal overhead.

• Adjust initial reserve size and slab size for memory tuning.