malloc error

[denis.be...@gmail.com]

In Clips 6.3 or 6.4, I get the following malloc error message, several times:

Clips(17442,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

There's plenty of RAM still available and this doesn't prevent the process to continue.

What happens? And can the results be corrupted by this problem?

[CLIPS Support]

That message is coming from the malloc function, not CLIPS. If malloc returns a NULL pointer to CLIPS, CLIPS will retry the malloc call after releasing some cached memory. If the malloc call returns NULL again, CLIPS will release all cached memory and then retries the malloc call. If the call fails again, CLIPS will print an out of memory error message, stop execution, and exit. So I don't know why malloc would print an error message like that if it wasn't returning NULL back to CLIPS. If CLIPS didn't exit when malloc returned NULL, it would crash when it tried writing anything to that memory address. 

[denis.be...@gmail.com]

The fact is, the 1st malloc message appears when lots of RAM are available. After it, CLIPS continues to work. 

Then the same malloc message appears again when lots of RAM are still available, and CLIPS doesn’t exit.

I tried the same problem with a different setting, putting less stress on memory. A different malloc message appears:

Clips(17612,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

The error is perfectly reproducible (with a different 1st number - process number):

Clips(23275,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

It occurs at exactly the same point of processing, when CLIPS uses less than 9 GB of RAM and no virtual memory (64 available).

CLIPS continues to  run and to claim memory (18.8 GB right now)

I therefore asked the question to ChatGPT and I got the following answer, which seems quite relevant:

It’s Apple’s libmalloc telling you it ran out of virtual address space for a given allocation zone.

---

What it means

• On macOS, malloc doesn’t just grab memory linearly — it uses zones (like the nano allocator, scalable allocator, etc.).

• Large objects are carved into segments within those zones.

• The message

malloc: Failed to allocate segment from range group - out of space

• appears when the allocator couldn’t reserve virtual memory for a new segment in the requested zone.

So it’s not about CLIPS freeing something invalid — it’s genuinely an out-of-memory (or out-of-address-space) situation at the allocator level.

---

Why it shows up now in Tahoe

• Apple has changed the way zones are managed in recent macOS versions (especially with 64-bit nano allocator and address space randomization).

• Tahoe may use a different range group granularity (larger chunks reserved at once), so when CLIPS asks for more memory, the allocator can run out of contiguous space sooner.

• Even if physical RAM isn’t exhausted, the allocator’s range groups can hit a ceiling (fragmentation, per-zone limits).

---

Interaction with CLIPS’s error handling

• CLIPS only knows if malloc() returned NULL.

• In your run, before CLIPS can even see a NULL, libmalloc itself prints that diagnostic because the failure happened at the segment-reservation level.

• Then CLIPS likely does its retry logic — but you see Apple’s message first.

---

✅ So in plain terms:

On macOS Tahoe, CLIPS is asking for memory, but the allocator can’t extend its zone (segment reservation failed). That triggers Apple’s own malloc error message, before CLIPS does its retry/exit.

---

[denis.be...@gmail.com]

.

This is a follow-up to my previous message. I ran the same process on an old Mac and the malloc error doesn't appear.

This seems to confirm ChatGPT's  analysis about Tahoe.

[denis.be...@gmail.com]

Having received no response, I wonder whether you are working on the problem or whether you consider it's just a system message that can be discarded as irrelevant as it doesn't lead CLIPS to fail.

Partly for the fun of using ChatGPT (and knowing how unreliable it is on precise technical questions, even elementary ones), I had some "talk" with it (him? her?). After eliminating suggestions that would require giving up the default GENERIC setting or that would decrease overall performance or that wouldn't take into account the possibly very large numbers of tiny facts or partial matches in the RETE network, I finally got the following answer. 

ChatGPT also provided C code for implementing its proposal. I'll post it  if anyone is interested. I have some notions of C but I'm not a C programmer and I wonder whether this code is worth anything.

1. Short summary

On macOS 15 (Tahoe), CLIPS sometimes fails with:

Clips(17442,0x1f8ca0800) malloc: Failed to allocate segment from range group - out of space

This does not happen on older macOS versions. The new allocator in Tahoe (libmalloc) manages “range groups” differently, and CLIPS’s heavy use of millions of small allocations (facts, RETE nodes) can exhaust these groups before malloc even returns NULL.

Workaround

Disabling the Nano allocator (MallocNanoZone=0) avoids the crash, but may reduce performance, since Nano is optimized for small objects (which CLIPS uses heavily).

Proposed solution

Add a custom pool allocator inside CLIPS for tiny objects (facts, RETE nodes, symbols):

• Reserve a large block of memory at startup.

• Partition into size classes (16B, 32B, 64B, …).

• Manage allocations via free lists.

• Use system malloc only for large allocations.

Benefits

• Works across all macOS versions.

• Avoids Tahoe allocator failures.

• Likely improves performance (O(1) alloc/free, better memory locality).

• Portable: same code on Windows, Linux, macOS.

  
  

  2. Detailed proposal

  Problem Analysis

  • Tahoe’s allocator imposes stricter limits on segment allocation (“range groups”).

  • CLIPS relies on system malloc for millions of tiny objects.

  • MallocNanoZone=0 avoids crashes but disables the Nano allocator, which reduces allocation efficiency for small objects.

  Option A – Custom Pool Allocator (Recommended)

  • At startup, reserve a large contiguous block (e.g., 256 MB).

  • Divide it into size classes, maintain free lists per class.

  • Provide clips_pool_alloc(size) / clips_pool_free(ptr) wrappers for hot-path allocations (facts, RETE nodes, symbols).

  • Fallback to system malloc for large objects.

  Advantages

  • Eliminates Tahoe-specific allocation failures.

  • Maintains high performance for millions of tiny allocations.

  • Portable and fully under CLIPS control.

  • Predictable memory usage and low fragmentation.

  Implementation Sketch

void *clips_pool_alloc(size_t size) { /* allocate from pool or fallback malloc */ }
void clips_pool_free(void *ptr) { /* return to pool or free */ }
void clips_pool_init(size_t reserve_bytes); /* call at startup */

Testing Recommendations

• Micro-benchmark: 1M allocations/frees at 16–64B sizes.

• Compare pool allocator vs. system malloc vs. Nano disabled (MallocNanoZone=0).

• Validate that previous Tahoe “range group” failures no longer occur.

Optional Refinements

• Make the pool thread-safe if needed (pthread_mutex), or keep single-threaded for minimal overhead.

• Adjust initial reserve size and slab size for memory tuning.

[CLIPS Support]

Sorry, this one slipped between the cracks and I first thought from your prior message that this was just a weird message that macOS occasionally spit out but CLIPS wasn't failing. I use xcode for development of CLIPS and iOS apps and I periodically get weird messages popping up in the console that the online consensus usually ends up being to ignore them. If you're also getting crashes in some situations, we had a similar problem decades ago with some operating systems that allowed a very minimal number of malloc allocations. There was some code in CLIPS 6.24 and prior versions which implemented large block allocations from the OS in large 32K blocks and then these blocks were split by CLIPS for smaller memory requests. I'll look into moving that code back into version 6.4.

[CLIPS Support]

I restored the BLOCK_MEMORY functionality to the 63x and 64x branches in the SourceForge SVN repository. Recompile the source with the BLOCK_MEMORY flag set to 1 in the setup.h source file to see if that resolves the issue you're seeing.

On Tuesday, September 30, 2025 at 12:38:48 AM UTC-5 denis.be...@gmail.com wrote:

[denis.be...@gmail.com]

It works for the cases I found previously: the error message disappears.
However, a much worse problem appears: memory growth is considerably slowed when lots of it are required. 

It seems the interaction between CLIPS and Apple memory managers doesn't work well.

The only examples I have for this new problem are linked to my own software. However, if you want to observe it by yourself:

- download CSP-Rules from GitHub (https://github.com/denis-berthier/CSP-Rules-V2.1), 

- rename directory CSP-Rules-V2.1-master to CSP-Rules-V2.1

- edit file "SlitherRules-V2.1-config.clp" by changing " (defglobal ?*CSP-Rules* = "/Users/berthier/Projects/CSP-Rules/") to  (defglobal ?*CSP-Rules* = "/xxxxxxx/CSP-Rules/"), where "CSP-Rules" is the directory above CSP-Rules-V2.1. Don't change anything else.

Now you can batch SlitherRules-V2.1-config.clp into CLIPS and launch the example by command:

(solve

15 15

. . . 3 . . 3 3 . . 1 . . . . 

2 . 1 . . . . 2 . . . 2 . 2 3 

1 3 2 3 . . 2 2 3 2 . 1 3 . . 

3 . . . . . . . . . 1 . . . . 

3 2 . 0 . 2 . 1 1 3 . 2 0 2 2 

. . 3 2 . . 2 1 . . . . . . . 

2 2 . . 3 . . 3 . . . 1 2 1 . 

. . 3 1 . 2 . . 3 . . . 3 2 . 

. 2 . 2 . . . 2 . 1 3 . 1 . 1

. 3 2 3 2 . 2 0 3 . 3 . 3 . 2

2 2 . . . . 2 . 2 . . . . . . 

. . 1 . 2 1 . . . . . . . . 2

. . 2 2 2 3 . . 2 . 1 . 2 3 2

3 . 3 . 2 1 . 3 . . . 2 2 3 . 

. . . . . . . 0 . . . . . . . 

)

Before the change, you'll get the final results:

nb-facts = <Fact-454316>

computer = MacBookPro 16'' M1Max 2021 3.2GHz, 64GB LPDDR5, MacOS 26.1

init-time = 0.47s, solve-time = 1m 30.89s, total-time = 1m 31.36s

nb-facts=<Fact-454316>

Quasi-Loop max length = 240

Colours effectively used

and you can check that 10.64 GB have been used.

After the change, and if you wait for an eternity (which I haven't done), you will eventually get the same solution. But after 10 mins or so, we are stil very far from the solution and the memory used is only 500 MB.

[CLIPS Support]

Not too surprising. The block memory code was written forty years ago to allow CLIPS to run on some systems where the malloc library function only allowed a small fixed number of allocation requests. It was never optimized for performance. 

[denis.be...@gmail.com]

Got it. Thanks for the additions allowing me to try it. In 2025, it is probably impossible to compete with the built-in system-optimised malloc function.

So, just to make sure, could you please confirm that the best thing to do now is:

- suppose ChatGPT analysis of what's happening is correct,

- therefore, let MacOS send its error message and let CLIPS continue its work because this message (and its cause) has no impact on what CLIPS does?

[CLIPS Support]

Yes, I think if it's running to completion, you should just ignore the messages. If it's running out of memory because the nano allocator makes it run out sooner because of the large number of small allocations, then you'll need to disable the nano allocator. There's also the option of using a third party malloc dropin like mimalloc. I'd think that the built-in system-optimized malloc would normally be the best bet, but maybe there's one that's better for your specific case.