[FFI] malloc, maximum number of bytes

[Peter Samarin]

Hi all,

I am using Racket v7.3, Linux x86_64 version.

I want to allocate large (tens of gigabytes) chunks of memory with Racket's malloc and use them for some BLAS/LAPACK routines.

The downloaded version of Racket lets me allocate only 2^31 "atomic" bytes.

2^31+1 bytes produce an "out of memory" error, even though I still have plenty of memory left (much more than requested 2GB) on my machine.

In raw mode, I can go to 2^33 bytes, but not 2^34. My machine has 16GB RAM with swap disabled.

Is that a bug or is the amount of allocated atomic memory limited to 2^31 bytes?

Here is the code to reproduce the problem:

(require ffi/unsafe)

(malloc (expt 2 31) _uint8 'atomic) ; => #<cpointer>
(malloc (add1 (expt 2 31)) _uint8 'atomic) ; => out of memory

(define mem (malloc (expt 2 33) _uint8 'raw)) ; => #<cpointer>
(memset mem 0 #x00 (expt 2 33) _uint8) ; ok

--Peter

[Matthew Flatt]

There should be no special limit on 'atomic memory, and I haven't been
able to replicate this behavior, yet.

Can you say more about how you're running (e.g., in DrRacket versus
command-line `racket`)? Also, although it's difficult to see how it
could matter, which Linux distribution do you use?

[Peter Samarin]

I just rerun the example on the commandline and it works just fine with larger numbers of bytes.

I use emacs and racket-mode by Greg Hendershot.

I will check how that starts racket at some other time.

Thanks Matthew!

[Matthew Flatt]

Ah, it's possible that Racket mode sets a limit on Racket's memory use,
just like DrRacket does. That kind of memory limit (enforced within
Racket, instead of by the OS) is based on GCable memory, so it doesn't
limit 'raw allocation.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/80d40912-6f55-4906-a0f9-b10cd78b
> 423c%40googlegroups.com.

[Peter Samarin]

Okay, got to the bottom of it. racket-mode uses racket-memory-limit that is set to 2048 by default.

Setting it to 0 makes it unlimited.

> email to racket...@googlegroups.com.