DosSubAllocMem/DosSubFreeMem

[Iron Spring Software]

Everyone probably knows this, but FWIW here's an interesting problem
where I ran out of heap on OS/2 for an application that worked for Linux.

I think the heap is being checkerboarded with a mix of large and and
small allocations. I pulled out some large allocations (one or more
chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
everything worked. I assume this is because the OS/2 heap is
preallocated while the Linux heap just grows.

[Steven Levine]

On Mon, 4 Apr 2022 21:56:12 UTC, Iron Spring Software
<Peter...@Yahoo.com> wrote:

Hi Peter,

> small allocations. I pulled out some large allocations (one or more
> chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
> everything worked. I assume this is because the OS/2 heap is
> preallocated while the Linux heap just grows.

Not really. It's related to how address space is handled by OS/2.
You don't mention if the runtime for your application supports
allocating from upper memory. The lack of this capability can cause
an app to run out of heap space pretty quickly if the has large-ish
memory needs.

The kLIBC and OpenWatcom runtimes can allocate heap space from upper
memory. The VAC runtimes cannot.

One thing many developers don't understand is that DosAllocMem
allocates address space in 64K multiples. This is why well written
runtimes allocate heap chunks in 64K multiples. This avoids wasting
address space.

There are a few ancient runtimes that have a fixed heap size. The
Modula/2 runtime comes to mind. However, this is an exception. Most
runtimes will allow the heap to expand until the process runs out of
address space.

Steven

--
---------------------------------------------------------------------
Steven Levine <ste...@earthlink.bogus.net>
DIY/ArcaOS/Warp etc. www.scoug.com www.arcanoae.com www.warpcave.com
---------------------------------------------------------------------

[Marcel Mueller]

Am 04.04.22 um 23:56 schrieb Iron Spring Software:

> Everyone probably knows this, but FWIW here's an interesting problem
> where I ran out of heap on OS/2 for an application that worked for Linux.

An application using DosSubAllocMem likely /never/ works on Linux. ;-)

> I think the heap is being checkerboarded with a mix of large and and
> small allocations. I pulled out some large allocations (one or more
> chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
> everything worked. I assume this is because the OS/2 heap is
> preallocated while the Linux heap just grows.

No, you simply fragmeted the address space. There are some patterns that
force this. In this case the memory requirement may raise with O(n²).

Normally, I recommend to prefer the C library allocators over using the
kernel funktions. They are not intended to be used heavily.
The glibc allocator performs quite well with millions of allocations and
frees.

If you really want to write your own allocator, this can be a hard work.
And it is always a trade off between speed and fill factor.

AFAIR DosSub... is simply a linked list with not further optimizations.
If you want to deal with this functions anyway you could use different
pools for different object sizes - small, medium, large. And for very
large allocations you could allocate directly from the kernel. However,
keep in mind that the latter might cause the TLB of your CPU to overflow
which can cause a significant performance impact. So I do not recommend
it unless you have really large chunks of e.g. 1MB.


Marcel

[Peter Flass]

Marcel Mueller <news.5...@spamgourmet.org> wrote:
> Am 04.04.22 um 23:56 schrieb Iron Spring Software:
>> Everyone probably knows this, but FWIW here's an interesting problem
>> where I ran out of heap on OS/2 for an application that worked for Linux.
>
> An application using DosSubAllocMem likely /never/ works on Linux. ;-)

Iron Spring PL/I started out as an OS/2 application. It ‘s pure PL/I, and
doesn’t use an C code, so I had to port an allocator for the Linux version.
I could probably move it to OS/2 if there are no other options.

>
>> I think the heap is being checkerboarded with a mix of large and and
>> small allocations. I pulled out some large allocations (one or more
>> chunks of 32K) and used DosAllocMem and DosFreeMem for these, and
>> everything worked. I assume this is because the OS/2 heap is
>> preallocated while the Linux heap just grows.
>
> No, you simply fragmeted the address space. There are some patterns that
> force this. In this case the memory requirement may raise with O(n²).

Same thing in different words. This goes back to the beginning of the
compiler many years ago. I had no idea how much memory would be required
for the parse table of a large program, so QandD was a a linked list of 32K
blocks. There are better ways to handle this. The compiler works fine, but
I reused the parser logic for the preprocessor, so there’s a different
pattern of memory use that breaks the system.

>
> Normally, I recommend to prefer the C library allocators over using the
> kernel funktions. They are not intended to be used heavily.
> The glibc allocator performs quite well with millions of allocations and
> frees.
>

See above. I don’t want any dependencies on the C library.

> If you really want to write your own allocator, this can be a hard work.
> And it is always a trade off between speed and fill factor.
>
> AFAIR DosSub... is simply a linked list with not further optimizations.
> If you want to deal with this functions anyway you could use different
> pools for different object sizes - small, medium, large. And for very
> large allocations you could allocate directly from the kernel. However,
> keep in mind that the latter might cause the TLB of your CPU to overflow
> which can cause a significant performance impact. So I do not recommend
> it unless you have really large chunks of e.g. 1MB.

Thanks. I tried looking in the debug handbooks, etc., but couldn’t find any
detail on DosSub.. I miss having source to look at. I guess I never
stressed it before. As I said, it was QandD that always just worked, but
it’s very inefficient and could probably be improved with minimal effort.

>
>
> Marcel
>



--
Pete

[Steven Levine]

On Sun, 17 Apr 2022 14:04:17 UTC, Peter Flass <peter...@yahoo.com>
wrote:

Hi Peter,

>
> Thanks. I tried looking in the debug handbooks, etc., but couldn t find any
> detail on DosSub.. I miss having source to look at. I guess I never
> stressed it before. As I said, it was QandD that always just worked, but
> it s very inefficient and could probably be improved with minimal effort.

Switch to allocating 64K blocks with DosAllocMem and you will run out
of memory much less frequently. See my previous post.