Will 'free' return the memory Immediately to the OS ?

[Raj Pashwar]

As per subject-line, thank-you

[Ian Collins]

On 05/16/12 09:34 AM, Raj Pashwar wrote:
> As per subject-line, thank-you

Why couldn't you just write

Will 'free' return the memory Immediately to the OS ?

Assuming there is an OS, Yes and no. It depends on how the OS works.
Memory is typically mapped to a process in pages and the allocator
allocates memory from these. Most operating systems will leave unused
pages mapped to the process unless they are needed elsewhere.

--
Ian Collins

[James Kuyper]

On 05/15/2012 05:34 PM, Raj Pashwar wrote:
> As per subject-line, thank-you

The malloc family is often implemented by allocating large blocks from
the operating system, and handing out only a small portion of each block
at a time. free() cannot possibly return a block of memory to the OS
until all allocations from that block are free()'d, so you shouldn't
expect an immediate increase in memory available to other processes.

Even when free() could return a block of memory to the OS, it might not
choose to do so; the standard doesn't require it.

On many systems, it doesn't matter much whether or not the memory is
returned; if a sufficiently large block of memory remains unused for a
sufficiently long period of time, the OS may have means of detecting
that fact and swapping it out to disk, transparently to the user. The
memory needn't even have been free()d yet.

[Keith Thompson]

Raj Pashwar <raj1...@hotmail.NOSPAM.com> writes:
> As per subject-line, thank-you

Please include the question in the body of your message:

Will 'free' return the memory Immediately to the OS ?

Here's what the C standard says:

The free function causes the space pointed to by ptr to be
deallocated, that is, made available for further allocation.

It's unspecified whether "made available for further allocation"
means that it's returned to the OS. Typically it isn't.
Some implementations might return free()d memory to the OS, but
it's not practical to do so in all cases. For example, if you have
three allocated chunks of memory that happen to be contiguous, and
you free() the second one, it's likely to be difficult to remove
it from the current program's ownership while keeping its neighbors.

--
Keith Thompson (The_Other_Keith) ks...@mib.org <http://www.ghoti.net/~kst>
Will write code for food.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Ben Pfaff]

The FAQ says:

7.25: I have a program which mallocs and later frees a lot of memory,
but I can see from the operating system that memory usage
doesn't actually go back down.

A: Most implementations of malloc/free do not return freed memory
to the operating system, but merely make it available for future
malloc() calls within the same program.

[Eric Sosman]

On 5/15/2012 5:34 PM, Raj Pashwar wrote:
> As per subject-line, thank-you

As per signature, you're welcome.

--
Eric Sosman
eso...@ieee-dot-org.invalid
Question 7.25 in <http://www.c-faq.com/>

[David T. Ashley]

On Tue, 15 May 2012 21:34:57 +0000 (UTC), Raj Pashwar
<raj1...@hotmail.NOSPAM.com> wrote:

>As per subject-line, thank-you

Generally, it depends on the "division of labor" between the run-time
library and the operating system.

Some operating systems (in the old days before PCs used hardware
memory management and all that) had OS function calls that would
handle small blocks quite efficiently. malloc() and free() were just
empty wrappers, and dynamically-allocated memory belonging to the OS
and the application were freely mixed. Some embedded systems may
still have this kind of division of labor.

Nowadays, on a modern PC, the landscape is different.

No OS is likely to have the capability to hand the run-time library a
chunk of memory smaller than the page size supported by the memory
management hardware. I don't know what this figure is nowadays, but
I'd guess maybe between 256K and 1M. You typically wouldn't be able
to allocate from the operating system, for example, a 30-byte chunk of
memory.

The run-time library is likely to take these large chunks allocated by
the operating system and subdivide them so it can hand out smaller
chunks via malloc().

There is no guarantee that the algorithm used by malloc() and free()
would allow any memory to be returned to the OS until every block had
been free()'d.

There is no reason usually to return memory to the OS until the
process terminates. Most programs that use dynamic memory use a bunch
and then terminate ... it wouldn't benefit the average program to
return anything to the OS until the program terminates ... just adds
complexity and decreases speed.

DTA

[Stephen Sprunk]

On 16-May-12 12:29, David T. Ashley wrote:
> No [modern PC] OS is likely to have the capability to hand the run-time

> library a chunk of memory smaller than the page size supported by the
> memory management hardware. I don't know what this figure is nowadays,
> but I'd guess maybe between 256K and 1M.

On x86, the page sizes are 4kB (normal) and 2MB/4MB (large), with the
latter depending on whether PAE is enabled. Other architectures vary.

> You typically wouldn't be able to allocate from the operating system,
> for example, a 30-byte chunk of memory.

Nor would you want them to, even if they were able, because that would
greatly increase the number of (relatively expensive) system calls.

> The run-time library is likely to take these large chunks allocated by
> the operating system and subdivide them so it can hand out smaller
> chunks via malloc().

Exactly; it's more efficient to do it that way because it reduces the
number of (relatively expensive) system calls.

> There is no guarantee that the algorithm used by malloc() and free()
> would allow any memory to be returned to the OS until every block had
> been free()'d.

Depending on the system calls available, it may not even be possible.
For instance, the traditional Unix brk()/sbrk() interface dictates a
fixed starting address and variable ending address for the entire heap.
The newer anonymous mmap()/munmap() interface allows allocating and
deallocating individual pages or groups of pages. I don't know how the
Windows interface works, but I'd guess it's similar to the latter.

S

--
Stephen Sprunk "God does not play dice." --Albert Einstein
CCIE #3723 "God is an inveterate gambler, and He throws the
K5SSS dice at every possible opportunity." --Stephen Hawking

[David T. Ashley]

On Wed, 16 May 2012 18:36:57 -0500, Stephen Sprunk
<ste...@sprunk.org> wrote:

>On 16-May-12 12:29, David T. Ashley wrote:
>> No [modern PC] OS is likely to have the capability to hand the run-time
>> library a chunk of memory smaller than the page size supported by the
>> memory management hardware. I don't know what this figure is nowadays,
>> but I'd guess maybe between 256K and 1M.
>
>On x86, the page sizes are 4kB (normal) and 2MB/4MB (large), with the
>latter depending on whether PAE is enabled. Other architectures vary.

I believe you, but I'm somewhat surprised at this figure. Hardware
memory management is usually a tradeoff between silicon complexity,
complexity of setting up the hardware by the software, and
granularity.

I figured with a modern PC having maybe 4G of memory, you might want
to divide the memory into maybe 1,000 - 10,000 pages, so I would have
figured page sizes in the range of 512K - 4M. 4K surprises me.

I'm too lazy to look it all up. The x86 world at the low levels has
an entry cost. Right now I'm in the ARM world.

DTA

[Stephen Sprunk]

On 17-May-12 08:47, David T. Ashley wrote:
> On Wed, 16 May 2012 18:36:57 -0500, Stephen Sprunk
> <ste...@sprunk.org> wrote:
>> On 16-May-12 12:29, David T. Ashley wrote:
>>> No [modern PC] OS is likely to have the capability to hand the run-time
>>> library a chunk of memory smaller than the page size supported by the
>>> memory management hardware. I don't know what this figure is nowadays,
>>> but I'd guess maybe between 256K and 1M.
>>
>> On x86, the page sizes are 4kB (normal) and 2MB/4MB (large), with the
>> latter depending on whether PAE is enabled. Other architectures vary.
>
> I believe you, but I'm somewhat surprised at this figure. Hardware
> memory management is usually a tradeoff between silicon complexity,
> complexity of setting up the hardware by the software, and
> granularity.
>
> I figured with a modern PC having maybe 4G of memory, you might want
> to divide the memory into maybe 1,000 - 10,000 pages, so I would have
> figured page sizes in the range of 512K - 4M. 4K surprises me.

Remember, these page sizes were dictated by the i386, when 4kB was still
a fairly big chunk of memory. Most PCs at the time had far less than
4MB of RAM. Virtual memory was mostly about swapping to disk, so the
smaller the pages, the faster a page fault is handled. Also, smaller
pages meant you could get more of the active set in RAM instead of
constantly swapping in and out large pages that were mostly inactive.

The only real cost to small pages is the need for large TLBs to avoid
walking the page table more than once per page, but in most cases that
simply isn't a problem these days for most workloads--or at least not
enough of a problem to justify the incredibly high cost of changing the
ISA. For certain workloads, eg. database servers, there are special OS
APIs to get large pages. This can be a major performance improvement,
but only if you have enough RAM to prevent them from being evicted.

[Kaz Kylheku]

On 2012-05-17, David T Ashley <das...@gmail.com> wrote:
> On Wed, 16 May 2012 18:36:57 -0500, Stephen Sprunk
><ste...@sprunk.org> wrote:
>
>>On 16-May-12 12:29, David T. Ashley wrote:
>>> No [modern PC] OS is likely to have the capability to hand the run-time
>>> library a chunk of memory smaller than the page size supported by the
>>> memory management hardware. I don't know what this figure is nowadays,
>>> but I'd guess maybe between 256K and 1M.
>>
>>On x86, the page sizes are 4kB (normal) and 2MB/4MB (large), with the
>>latter depending on whether PAE is enabled. Other architectures vary.
>
> I believe you, but I'm somewhat surprised at this figure. Hardware
> memory management is usually a tradeoff between silicon complexity,
> complexity of setting up the hardware by the software, and
> granularity.
>
> I figured with a modern PC having maybe 4G of memory, you might want
> to divide the memory into maybe 1,000 - 10,000 pages, so I would have
> figured page sizes in the range of 512K - 4M. 4K surprises me.

At 1000 pages, the external fragmentation would be bad. The nice thing
about small page sizes is that it is easier for a process to return
unneeded pages to the operating system. It's easier to find a contiguous
page-aligned 4K chunk of memory which is not in use and free it, than to
find such a 4Mb chunk of memory.

But, never mind that, the real problem is that address spaces do not share
pages, and neither do distinct memory mappings within address spaces.

Firstly, look at your "ps" or "task manager" and see how many processes are
running. You need at least that many pages because you would never allocate two
processes into the same page: you lose protection (as well as the abilility to
link the base executables to run at the same fixed virtual address).

Then, within a process, it is customary to use different page ranges for
different memory mappings, like stack (main one and per-thread), executable, or
shared library. For large malloc requests, it is good to use a function like
mmap. When free is called on that, it can be entirely returned to the operating system.

Every single one of these uses would eat into your allotment of 1000 pages,
quickly gobbling up all of it.

> I'm too lazy to look it all up. The x86 world at the low levels has
> an entry cost. Right now I'm in the ARM world.

4K pages sizes are common in the ARM world, the MIPS, world.

Linux/MIPS only goes up to 64K page sizes (at kernel compile time)
though the R8000 TLB goes from 4K to 16M.

It doesn't make sense to use very large page sizes globally.

--
If you ever need any coding done, I'm your goto man!

[Robert Wessel]

On Wed, 16 May 2012 13:29:48 -0400, David T. Ashley
<das...@gmail.com> wrote:

>On Tue, 15 May 2012 21:34:57 +0000 (UTC), Raj Pashwar
><raj1...@hotmail.NOSPAM.com> wrote:
>
>>As per subject-line, thank-you
>
>Generally, it depends on the "division of labor" between the run-time
>library and the operating system.
>
>Some operating systems (in the old days before PCs used hardware
>memory management and all that) had OS function calls that would
>handle small blocks quite efficiently. malloc() and free() were just
>empty wrappers, and dynamically-allocated memory belonging to the OS
>and the application were freely mixed. Some embedded systems may
>still have this kind of division of labor.
>
>Nowadays, on a modern PC, the landscape is different.
>
>No OS is likely to have the capability to hand the run-time library a
>chunk of memory smaller than the page size supported by the memory
>management hardware. I don't know what this figure is nowadays, but
>I'd guess maybe between 256K and 1M. You typically wouldn't be able
>to allocate from the operating system, for example, a 30-byte chunk of
>memory.

The vast majority of systems use 4-64KB pages (and 4KB pages are most
common of all). ARM allowed (in the past) smaller pages. Many
systems also allow large pages, usually by omitting the lowest level
of the page table hierarchy (x86, for example, allows 2MB or 4MB
pages, depending on which format page tables you're using), but these
tend to be used mainly to replace large contiguous allocations of
"normal" sized pages, and particularly for areas some subject to
paging or allocation (for example, the OS areas are fairly commonly
mapped with large pages).

As for OS's allowing small allocations... Obviously you need to
define the OS's boundaries rather carefully. For example Windows
Heap*() functions are defined as part of the OS API, but act more like
what you'd expect malloc() to do (and are implemented largely in user
space - in fact some Windows malloc-like implementations map simply to
HeapAlloc). As another example, zOS has the "GETMAIN" macro as its
primary memory allocation API, which also handles small chunks.

>The run-time library is likely to take these large chunks allocated by
>the operating system and subdivide them so it can hand out smaller
>chunks via malloc().
>
>There is no guarantee that the algorithm used by malloc() and free()
>would allow any memory to be returned to the OS until every block had
>been free()'d.

True.

>There is no reason usually to return memory to the OS until the
>process terminates. Most programs that use dynamic memory use a bunch
>and then terminate ... it wouldn't benefit the average program to
>return anything to the OS until the program terminates ... just adds
>complexity and decreases speed.

That's sometimes true, sometimes not. Consider an application that
lets you load a large file to work on for a while, and then lets you
work on something else. Returning that memory to the OS would only be
polite.

In any event, many implementations do things like handle small
allocations internally (by subdividing larger chunks allocated from
the OS), and large allocations as OS calls. So often freeing a large
allocated chunk actually does return the storage to the OS.

In short, it's impossible to generalize.

[Keith Thompson]

An x86 system might have hundreds of processes running simultaneously,
and some of those are going to be quite small. A large page size would
substantially increase total memory usage.

[Eric Sosman]

<off-topic> Consider the silicon cost of things like a TLB, and
the run-time cost of a TLB miss, and realize that a large page size
can reduce both at the same time. Many systems nowadays support several
page sizes simultaneously, just for better TLB use. </off-topic>

--
Eric Sosman
eso...@ieee-dot-org.invalid

[Stephen Sprunk]

On 17-May-12 11:54, Kaz Kylheku wrote:
> On 2012-05-17, David T Ashley <das...@gmail.com> wrote:
>> On Wed, 16 May 2012 18:36:57 -0500, Stephen Sprunk
>> <ste...@sprunk.org> wrote:
>>> On 16-May-12 12:29, David T. Ashley wrote:
>>>> No [modern PC] OS is likely to have the capability to hand the run-time
>>>> library a chunk of memory smaller than the page size supported by the
>>>> memory management hardware. I don't know what this figure is nowadays,
>>>> but I'd guess maybe between 256K and 1M.
>>>
>>> On x86, the page sizes are 4kB (normal) and 2MB/4MB (large), with the
>>> latter depending on whether PAE is enabled. Other architectures vary.
>>
>> I believe you, but I'm somewhat surprised at this figure. Hardware
>> memory management is usually a tradeoff between silicon complexity,
>> complexity of setting up the hardware by the software, and
>> granularity.
>>
>> I figured with a modern PC having maybe 4G of memory, you might want
>> to divide the memory into maybe 1,000 - 10,000 pages, so I would have
>> figured page sizes in the range of 512K - 4M. 4K surprises me.
>
> At 1000 pages, the external fragmentation would be bad. The nice thing
> about small page sizes is that it is easier for a process to return
> unneeded pages to the operating system. It's easier to find a contiguous
> page-aligned 4K chunk of memory which is not in use and free it, than to
> find such a 4Mb chunk of memory.

Of course. Also, swapping a 4MB page to/from disk is a lot slower and
may require evicting another 4MB page that is still (partially) active.
Smaller pages allows faster swapping and keeping more of the active set
in memory, at the expense of more page tables and larger TLBs.

> But, never mind that, the real problem is that address spaces do not share
> pages, and neither do distinct memory mappings within address spaces.

What? I thought that shared libraries and COW for forking relied on
using the same physical pages in multiple processes--though in the
former case they might be mapped to different virtual locations via
different sets of page tables.

> Every single one of these uses would eat into your allotment of 1000 pages,
> quickly gobbling up all of it.

I think he meant that a single process's address space shouldn't require
more than 1,000 (or 10,000) pages, not a global limit on the number of
pages for all processes.

[Joe keane]

In article <i20ar71d8m427086m...@4ax.com>,

David T. Ashley <das...@gmail.com> wrote:

>I figured with a modern PC having maybe 4G of memory, you might want
>to divide the memory into maybe 1,000 - 10,000 pages, so I would have
>figured page sizes in the range of 512K - 4M. 4K surprises me.

The OS can have a larger page size. It's easy to make 16 KB pages on
hardware that is designed for 4 KB pages. But not the other way around.

The size of a 'segment' [unit that the memory management requests more
space from the OS] would be more like 2 MB.

[Stephen Sprunk]

On 18-May-12 14:44, Joe keane wrote:
> In article <i20ar71d8m427086m...@4ax.com>,
> David T. Ashley <das...@gmail.com> wrote:
>> I figured with a modern PC having maybe 4G of memory, you might want
>> to divide the memory into maybe 1,000 - 10,000 pages, so I would have
>> figured page sizes in the range of 512K - 4M. 4K surprises me.
>
> The OS can have a larger page size. It's easy to make 16 KB pages on
> hardware that is designed for 4 KB pages. But not the other way around.

You can _emulate_ larger pages by always handling several of them at the
same time, but the hardware does what the hardware does.

I question whether this is a good idea. Any reduction in page faults
due to loading multiple pages at a time is likely to be offset by an
increase in page faults due to evicting multiple pages at a time. So,
overall, you spend more time swapping pages to/from disk for no gain.

> The size of a 'segment' [unit that the memory management requests more
> space from the OS] would be more like 2 MB.

"Segment" has a very different meaning on many architectures.

Also, I would expect malloc() et al to request/release memory from the
OS in whatever multiple of the page size it found convenient, not some
arbitrarily large (or small, for some workloads) amount like 2MB.

[Stephen Sprunk]

OTOH, as soon as you have to start swapping those large pages to and
from disk, you blow away the gains from more efficient TLB usage.

Except for workloads where a single process randomly accesses enough
active pages to overflow the TLBs _and_ the machine has enough RAM that
those pages can be locked, large pages don't seem to make sense. It
apparently occurred to the i386 designers to allow for that case, and
hear it's common for large database servers, but that's about it.

It seems advantageous to have intermediate page sizes, but that also
adds hardware complexity--and forces the OS to decide what page size to
use for what, which it will inevitably get wrong.

[Eric Sosman]

On 5/18/2012 6:41 PM, Stephen Sprunk wrote:
> On 17-May-12 20:05, Eric Sosman wrote:
>>
>> <off-topic> Consider the silicon cost of things like a TLB, and
>> the run-time cost of a TLB miss, and realize that a large page size
>> can reduce both at the same time. Many systems nowadays support several
>> page sizes simultaneously, just for better TLB use.</off-topic>
>
> OTOH, as soon as you have to start swapping those large pages to and
> from disk, you blow away the gains from more efficient TLB usage.

True. "Seven decimal orders of magnitude" has a charming
way of obliterating small gains and losses.

(In other words: As soon as you swap, you've already blown
your whole performance story. CPU ~= 2GHz, disk ~= 0.0000001GHz,
CPU runs ~20,000,000 cycles while waiting for one count them one
disk operation. If each CPU cycle were one breath at an adult
rate of <20 respirations per minute, one disk operation would
take upwards of two years. Hell, by the time the disk finishes
the cache is stony cold; the CPU has "forgotten" what it was
doing. What were *you* doing on May 18, 2010, and what did you
have for dinner on that day?)

> Except for workloads where a single process randomly accesses enough
> active pages to overflow the TLBs _and_ the machine has enough RAM that
> those pages can be locked, large pages don't seem to make sense. It
> apparently occurred to the i386 designers to allow for that case, and
> hear it's common for large database servers, but that's about it.
>
> It seems advantageous to have intermediate page sizes, but that also
> adds hardware complexity--and forces the OS to decide what page size to
> use for what, which it will inevitably get wrong.

Yes, it will inevitably be wrong -- sometimes. Unlike us,
who are inevitably wrong nearly always.

--
Eric Sosman
eso...@ieee-dot-org.invalid

[Stephen Sprunk]

On 18-May-12 19:19, Eric Sosman wrote:
> On 5/18/2012 6:41 PM, Stephen Sprunk wrote:
>> On 17-May-12 20:05, Eric Sosman wrote:
>>>
>>> <off-topic> Consider the silicon cost of things like a TLB, and
>>> the run-time cost of a TLB miss, and realize that a large page size
>>> can reduce both at the same time. Many systems nowadays support several
>>> page sizes simultaneously, just for better TLB use.</off-topic>
>>
>> OTOH, as soon as you have to start swapping those large pages to and
>> from disk, you blow away the gains from more efficient TLB usage.
>
> True. "Seven decimal orders of magnitude" has a charming
> way of obliterating small gains and losses.
>
> (In other words: As soon as you swap, you've already blown
> your whole performance story. CPU ~= 2GHz, disk ~= 0.0000001GHz,
> CPU runs ~20,000,000 cycles while waiting for one count them one
> disk operation. If each CPU cycle were one breath at an adult
> rate of <20 respirations per minute, one disk operation would
> take upwards of two years. Hell, by the time the disk finishes
> the cache is stony cold; the CPU has "forgotten" what it was
> doing. What were *you* doing on May 18, 2010, and what did you
> have for dinner on that day?)

There's that problem too, but I was thinking more along the lines of 4MB
pages taking ~1000 times as long as 4kB pages to evict and reload.

Also, the only time I've heard of TLBs being a real-world problem is in
the case of database servers, which apparently spend most of their time
following long chains of pointers hither and yon around the address
space. Large pages reduce the number of TLB entries needed in that case
by a factor of ~1000, preventing TLB overflow. But that case only
arises when all of the pages are locked in RAM; if they aren't, page
faults to repopulate the TLBs are be a minor problem compared to disk
access, per your argument.

[Ben Pfaff]

Stephen Sprunk <ste...@sprunk.org> writes:

> There's that problem too, but I was thinking more along the lines of 4MB
> pages taking ~1000 times as long as 4kB pages to evict and reload.

I doubt that that is true, though. I'd suspect that 4 MB pages
take less than 100 times as long as 4 kB pages to evict and
reload, maybe much less.

I tried to find some measurements online to support this point,
but couldn't quickly find anything relevant.

[Joe keane]

In article <jp6ic1$lp6$1...@dont-email.me>,

Stephen Sprunk <ste...@sprunk.org> wrote:
>I question whether this is a good idea.

It is a good idea. These are -disk drives-.

If your drive has a decent buffer, the time to get additional pages
could be no more than to transfer data over the bus.

[Ian Collins]

If only life were that simple! Consider a diskless client.

The point you snipped:

"The OS can have a larger page size. It's easy to make 16 KB pages on
hardware that is designed for 4 KB pages. But not the other way around."

A reasonable OS will allow multiple page sizes, in part to minimise the
use of MMU slots when applications use large amounts of memory. One
negative side effect is the bigger the page size an application uses,
the more likely it is to hit an out of memory condition due to
fragmentation.

--
Ian Collins

[Robert Wessel]

On Tue, 22 May 2012 13:45:21 -0700, Ben Pfaff <b...@cs.stanford.edu>
wrote:

There are several disk IOPS calculators floating around, which will
give at least an approximate result. One is:

http://www.wmarow.com/strcalc/


But basically, for approximately current drives, 4MB random I/Os
instead of 4KB random I/Os will knock your IOPS down by a factor of
about 15, while increasing your data throughput about 60 fold.

FWIW, 4MB is a bit large for "random" I/Os, as that's typically larger
than most current track sizes, and I/Os require several revolutions to
perform. Keeping the I/Os to under a track will substantially
increase the IOPS rate relative to the baseline for the drive
(typically going from 1KB to 256KB I/Os will only halve your IOPS
rate).

The exact results are dependent on the I/O mix (which should be close
to 50/50 for paging) and the physical parameters of the disk drives
and arrays they're in.

SSDs behave rather differently, although paging to those is not often
a good idea.

[Robert Wessel]

On Wed, 23 May 2012 11:12:43 +1200, Ian Collins <ian-...@hotmail.com>
wrote:

I think his point is that most MMUs (and their associated TLBs) are
very restrictive in the sizes of pages they support, and while you can
certainly emulate 16KB pages on an MMU supporting only 4KB pages, each
"16KB" page will take four TLB entries to map (and four TLB misses to
fill). So emulating larger pages does not help with TLB pressures.

Even when different page sizes are supported, TLB usage is not
necessarily simply split between the sizes. For example, there have
been x86 implementations (which support 4KB and 2/4MB pages) where the
large pages had a separate, and quite small, TLB, and it was easy to
thrash the large-page TLB. Some ISAs that supported several (small)
pages sizes have had implementations that did not directly support all
the sizes in the TLB, and would use several TLB slots to map the
bigger pages (leading to similar issues as OS emulation of larger
pages).

[Robert Miles]

I'd expect that to depend heavily on the rotation speed of the hard
drive. Eviction and reloading should each take at least an average
of half a rotation, plus however long it takes to read or write that
much data.

The fastest common speed for hard drives for home computers is
7200 RPM, although 10000 RPM is available but not common yet.

Some hard drives have memory caches to help with this if a read
request comes soon enough after a write request for the same data.

Also, SSD drives don't use a rotating disk - they only use a slower
interface to memory chips, and are therefore much faster than hard
drives.

[Stephen Sprunk]

AFAIK, 10,000rpm drives, 7,200rpm drives, caching drives, and SSD drives
were not available in the consumer x86 market at the time the i386 was
being designed, which is when 4kB/4MB page sizes were chosen.

And, if you study the x86 (and even x64) page table system, there's
really no good way to add an intermediate page size without basically
revamping the entire system--which means OS developers would have to
throw away all of the paging code and experience they've built up over
the last two and a half decades of x86 development.