hw.physmem/hw.realmem question

[Daniel Braniss]

Hi,
to run some tests, I reduced the physical memory by setting hw.physmem,
which got me to do some comparisons, and the more I looked around the
more confused I got.

for example, this host has has 32G of physical memory
from dmesg:
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-STABLE #26: Thu Jun 20 16:00:00 IDT 2013
danny@rnd:/home/obj/rnd/r+d/stable/9/sys/HUJI amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz (3300.06-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping
= 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,
CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,C
X16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,A
VX>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 34359738368 (32768 MB)
avail memory = 32191340544 (30700 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL DCSRADON>

and from sysctl:
hw.physmem: 34284916736
hw.usermem: 32964923392
hw.realmem: 36507222016

after setting
hw.physmem=16G

from dmesg:
real memory = 34359738368 (32768 MB)
avail memory = 13999382528 (13350 MB)

and sysctl:

hw.physmem: 14957563904
hw.usermem: 10094678016
hw.realmem: 17179869184

from the numbers, I can assume that realmem is the real physical memory,
(or whatever is set in hw.physmem),
if so, where did almost 2G go? (realmem - physmen)

the only info I found so far is:
from loader(8):

hw.physmem Limit the amount of physical memory the system will use.
By default the size is in bytes, but the k, K, m, M, g and
G suffixes are also accepted and indicate kilobytes,
megabytes and gigabytes respectively. An invalid suffix
will result in the variable being ignored by the kernel.

what is physmem and realmem, and what's the relationship - if any - between
them?

cheers,
danny



_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Chris Torek]

>for example, this host has has 32G of physical memory ...
>[snip - dmesg:]

>real memory = 34359738368 (32768 MB)

>avail memory = 32191340544 (30700 MB)
>[snip]

>and from sysctl:
>hw.physmem: 34284916736
>hw.usermem: 32964923392
>hw.realmem: 36507222016
>
>after setting
> hw.physmem=16G
>
>from dmesg:
>real memory = 34359738368 (32768 MB)
>avail memory = 13999382528 (13350 MB)
>
>and sysctl:
>
> hw.physmem: 14957563904
> hw.usermem: 10094678016
> hw.realmem: 17179869184
>
>from the numbers, I can assume that realmem is the real physical memory,
>(or whatever is set in hw.physmem),

I think there are three numbers (though one is not always a single
number) of interest, which I believe were the original intent of
the three sysctl values ("real", "phys", and "user" memory,
specially called out in sysctl.h under the "HW" section). These
are:

- The RAM present in hardware. This would be 32 GB on
your particular system.

On some machines, RAM is not even close to physically
contiguous, e.g., there might be 4 GB of RAM starting at
address 8 TB, then 32 GB of RAM starting at address 16 TB,
then 1 TB of RAM starting at address 24 TB, on a machine where
each DIMM slot is simply located at each 8-TB position (with
"architectural room" for up to 8 TB per slot), and "slot 0"
reserved for boot ROMs (hence physical space starting at 8
TB). These enormous "holes" in the physical space should (my
opinion) be reflected in whatever interface gets you the RAM
map. That makes this one not a "single number". Of course,
if you can't or won't show the holes, you can make up a single
number.

(On x86 systems you don't have holes this big, although you do
have the historical "I/O space" windows, e.g., that annoying
gap at 640kB :-) . Worse, there's always a half-GB gap at
3.5GB, although some memory controllers work around this.
See:

http://lists.freebsd.org/pipermail/freebsd-amd64/2005-August/005849.html

for example.)

- The amount of "usable" RAM for the OS. This subtracts off
any spaces reserved by boot ROMs / BIOSes / what-have-you, and
in the case of (e.g.) FreeBSD-9 on amd64, the 1 TB direct-map
limit (which you must take care of manually with the loader's
hw.physmem setting).

This is what "phys mem" should be and mostly is. If you boot
a machine with 1.5 TB of RAM but the OS is limited to 1 TB,
hw.physmem should be 1 TB minus a bit for the BIOS, etc.

- The amount of memory left after subtracting more or less fixed
kernel resources, such as kernel text and data (including loaded
modules), page table pages, "vm_page" array data structures, and
so on. This will shift over time.

This is what "user mem" is, more or less -- to be exact, it's:

ctob(physmem - cnt.v_wire_count)

where "ctob" is "clicks to bytes" which is really "pages to
bytes", as these things count in terms of pages (4K at a time,
on the x86). The "wire count" is the number of pages that are
not page-in/out-able, so subtracting that from physmem gives
you the number of pageable, hence user-use-able, pages.

There's a fourth number, which is not really very useful, but I
need to describe for the below:

- The highest useable physical address in the system (or
really, just after that -- the same way if the machine is to
count to 4, you go "0 1 2 3" and that gives you 4). This
is called "Maxmem" in amd64/amd64/machdep.c (and pmap.c).

Note that the printf output:

real memory = <number> (<number> MB)

generally comes from the amount reported by the BIOS, which is
different yet again! My box with 8GB says:

real memory = 8589934592 (8192 MB)

but on my machine, Maxmem is 8.5 GB, which also shows up in
sysctl (more on that in a moment):

hw.realmem: 9126805504

I have the Intel memory controller that "moves up" the shadowed
(by PCI hole) RAM at 3.5 GB. If I were to set hw.physmem to 8
GB in my loader.conf, I would actually give up half a gigabyte.

So, on to this:

>if so, where did almost 2G go? (realmem - physmem) ...

>what is physmem and realmem, and what's the relationship - if any
>- between them?

"hw.realmem" is a snapshot of the value of Maxmem, before a few
adjustments are made. If you have 32 GB of physical RAM and you
have not limited it (and you don't have the remapping noted
below), "realmem" will be 32 GB. If you *have* limited it,
realmem captures the limit (which I think is wrong -- the snapshot
should happen earlier, at the least).

"hw.physmem" results from counting up useable pages. After
finding Maxmem, which gives you "maximum valid address plus 1",
the machdep.c code goes through all the segments -- segments being
stuff like "64k to 640k", with the first 64k off limits because
BIOSes tend to munch on it and the 640k limit due to the ISA hole
-- and checks that each page in that segment for useability. If
the page is good, it's added to physmem.

Your 2 GB (542555 pages, to be exact) is space eaten up by your
BIOS and architectural holes, including the large PCI hole. Your
BIOS and motherboard etc may (or may not) allow you to remap some
of your "hidden" or "shadowed" RAM (out of the PCI hole),
increasing the boot-time value of Maxmem, and hence also
increasing both "hw.realmem" and actual, useable pages.

Note: the x86's architectural holes still use up some dedicated
kernel memory, even with shadowed-memory remapping: if addresses
from zero to 8.5 GB are valid (as they are on my box), the kernel
allocates enough "vm_page" data structures to have one for all 8.5
GB, even though there's a .5 GB PCI hole with no RAM behind it.
Those pages are marked "not here, never use these" -- but they
still take sizeof(struct vm_page) bytes (120 bytes) to represent.
512 MB of hole = 512*1024*1024 / 4096 = 131072 pages, which means
the kernel is using 15728640 bytes (15 MB) to track this empty
area. So my remapping hardware gains me 512 MB and then the
kernel loses 15, for a net of 497 MB recovered.

Chris

[Wojciech Puchar]

> AMD Features2=0x1<LAHF>
> TSC: P-state invariant, performance statistics
> real memory = 34359738368 (32768 MB)
> avail memory = 32191340544 (30700 MB)

2GB memory "disappears" too even when you don't set anything.

i asked such a question for other machine some time ago without much
answer.


in your laptop it may be shared graphics memory reserved by chipset

still on my dell server

real memory = 34359738368 (32768 MB)

avail memory = 33166921728 (31630 MB)

i have over 1GB unavailable and it doesn't have shared graphics memory.

it would be nice to be able to look exactly how memory is used.

[Daniel Braniss]

Thanks Chris!
It will take me some time do fully digets all this!,
but at least the picture is less murky.
danny

[Sergey Kandaurov]

On 3 July 2013 01:45, Wojciech Puchar <woj...@wojtek.tensor.gdynia.pl> wrote:
>> AMD Features2=0x1<LAHF>
>> TSC: P-state invariant, performance statistics
>> real memory = 34359738368 (32768 MB)
>> avail memory = 32191340544 (30700 MB)
>
>
> 2GB memory "disappears" too even when you don't set anything.
>
> i asked such a question for other machine some time ago without much answer.
>
>
> in your laptop it may be shared graphics memory reserved by chipset
>
> still on my dell server
>
>
> real memory = 34359738368 (32768 MB)
> avail memory = 33166921728 (31630 MB)
>
> i have over 1GB unavailable and it doesn't have shared graphics memory.
>
> it would be nice to be able to look exactly how memory is used.

On amd64 about 3% is cut on startup for page structures, see vm_page_startup().

--
wbr,
pluknet