VirtualAlloc
qfel
boundary. Because I didn't reserve the whole range of pages, the rest of
them (allocation granularity - 1 page) will be marked as free. Can they be
used in any way before decommitting previously allocated page? Looks like
there is no way to commit them, because of allocation granularity.
Arkady Frenkel
can commit the rest any time
Arkady
"qfel" <q_...@aster.pl> wrote in message news:fbtobi$dn$1...@inews.gazeta.pl...
qfel
void *ptr=VirtualAlloc(null,4096,MEM_COMMIT,PAGE_READWRITE);
VirtualAlloc((char*)ptr+4096,4096,MEM_COMMIT,PAGE_READWRITE);
Page at (char*)ptr+4096 is marked as free (with allocation base of
0x00000000)
Kellie Fitton
Hi,
You should use the following APIs when appropriate and check
to see why the function call had failed, simply by checking the
returned error code and respond accordingly:
GetLastError()
FormatMessage()
http://msdn2.microsoft.com/en-us/library/ms679360.aspx
http://msdn2.microsoft.com/en-us/library/ms679351.aspx
Kellie.
Alexander Grigoriev
"qfel" <q_...@aster.pl> wrote in message
news:fbuolr$nrl$1...@inews.gazeta.pl...
qfel
reserving whole 64kb range), are remaining pages just unusable (until I free
what I allocated)?
Alexander Grigoriev
"qfel" <q_...@aster.pl> wrote in message
news:fc0dt0$pli$1...@inews.gazeta.pl...
Kellie Fitton
wrote:
> qfel schrieb:
>
> > And that's what my question was about - after small allocation
> > (without reserving whole 64kb range), are remaining pages just
> > unusable (until I free what I allocated)?
>
> (there's even an API-call where you can query the granularity; does anyone
> recall the name?),
Hi,
You can use the following API to retrieve the minimum size of a large
page:
GetLargePageMinimum()
http://msdn2.microsoft.com/en-us/library/aa366568.aspx
Kellie.
Alexander Grigoriev
page is initially mapped to a zero page, they have a page allocated either
in a pagefile or a real one. If there's no pages left, VirtualAlloc will
fail.
"Elcaro Nosille" <Elcaro....@mailinator.com> wrote in message
news:46e451c3$0$4528$9b4e...@newsspool3.arcor-online.net...
> qfel schrieb:
>> And that's what my question was about - after small allocation
>> (without reserving whole 64kb range), are remaining pages just
>> unusable (until I free what I allocated)?
>
> At the first point, reserve the whole block up to the next 64kB-boundary
> (there's even an API-call where you can query the granularity; does anyone
> recall the name?), then commit as much as you need and if you need more,
> commit the rest.
> But this isn't really necessary; you could commit the whole block at the
> first place because the NT-based Windows-systems physically assign unique
> physical pages not until you try to write to a commited page the first
> time.
> Before that point, all pages to which there weren't any write-accesses be-
> fore are assigned to the same physical zero-page which is read-only (on
> SMP-system there's a per processor zero-page and the allocating processor
> determines which zero-page ist assigned commited pages).
Pavel Lebedinsky [MSFT]
> Unlike Linux/Unix, Windows doesn't do overcommit. Even though a committed
> page is initially mapped to a zero page, they have a page allocated either
> in a pagefile or a real one.
Committed pages are not initially mapped to anything. The page tables
necessary to describe a newly committed range may not even be
allocated yet, or if they happen to be allocated, the PTEs for the new
pages will be zeroed until the pages are accessed for the first time.
>> But this isn't really necessary; you could commit the whole block at the
>> first place because the NT-based Windows-systems physically assign unique
>> physical pages not until you try to write to a commited page the first
>> time.
>> Before that point, all pages to which there weren't any write-accesses
>> be-
>> fore are assigned to the same physical zero-page which is read-only (on
>> SMP-system there's a per processor zero-page and the allocating processor
>> determines which zero-page ist assigned commited pages).
I don't think this is true. If an application commits two pages then reads
from both of them it will get two separate physical pages, each filled with
zeroes. On NUMA systems these physical pages will be allocated from
the node corresponding to the current (or, starting with Vista, ideal)
processor of the thread that performs the reads.
--
This posting is provided "AS IS" with no warranties, and confers no
rights.
robert...@yahoo.com
wrote:
> > Unlike Linux/Unix, Windows doesn't do overcommit.
>
> program that tests this behaviour with a little benchmark.
> This benchmark allocates a block of 16MB and repeatedly *reads*
> this block and outputs how many clock-cycles this read-process
> took. After each iteration it writes to an additional page so
> that in the end, all pages which are read in each iteration
> are pages which all were written before.
> If my Win-XP woud assign all pages of a freshly reserved and
> commited block would assin to the same zero-page as I assumed,
> reading this block would be extremely fast because all pages
> of this block could fit into the physically tagged L1-cache
> of today's x86-CPUs. So the time to read the block woud raise
> after another page has been filled through writes (so that
> unique physical pages are assigned to that block).
> Because there could be other tasks which could run while this
> benchmark runs, I did two things to have a half-way accurate
> gauging: First I assigned realtime-priority to the task and
> to the thread (so this task could only run at adminstrator
> privileges); second, every pass in which I read the block
> is performed ten times and the fastest time ist taken.
> But there's one thing I'm curious about: Some time ago I read
> in a Web-article that some Windows-version does this assignment
> of zero-pages (even the assignment of per-processor zero-pages).
> Of course it looks like if this isn't really true, but maybe
> the newer Windows-versions Sever 2003 and Vista do that actu-
> ally. Can anyone compile the code an run it under Vista or
> W2K3s? If anyone needs compiled code, I'll provide it.
>
> #include <windows.h>
> #include <stdio.h>
>
> long long __stdcall memrd( void *p, size_t size );
>
> int main()
> {
> size_t const RESERVE = 16 * 1024 * 1024,
> PAGE_SIZE = 4096;
> void *pBlock;
> size_t touch;
> DWORDLONG dwlFastestPass;
>
> SetPriorityClass( GetCurrentProcess(), REALTIME_PRIORITY_CLASS );
> SetThreadPriority( GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL );
>
> pBlock = VirtualAlloc( NULL, RESERVE, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE );
>
> for( touch = 0; touch < RESERVE; touch = (touch != 0) ? (touch * 2) : PAGE_SIZE )
> {
> int passes;
> DWORDLONG dwlPassTime;
>
> if( touch != 0 )
> memset( (char *)pBlock + touch, -1, PAGE_SIZE );
>
> for( passes = 10, dwlFastestPass = (DWORDLONG)-1;
> pass != 0; pass-- )
> if( (dwlPassTime = (DWORDLONG)memrd( pBlock, RESERVE )) < dwlFastestPass )
> dwlFastestPass = dwlPassTime;
>
> printf( "%I64d: %I64d\n", (__int64)touch, (__int64)dwlPassTime );
> }
>
> VirtualFree( pBlock, 0, MEM_RELEASE );
>
> return 0;
>
> }
>
> __declspec(naked)
> long long __stdcall memrd( void *p, size_t size )
> {
> __asm
> {
> push esi
> push edi
>
> mov esi, [esp + 12]
> mov edi, [esp + 16]
> and edi, 0xFFFFFFFC
> add edi, esi
>
> rdtsc
> mov ebx, eax
> mov ecx, edx
>
> jmp readLoopIn
> readLoop:
> add eax, [esi]
> add esi, 4
> readLoopIn:
> cmp esi, edi
> jne readLoop
>
> rdtsc
> sub eax, ebx
> sbb edx, ecx
>
> pop edi
> pop esi
> ret 8
> }
>
>
>
> }
First, this doesn't compile, I assume you mean "passes" in the second
two parts of the for() statement.
Second, this either fails to run correctly or crashes depending on
which optimization level you turn on, even after initializing a few
obviously uninitialized variables. So there's a latent bug which I'm
insufficiently motivated to find that leaves any results from this
program in question.
Third it's not clear exactly what you think you're testing. Assigning
a new page at first *write* to an overcommitted are is *not* the only
way to do it. In fact, it's probably more common to do it at first
*access.* Also, I'm not sure how the write pattern you've chosen will
test the condition you want to test.
In any event, if you take out the "passes" loop, you'll see that the
first read through the allocated block takes considerably longer than
all subsequent accesses, lending support to the notion that pages are
assigned at that time.
robert...@yahoo.com
wrote:
> robertwess...@yahoo.com schrieb:
>
> > First, this doesn't compile, I assume you mean "passes" in the
> > second two parts of the for() statement. Second, this either
> > fails to run correctly or crashes depending on which optimization
>
> I'm really sorry, but I made some changes in the code at the posting.
> I needed sleep a this point but I didn't go to bed ...
>
> > So there's a latent bug which I'm insufficiently motivated to
> > find that leaves any results from this program in question.
>
>
> (snipped)
It still runs incorrectly with -O2 or -Ox (VS2005).
> > Assigning a new page at first *write* to an overcommitted are is
> > *not* the only way to do it. In fact, it's probably more common
> > to do it at first *access.*
>
> Of course, but it is like I read in the article, the pages would get a
> zero-page on the first read-access.
Which is not inconsistent with allocating the (actual) page at first
access. IOW, before the first access a part of the VirtualAlloc'd
address space has no assigned page, but is marked by the OS as
zeroed. At the first access the OS traps, a real page is acquired,
zeroed, and mapped into the address space.
> > Also, I'm not sure how the write pattern you've chosen will test
> > the condition you want to test.
>
> It is.
>
> > In any event, if you take out the "passes" loop, you'll see that the
> > first read through the allocated block takes considerably longer than
> > all subsequent accesses, lending support to the notion that pages are
> > assigned at that time.
>
> This isn't really a bug, but it would make more sense to iterate to the
> pages in the magnitude of pages than DWORDs.
Huh? The problem with the "passes" loop is that if pages are being
allocated at first access, as opposed to first write, your find-the-
minimum-time process will discard the only meaningful datum from the
run. Your test could (at best) work only if your assumption as to
when real pages are allocated is correct, and you've generalized that
assumption into the assumption that memory overcommit has to work in a
specific way (and it doesn't).
If you just want to test if Windows will actually overcommit memory,
just VirtualAlloc enough memory to fill your page file (you may need
to fix the size of the page file first, and you may have to run
several processes).
Alexander Grigoriev
<robert...@yahoo.com> wrote in message
news:1189672392.7...@y42g2000hsy.googlegroups.com...
Alexander Grigoriev
"Elcaro Nosille" <Elcaro....@mailinator.com> wrote in message
news:46e905f2$0$7690$9b4e...@newsspool2.arcor-online.net...
> robert...@yahoo.com schrieb:
>
>>>> So there's a latent bug which I'm insufficiently motivated to
>>>> find that leaves any results from this program in question.
>>> Ok, here's the code which runs.
>
>>> (snipped)
>
>> It still runs incorrectly with -O2 or -Ox (VS2005).
>
> I'm pretty sure that's an optimizer-bug in this case.
> That wouldn't be the first I discovered.
>
>>> Of course, but it is like I read in the article, the
>>> pages would get a zero-page on the first read-access.
>
>> Which is not inconsistent with allocating the (actual) page at first
>> access.
>
> Of course not, but I could discover whether XP-SP2 does implement
> this zero-pages which are disambiguatet to unique pages on writes.
>
>> IOW, before the first access a part of the VirtualAlloc'd address
>> space has no assigned page, ...
>
> You might be right here.
>
>> ... but is marked by the OS as zeroed.
>
> Whatcelse could zeroed mean other than being assigned to a page
> with zeroes (unique or not)? And a not assigned page to be trapped
> by a page-fault on every acess isn't what I would call zeroed.
>
>
>>> This isn't really a bug, but it would make more sense to iterate
>>> to the pages in the magnitude of pages than DWORDs.
>
>> Huh? The problem with the "passes" loop is that if pages are being
>> allocated at first access, as opposed to first write, your find-the-
>> minimum-time process will discard the only meaningful datum from the
>> run.
>
> Of course, but that's not what I wanted to investigate.
>
>> Your test could (at best) work only if your assumption
>> as to when real pages are allocated is correct, ...
>
> Right.
>
>> ... and you've generalized that assumption into the assumption
>> that memory overcommit has to work in a specific way (and it
>> doesn't).
>
> Yes, at the first point; but not in the posting you've fupped last.
>
>> If you just want to test if Windows will actually overcommit
>> memory, just VirtualAlloc enough memory to fill your page
>> file (you may need to fix the size of the page file first,
>> and you may have to run several processes).
>
> That's what I did some time ago and Windows doesn't swap if I reserve
> and commit 1,5GB in one piece. But I couldn't test that again today
> without removing memory because I've got more physical memory than
> a single process could allocate.
Chris Thomasson
> robert...@yahoo.com schrieb:
>
>>>> So there's a latent bug which I'm insufficiently motivated to
>>>> find that leaves any results from this program in question.
>>> Ok, here's the code which runs.
>
>>> (snipped)
>
>> It still runs incorrectly with -O2 or -Ox (VS2005).
>
Humm. I am not sure I would blame the compiler just yet... IMHO, you most
likely have a subtle bug in your code if it produces undefined behavior
(e.g., race-condition) when you crank up the optimization level.
> That wouldn't be the first I discovered.
[...]
What were some of the other optimizer bugs you discovered in vs2005? BTW,
don't worry about posting a correction; I won't think your crazy:
http://groups.google.com/group/comp.arch/msg/5e47ab949628387c
Chris Thomasson
news:_9mdnZrGTc2k03Tb...@comcast.com...
> What were some of the other optimizer bugs you discovered in vs2005? BTW,
> don't worry about posting a correction; I won't think your crazy:
>
> http://groups.google.com/group/comp.arch/msg/5e47ab949628387c
Just to clarify: Elcaro claims I have a mental disorder because I document
corrections to some of the pseudo-code I post on USENET.
Chris Thomasson
>> Chris Thomasson schrieb:
>> BTW, don't worry about posting a correction; I won't think your crazy:
>>
>> http://groups.google.com/group/comp.arch/msg/5e47ab949628387c
>
> me in a frequency where others clearly see, that I wouldn't have
> a stable advertence; like it is in your case. *g*
PLONK!
Ken Hagan
<Elcaro....@mailinator.com> wrote:
> Oh no! Your affinity to fixed action patterns, i.e. you transfer
> possible problems of other cases to this code - even here where
> the code is _single-threaded_; that's absolutely mad and just
> another sign of your disorder.
Ho ho. I dunno about this fixed action pattern mularkey, but where I come
from the application of past experience to new-but-similar situations is
usually referred to as "learning from your mistakes" or "common sense".
Disordered and proud of it, me. :)
Eric P.
Maybe he just needs a hug.
Eric
robert...@yahoo.com
wrote:
> > ... but is marked by the OS as zeroed.
>
> Whatcelse could zeroed mean other than being assigned to a page
> with zeroes (unique or not)? And a not assigned page to be trapped
> by a page-fault on every acess isn't what I would call zeroed.
No, the OS remembers in some fashion that the still unassigned page is
to be zeroed (it may just be the normal action in some systems). When
the page *is* accessed, a real page is acquired, mapped in and zeroed
(if necessary). After that, no more traps.
Jan Vorbrüggen
>>with zeroes (unique or not)? And a not assigned page to be trapped
>>by a page-fault on every acess isn't what I would call zeroed.
> No, the OS remembers in some fashion that the still unassigned page is
> to be zeroed (it may just be the normal action in some systems). When
> the page *is* accessed, a real page is acquired, mapped in and zeroed
> (if necessary). After that, no more traps.
Just so. And as WNT is derived (at least conceptually) from VMS, which had
DZERO - demand zero - pages since day one that operate just as described, I'll
bet it behaves just the same.
Jan