code_seg / alloc_text pragma - static library

[David F.]

Hi,

If I have a stoic library built that I want to link and say it's part of the
"PAGE" code segment how do I do that? The library could be used in either
case (non-paged or paged) when I know it's okay to use in PAGE segment I
want to tell the build to make it paged code to not allocate non-paged pool
space for the code.

Also how exactly does the code_seg with class information work? Can I
provide the segment and class name in the header file or in a .CPP module
and does that make that class always go to that segment even if that class
is compiled to its own static library too?

TIA!!

[Maxim S. Shatskih]

> If I have a stoic library

"stoic"? really? or maybe neo-Platonic? :-)

I expect you mean "static".

> "PAGE" code segment how do I do that?

Try playing with scripts and options to LINK.

The default way is to use pragmas at compile time, which says the compiler to put the code to PAGE section in the .obj file.

But, if the library is already compiled without the pragmas, then its code is in .text default sections, and you will need some options to LINK to treat the library's .text as PAGE.

I don't know whether this is possible.

> Also how exactly does the code_seg with class information work?

Usual functions obey the same rules as with C, but the auto-generated C++ thunks like vtable/vbase stuff, "scalar deleting destructor" etc _do not obey any such rules_ and are just plain undocumented in this aspect. Same is true on template and "inline" expansions.

This is one of the most major issues with C++ in Windows kernel.

--
Maxim S. Shatskih
Windows DDK MVP
ma...@storagecraft.com
http://www.storagecraft.com

[David F.]

I was thinking about the linker option ... I'll see if I can find anything?

"Maxim S. Shatskih" <ma...@storagecraft.com.no.spam> wrote in message
news:uXrrCU6g...@TK2MSFTNGP04.phx.gbl...

[Alexander Grigoriev]

Using paged code is not worth trouble. Unless the code is over 20MB. Save
yourself from grief and forget about using PAGE sections.

"David F." <df2...@community.nospam> wrote in message
news:059BB54E-A6FB-477C...@microsoft.com...

[Don Burn]

I have to disagree. The 20MB figure is big even for todays systems, and if
your driver can potentially be used in an embedded system or similar limited
then forgetting PAGE sections is a bad idea. I know of a major vendor who
lost a number of lucrative contracts thanks to this approach. In one case
a client of mine acquired the sources, and by simple work on using PAGE
sections produced an overall improvment in their embedded system by well
over 50%.

--
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

"Alexander Grigoriev" <al...@earthlink.net> wrote in message
news:%23A2k4NL...@TK2MSFTNGP04.phx.gbl...

> __________ Information from ESET NOD32 Antivirus, version of virus
> signature database 4715 (20091224) __________
>
> The message was checked by ESET NOD32 Antivirus.
>
> http://www.eset.com
>
>
>

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4715 (20091224) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com

[Alexander Grigoriev]

I've had my share of troubles because of paged code. The most recent was
paged code in win2008 acpi.sys which happened to be at DISPATCH_LEVEL, when
DriverVerifier marked all such pages non-present because a spinlock was
being acquired by another processor. Happened during DTM test for hot
processor plug simulation.

2GB of RAM costs 50$. 20MB costs $0.50 then. I'm not sure it's worth the
trouble.

"Don Burn" <bu...@stopspam.windrvr.com> wrote in message
news:eU1W2RLh...@TK2MSFTNGP06.phx.gbl...

[Kalle Olavi Niemitalo]

"David F." <df2...@community.nospam> writes:

> If I have a stoic library built that I want to link and say it's part
> of the "PAGE" code segment how do I do that? The library could be
> used in either case (non-paged or paged) when I know it's okay to use
> in PAGE segment I want to tell the build to make it paged code to not
> allocate non-paged pool space for the code.

I suppose you could specify a custom segment name
(e.g. yourlib.text) when building the static library, then use
LINK /MERGE to rename it to PAGETEXT or .text in each driver that
uses the static library. Dunno how to best do that in BUILD.

[Doron Holan [MSFT]]

it does not quite work that way. by consuming NP pool you take away the
ability to use that memory for anything else. the friendlier you are with
NP pool (e.g. the less you consume), the better the system can respond to
memory pressure. a few megabytes make a huge difference when a machine is
running a lot of apps or is under load

d

--

This posting is provided "AS IS" with no warranties, and confers no rights.

"Alexander Grigoriev" <al...@earthlink.net> wrote in message

news:u#seTjMhK...@TK2MSFTNGP02.phx.gbl...

[Alexander Grigoriev]

I don't buy that at all. You know that Vista/2008 doesn't have an explicit
limit on NP pool size anymore. In any case, if you hit NP pool limit, you've
perhaps already hit other bottlenecks, such as IO throughput, etc.

And it doesn't matter if you hit it 20MB earlier or later. If it happens,
your system was already fragile. I don't believe, also that amount of loaded
pageable KM code in a x86 system is any more than 10 MB. Not worth trouble.
These paged code/paged stack bugs happen to be quite obscure and hard to
diagnose.

"Doron Holan [MSFT]" <doron...@online.microsoft.com> wrote in message
news:OhwzGBXj...@TK2MSFTNGP02.phx.gbl...

[Maxim S. Shatskih]

> These paged code/paged stack bugs happen to be quite obscure and hard to
> diagnose.

PREfast and Verifier find most of them.

[Doron Holan [MSFT]]

well, come and argue with the windows perf team then ;). every page counts
and we can see measurable improvements as well NP pool usage. it is a
facetious argument to say that other limits in the system will be hit before
you reach NP pool limits, a piggy set of drivers can easily consume a ton of
NP pool while being idle. also, it is not about exhausting the limits of NP
pool, it is the effects that NP pool have on the rest of the memory manager
in terms of how much flexibility it has to manage paged pool. you are also
thinking of just the client. now take your thinking and apply it to N VMs
running on one machine, each overcommitted in terms of the memory assigned
to each.

d
--

This posting is provided "AS IS" with no warranties, and confers no rights.

"Alexander Grigoriev" <al...@earthlink.net> wrote in message

news:#HFeAGbj...@TK2MSFTNGP02.phx.gbl...

[alberto]

It depends on the application mix, no ? In my line of work, graphics
frames per second is the one overriding performance figure, and I'm
not shy of grabbing page frames if that helps me achieve my target
spinning rate. If the processor runs a tad slower, so be it! My chip
doesn't. :-)

I can afford that luxury because my environment is quite specialized
and my application suite can be expected to be by far the most
important thing - if not the only one - running in that machine; I
know that people won't be going to the web or doing email on the same
server. Still, different environments may have different needs, and
after all the NP pool is just a resource.

Alberto.

On Jan 5, 12:59 pm, "Doron Holan [MSFT]"

<doron.ho...@online.microsoft.com> wrote:
> well, come and argue with the windows perf team then ;).  every page counts
> and we can see measurable improvements as well NP pool usage. it is a
> facetious argument to say that other limits in the system will be hit before
> you reach NP pool limits, a piggy set of drivers can easily consume a ton of
> NP pool while being idle.  also, it is not about exhausting the limits of NP
> pool, it is the effects that NP pool have on the rest of the memory manager
> in terms of how much flexibility it has to manage paged pool.  you are also
> thinking of just the client. now take your thinking and apply it to N VMs
> running on one machine, each overcommitted in terms of the memory assigned
> to each.
>
> d
> --
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "Alexander Grigoriev" <al...@earthlink.net> wrote in message
>
> news:#HFeAGbj...@TK2MSFTNGP02.phx.gbl...
>
>
>
> > I don't buy that at all. You know that Vista/2008 doesn't have an explicit
> > limit on NP pool size anymore. In any case, if you hit NP pool limit,
> > you've perhaps already hit other bottlenecks, such as IO throughput, etc.
>
> > And it doesn't matter if you hit it 20MB earlier or later. If it happens,
> > your system was already fragile. I don't believe, also that amount of
> > loaded pageable KM code in a x86 system is any more than 10 MB. Not worth
> > trouble. These paged code/paged stack bugs happen to be quite obscure and
> > hard to diagnose.
>

> > "Doron Holan [MSFT]" <doron.ho...@online.microsoft.com> wrote in message

> >news:OhwzGBXj...@TK2MSFTNGP02.phx.gbl...
> >> it does not quite work that way.  by consuming NP pool you take away the
> >> ability to use that memory for anything else.  the friendlier you are
> >> with NP pool (e.g. the less you consume), the better the system can
> >> respond to memory pressure.  a few megabytes make a huge difference when
> >> a machine is running a lot of apps or is under load
>
> >> d
>
> >> --
>
> >> This posting is provided "AS IS" with no warranties, and confers no
> >> rights.
>
> >> "Alexander Grigoriev" <al...@earthlink.net> wrote in message
> >>news:u#seTjMhK...@TK2MSFTNGP02.phx.gbl...
> >>> I've had my share of troubles because of paged code. The most recent was
> >>> paged code in win2008 acpi.sys which happened to be at DISPATCH_LEVEL,
> >>> when DriverVerifier marked all such pages non-present because a spinlock
> >>> was being acquired by another processor. Happened during DTM test for
> >>> hot processor plug simulation.
>
> >>> 2GB of RAM costs 50$. 20MB costs $0.50 then. I'm not sure it's worth the
> >>> trouble.
>

> >>> "Don Burn" <b...@stopspam.windrvr.com> wrote in message

> >>>>http://www.eset.com- Hide quoted text -
>
> - Show quoted text -

[Alexander Grigoriev]

I would call this micro-optimization, rudiment from NT 3.1, which ran on 32
MB. I don't argue against paged pool, but paged kernel code should just go
away. And paged kernel stacks, too, though I didn't see swapped-out stacks
in W2008. Especially in kernel workitem threads. Yes, yes, kernel worker
threads that serve IoQueueWorkitem are waiting with UserMode and their
stacks could get swapped out, which I saw in Win2003. Too bad that our
driver stack that occasionally needed a kernel workitem to restore a target
connection, was in paging path. Bammm - deadlock.

By the way, HyperV doesn't use overcommit for virtual machines. And in any
case, 20 MB doesn't buy you anything. 512MB would, but only in low end
configs.

Better memory usage optimization would be to fix file caching. Which is
supposedly, at last, (though I have not run stress tests on that), done in
2008/Vista+. Before it was so awful, that a large file copy would just steal
all memory from under other processes and cause them to page-in like mad.
Looks like someone messed up MRU/LRU really bad.

"Doron Holan [MSFT]" <doron...@online.microsoft.com> wrote in message

news:OJIlmDjj...@TK2MSFTNGP06.phx.gbl...

[David F.]

"Kalle Olavi Niemitalo" <k...@iki.fi> wrote in message
news:87y6ks7...@Astalo.kon.iki.fi...

I'm delaying trying to figure this out - I could probably move 40 to 50K out
of NPP by doing it.

[Pavel Lebedinsky [MSFT]]

> I would call this micro-optimization, rudiment from NT 3.1, which ran
> on 32 MB. I don't argue against paged pool, but paged kernel code
> should just go away. And paged kernel stacks, too

If you ask the Windows perf team, they will probably say that
saving 10 MB of nonpaged memory is a major achievement, not a
micro-optimization.

During win7 the perf team filed hundreds of bugs against all kinds
of components across Windows (not just drivers), asking them to
reduce nonpaged footprint, copy-on-write pages in user space etc.
Sometimes the savings were just a few pages per component,
but together they are a big part of why win7 is the only version
of Windows (except maybe NT 3.5/3.51) to have a smaller
memory footprint than the previous one.

> Better memory usage optimization would be to fix file caching.

That's like saying that you shouldn't try to make a plane lighter
until you can figure out how to make the engines more powerful.
Why not try to do both at the same time?

--
Pavel Lebedinsky/Windows Fundamentals Test

[Alexander Grigoriev]

Considering that all new computers are x64 compatible, with memory >=2GB,
and most OEMs are shipping x64 flavor installed by default, this exercise
can be compared to collecting drops from liquor bottles for a morning
hangover fix... Not that I ever did that...

"Pavel Lebedinsky [MSFT]" <pa...@online.microsoft.com> wrote in message
news:OwmRwS2j...@TK2MSFTNGP05.phx.gbl...

[Don Burn]

No the exercise is called responsible driver writting. I have faced with a
bunch of drivers that were poorly written and take much more non-paged code
than they needed to. It is amazing how little work it took once I got the
source to reduce one drivers foot print by 50%. There will be 32-bit
systems for a long time, there will be a lot of 64-bit systems with low end
memory systems which limit memory to something small for a much longer time.

--
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

Remove StopSpam to reply

"Alexander Grigoriev" <al...@earthlink.net> wrote in message

news:%23C7XTiB...@TK2MSFTNGP06.phx.gbl...

> __________ Information from ESET NOD32 Antivirus, version of virus

> signature database 4753 (20100108) __________

>
> The message was checked by ESET NOD32 Antivirus.
>
> http://www.eset.com
>
>
>

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4753 (20100108) __________

[alberto]

I frankly don't see what's the point. If there's not enough space for
drivers to comfortably use the nonpaged pool for what they need -
without having developers to exercise what I'd call "dirty"
optimization and handicrafted embellishment - then the OS may need to
be revamped, or we may need a bigger machine, or both.

There's also the issue that no matter how we hack it, 32-bit systems
are limited in what they can handle, and, true enough, by the time the
NP pool runs out of space we may have hit a bottleneck way before.

Given that it is already complicated enough to write a driver, I
believe that bothering about substituting paged for nonpaged memory
only adds to the complexity - and consequently to the potential
instability - of the code. There are more important things that
require our developer's creative juices. I agree with Alexander that
drivers should not use paged storage, period. If the OS cannot handle
that, let's improve the OS.

One other point that may not be relevant to many of you but that is
quite relevant to me is the compatibility issue. I just cannot afford
to let nonportable mechanisms to infiltrate my driver, or I'm going to
have a hard time running it on a non-Windows OS. I need a memory
utilization model that is eminently portable between Windows and the
Unixen, else I'm not going to use it!

Last but not least, "responsible" driver writing is at the eyes of the
beholder. In my line of work, for example, not making strong use of
the system's capabilities to optimize its main mission is not what I
would call "responsible". In fact, to avoid this kind of problem, I
allocate a pretty large chunk of physical memory at init time -
hundreds of megabytes if I can - and I suballocate my memory from
there, and chuck the NP pool. When push comes to shove and what I need
is to make sure that the main mission of the system gets accomplished,
I'm not at all shy of taking control of things in my own hands, and I
call that as "responsible" as it can be - my users come first, the OS
second.

But hey, I may be wrong.

Alberto.

[Don Burn]

Gee, it is SO HARD to take a look at call and see if it is called only at
PASSIVE_LEVEL and does not use something like spin locks that raise IRQL.
And then of course once you add the #pragma page and a PAGED_CODE macro into
the driver, it is INCREDIBLY HARD to run PreFast to find out if there is
either a likelyhood the call is not a PASSIVE or that you are using an IRQL
raising function. And of course it is CLOSE TO IMPOSSIBLE to run the
checked version of the driver, and if by chance the analysis was wrong catch
things when the PAGED_CODE macro asserts.

Personally, I find it IMPOSSIBLE TO UNDERSTAND how people can be so lazy to
not develop the code correctly.

--
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply

"alberto" <amor...@ieee.org> wrote in message
news:42d9806e-1f8e-4d1d...@e20g2000vbb.googlegroups.com...

Alberto.

database 4754 (20100108) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4754 (20100108) __________

[Pavel A.]

This is optimization. Every developer can consider for his specific project
when to do it, or maybe not do it at all.

It seems that MS was motivated enough to optimize win7, and they did it
well.
Now we have more space and can afford being sloppy with our stuff ;)

Regards,
--pa

[Don Burn]

No this is good programming practice. I guess you are confusing SHIT with
SLOPPY. There is never enough resources in kernel programming and any
responsible developer is careful with the resources they use.

--
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply

"Pavel A." <pav...@12fastmail34.fm> wrote in message
news:OomheGJk...@TK2MSFTNGP02.phx.gbl...

> __________ Information from ESET NOD32 Antivirus, version of virus

> signature database 4755 (20100108) __________

>
> The message was checked by ESET NOD32 Antivirus.
>
> http://www.eset.com
>
>
>

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4755 (20100108) __________

[Alexander Grigoriev]

Yes. Unless your driver is in the paging path.

"Don Burn" <bu...@stopspam.windrvr.com> wrote in message

news:eeSHb7Ik...@TK2MSFTNGP06.phx.gbl...

[alberto]

As I said before, compatibility issues alone prevent me from letting
things such as IRQLs and PASSIVE_LEVEL to float out of the
subarchitectural level. In the 5 to 10% of the driver that's Windows
dependent I can splurge on that kind of stuff, but the rest of the
driver must not know what things like PASSIVE_LEVEL are all about.
That 90% + of the driver must be buildable by GCC as well, and
#pragmas that bring in nonportable things are, well, unacceptable.

And consider this. I can write a spinlock loop in plain vanilla
machine code that is totally portable between Windows, Linux, Solaris
and MacOS. Why should I bother using anything else ? That's why I
recommended in another post to mask away all his synchronization
constructs with macros, so that all that OS-dependent stuff is kept
away from the driver's mainstream. Helps debugging, helps
compatibility, saves coding.

And believe me, I just do not use paged code anywhere in my driver.
Does that make it unusable, or "incorrect" ? I don't think so. Right
now we're running it in stress mode, uninterrupted, for days, 24/7,
calm sea and prosperous voyage. The driver for the current board's
predecessor - I didn't write it - doesn't use paged code either, and
it has been in production for close to 10 years in a very demanding
medical environment.

So, hey, it's never as clear cut as it looks!

Alberto.

[Alexander Grigoriev]

A spinlock without raising DPC (which can't be done in plain vanilla machine
code) won't work. It might cause deadlocks.

"alberto" <amor...@ieee.org> wrote in message

news:6cc48be6-054a-49b8...@d30g2000vbl.googlegroups.com...

[Scott Noone]

>A spinlock without raising DPC (which can't be done in plain vanilla
>machine code) won't work. It might cause deadlocks.

You can write the spinlock and disable all interrupts with cli instead of
raising to DISPATCH_LEVEL (not that I recommend this as it unnecessarily
negatively impacts the system).

-scott

--
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

"Alexander Grigoriev" <al...@earthlink.net> wrote in message

news:efB7nMAl...@TK2MSFTNGP04.phx.gbl...

[alberto]

If a piece of memory is important enough that it has to be
synchronized at interrupt time, it may be important enough that the
overhead of locking all processors isn't that big of a deal. In fact,
I haven't seen that many situations where optimizing processor
performance makes that much difference, unless we're talking about big
time scientific or engineering computation.

Alberto.

On Jan 12, 10:47 pm, "Alexander Grigoriev" <al...@earthlink.net>
wrote:

> A spinlock without raising DPC (which can't be done in plain vanilla machine
> code) won't work. It might cause deadlocks.
>

> "alberto" <amore...@ieee.org> wrote in message

[Maxim S. Shatskih]

>I haven't seen that many situations where optimizing processor
>performance makes that much difference, unless we're talking about big
>time scientific or engineering computation.

Imagine 20 VMs with your code running on Hyper-V or VMWare.