cost of crossing kernel/user boundary
Oliver Dain
protection boundary is very expensive. I don't see why this has to be the
case. Many (most?) OSs don't switch of the MMU on a switch so the only
thing I can think of that changes is the protection level. This allows the
OS to execute instructions that are forbidden in user-mode code, but why
would that slow things down so much? Doesn't seem to me like it should be
any more expensive then a regular function call. Could somebody explain to
me where the cost comes from.
Thanks!
Nick Maclaren
Well, I get flamed every time that I say this, but a lot of the cause
is the now-standard mistake of forcing all such changes to be handled
by a single interrupt mechanism. This does slow down the hardware,
but it REALLY slows down the software, especially on large SMP systems.
The point is that an interrupt has to lock at least the CPU and its
associated resources, and often take out a global lock, until it has
classified the reason. Quite unnecessary.
Older machines (and I am talking 1950s and 1960s here, though some
used the techniques later) had more than one mechanism, which can
actually simplify both the hardware and software.
For example, there is absolutely no reason for program exceptions
(e.g. overflow, or even TLB misses) or system calls to do ANYTHING
except change mode until they have decided what needs to be done.
For example, why shouldn't a process be swapped out while it is in
the middle of fixing up a denormalised number exception? In fact, in
that case, there is no reason for it to enter kernel mode in the
first place!
With a suitable design, such operations should not be significantly
more expensive that the cost of an unpredicted branch plus that of
a routine call. As you say, that is effectively all they have to do.
Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email: nm...@cam.ac.uk
Tel.: +44 1223 334761 Fax: +44 1223 334679
Oliver Dain
> Well, I get flamed every time that I say this, but a lot of the cause
> is the now-standard mistake of forcing all such changes to be handled
> by a single interrupt mechanism. This does slow down the hardware,
> but it REALLY slows down the software, especially on large SMP systems.
> The point is that an interrupt has to lock at least the CPU and its
> associated resources, and often take out a global lock, until it has
> classified the reason. Quite unnecessary.
Thanks for the response. This still doesn't seem like it should take too
much time. "locking the CPU" involves disabling interrupts. Is there
anything else involved there. If not, disabling interrupts generally
involves setting some bits in a register which should be cheap. (If we've
got multiple CPUs some other work needs to be done, but lets ignore that
for now)
Now since we've got 1 routine handling everything we've got to look up the
routine we really want from a vector or something and then call it, but
this still seems pretty cheap. A total cost (including disabling
interrupts) of less than 20 instructions I would think. This is much less
than the cost typically attributed to a user/kernal mode switch (several
thousand clock cycles or more is the type of number I've heard).
What am I missing? Do some of these instructions (e.g. changing the
protection level) take a very long time to execute?
Thanks.
Anne & Lynn Wheeler
> Thanks for the response. This still doesn't seem like it should take too
> much time. "locking the CPU" involves disabling interrupts. Is there
> anything else involved there. If not, disabling interrupts generally
> involves setting some bits in a register which should be cheap. (If we've
> got multiple CPUs some other work needs to be done, but lets ignore that
> for now)
>
> Now since we've got 1 routine handling everything we've got to look up the
> routine we really want from a vector or something and then call it, but
> this still seems pretty cheap. A total cost (including disabling
> interrupts) of less than 20 instructions I would think. This is much less
> than the cost typically attributed to a user/kernal mode switch (several
> thousand clock cycles or more is the type of number I've heard).
>
> What am I missing? Do some of these instructions (e.g. changing the
> protection level) take a very long time to execute?
>
> Thanks.
because of various machine issues ... frequently disabling for
interrupts and changing machine state will draing/serialize the
processor. the because it is a generalized interrupt routine ... there
is a lot of saving state of the application program followed by
loading kernel state. then ere is some amount of generalized decode of
what was the application program status at the time of the kernel call
... then generalized decode of the kernel call parameters ... then
generalized decode of deciding if the application is allowed to
request the desired activity.
attempting to address this stuff in hardware try the whole access
register stuff. it somewhat started out as
1) some of the 3033 cross-memory stuff ... which was a solution to
running out of addressability with both kernel & application resident
in the same 16mbyte address space (originally data)
2) moving system services library code resident in application space
to different address space and some limited changes in privileges (w/o
having to go all the way to kernel mode) ... effectively trying to
have some of the efficiencies of subroutine library call with some
of the things that happen for change of privileges that come with
a kernel call
3) high cost of forcing things thru single/common kernel interrupt
structure
misc past cross-memory &/or access register postings (in the foloowing
there are some URL pointers to access register detailed implementation
description in POP):
http://www.garlic.com/~lynn/98.html#11 S/360 operating systems geneaology
http://www.garlic.com/~lynn/98.html#36 What is MVS/ESA?
http://www.garlic.com/~lynn/2000c.html#35 What level of computer is needed for a computer to Love?
http://www.garlic.com/~lynn/2000c.html#83 Is a VAX a mainframe?
http://www.garlic.com/~lynn/2000c.html#84 Is a VAX a mainframe?
http://www.garlic.com/~lynn/2000d.html#28 RS/6000 vs. System/390 architecture?
http://www.garlic.com/~lynn/2000e.html#57 Why not an IBM zSeries workstation?
http://www.garlic.com/~lynn/2000g.html#28 Could CDR-coding be on the way back?
http://www.garlic.com/~lynn/2001d.html#28 Very CISC Instuctions (Was: why the machine word size ...)
http://www.garlic.com/~lynn/2001d.html#30 Very CISC Instuctions (Was: why the machine word size ...)
http://www.garlic.com/~lynn/2001h.html#73 Most complex instructions
http://www.garlic.com/~lynn/2001k.html#16 Minimalist design (was Re: Parity - why even or odd)
http://www.garlic.com/~lynn/2002d.html#51 Hardest Mistake in Comp Arch to Fix
http://www.garlic.com/~lynn/2002g.html#5 Black magic in POWER5
http://www.garlic.com/~lynn/2002g.html#17 Black magic in POWER5
http://www.garlic.com/~lynn/2002h.html#21 PowerPC Mainframe
http://www.garlic.com/~lynn/2002l.html#57 Handling variable page sizes?
includes url pointer to pop ... plus table of contents of some of the factilities
http://www.garlic.com/~lynn/2002n.html#74 Everything you wanted to know about z900 from IBM
--
Anne & Lynn Wheeler | ly...@garlic.com - http://www.garlic.com/~lynn/
Chris Torek
>Oliver Dain <oda...@nospam.mindspring.com> wrote:
>>I have read in many places that the cost of crossing the user/kernel
>>protection boundary is very expensive. I don't see why this has to be the
In article <atij0f$p7e$1...@pegasus.csx.cam.ac.uk>
Nick Maclaren <nm...@cus.cam.ac.uk> writes:
>Well, I get flamed every time that I say this, but a lot of the cause
>is the now-standard mistake of forcing all such changes to be handled
>by a single interrupt mechanism.
What "single interrupt mechanism" is that?
>This does slow down the hardware, but it REALLY slows down the
>software, especially on large SMP systems. The point is that an
>interrupt has to lock at least the CPU and its associated resources,
>and often take out a global lock, until it has classified the reason.
>Quite unnecessary.
I will note that BSD/OS uses whatever mechanism is "obvious" or
"mandated" by the hardware or ABI for whichever hardware is in use.
On the IA32, this means that user code requests to the kernel use
the "lcall" instruction (except for the Linux compatibility hack,
which bounces the Linux-specific "int" [interrupt] instruction
directly out to user mode to be handled in the LAP library). There
are no locks involved in this process. (Of course, any individual
system call may need to get its own locks later.)
Interrupts, on the other hand, do have to identify the target of
the interrupt. If there is some mechanism to do a one-to-one map
from "interrupt source" to "target kernel thread", we use that
(without locks); otherwise we use whatever we must (with any locks
required) to locate the target thread. Having found the thread to
run, we do have to touch the (currently-global) scheduling lock in
order to make it runnable. The current IA32 version of BSD/OS has
some special-purpose hacks that avoid some of this in many cases,
but the plan is to go to atomic queues or similar where possible.
--
In-Real-Life: Chris Torek, Wind River Systems (BSD engineering)
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)
Domain: to...@bsdi.com http://67.40.109.61/torek/ (for the moment)
(you probably cannot email me -- spam has effectively killed email)
Greg Pfister
Others will undoubtedly talk about changing address spaces, TLB
misses, and many other issues. Here's one not on that level of radar
screen:
Going into the kernel, on most operating systems, always means that on
the way back out you get run through the scheduler. This may decide
that somebody else should be using the CPU, and whip you off into the
never-never land of "don't dispatch this guy for a while," so you
don't execute again for quite some time.
On the other hand, it might not. From the point of view of a program
being executed, which happens is a roll of the dice.
This is the single biggest thing that makes developers of databases
bend over backwards in their code to avoid entering the OS. It has
nothing to do with hardware issues, of course, and could be changed,
but in fact hasn't changed for a good two decades.
--
Greg Pfister http://pfister.userv.ibm.com/
Dale Pontius
understand WHY the original PC BIOS interrupts overlayed the reserved
area on the 8088 interrupt table, how they worked around it, and the
same set of things on choosing edge vs level interrupts.
Carl Sagan wrote a piece called "The Dragons of Eden," about how
evolution doesn't throw things away, new layers are glommed on. Deep
down we still have 'lizard brains' in our heads, despite the more
advanced layers over top.
Apparently computers are the same way. Don't forget that much of the
cruft in the original PC-DOS was an attempt to allow easy migration
from the even earlier CP/M. Residue of a single 8259 chip will still
be in the first Hammer machines. I wonder if some of that junk is in
IA-64 machines.
I'll presume you asked the question thinking about X86. It would be
fun to look at syscall and interrupt latencies on other architectures
and/or OS's which share no heritage with PC-DOS, then compare.
Dale Pontius
Jan C. Vorbrüggen
verify the integrity of the arguments being passed to it by the user code.
This can be a complicated and cycle-hungry issue. VAX has instructions to
do this, and Alpha emulates these in PAL code - but you end up looking at
the TLB and/or the associated PTEs in any case.
Of course, Nick is right that some OS calls just shouldn't cross the
protection
boundary. Surprise, some OSs (e.g., VMS) allow this to happen - that is, some
"syscalls" really _are_ just normal function calls.
There is also, of course, often a level of indirection similar to the use of
shareable libs/DLLs involved - after all, a syscall is conceptually part of a
(special) shareable lib.
Jan
Nick Maclaren
In article <atj2qg$dh6$1...@elf.eng.bsdi.com>,
Chris Torek <nos...@elf.eng.bsdi.com> writes:
|> >In article <atii5d$3qr$1...@slb3.atl.mindspring.net>,
|> >Oliver Dain <oda...@nospam.mindspring.com> wrote:
|> >>I have read in many places that the cost of crossing the user/kernel
|> >>protection boundary is very expensive. I don't see why this has to be the
|> >>case. ...
|>
|> In article <atij0f$p7e$1...@pegasus.csx.cam.ac.uk>
|> Nick Maclaren <nm...@cus.cam.ac.uk> writes:
|> >Well, I get flamed every time that I say this, but a lot of the cause
|> >is the now-standard mistake of forcing all such changes to be handled
|> >by a single interrupt mechanism.
|>
|> What "single interrupt mechanism" is that?
I am sure that you are asking that one rhetorically!
|> >This does slow down the hardware, but it REALLY slows down the
|> >software, especially on large SMP systems. The point is that an
|> >interrupt has to lock at least the CPU and its associated resources,
|> >and often take out a global lock, until it has classified the reason.
|> >Quite unnecessary.
|>
|> I will note that BSD/OS uses whatever mechanism is "obvious" or
|> "mandated" by the hardware or ABI for whichever hardware is in use.
Obviously the right approach. As you know, an awful lot of systems
(including many or most RISC ones) provide only a single mechanism.
The operating system can use only the mechanisms provided by the
hardware, and the run-time system only those provided by the
operating system.
Your explanation is precisely what I was referring to - if the
hardware provided the right primitive, you wouldn't take out a
global or even CPU lock for thread-specific interrupts in the first
place. But, as it is ....
Dale Morris
"Oliver Dain" <oda...@nospam.mindspring.com> wrote in message
news:atii5d$3qr$1...@slb3.atl.mindspring.net...
System calls can be done much more like a normal procedure call, if one has
a mechanism for promoting to the OS privilege without requiring an
interruption. As examples, Itanium architecture has the epc (enter
privileged code) instruction for this purpose, and PA-RISC has gate (gateway
branch). Obviously, there must be some way to prevent the user from abusing
this mechanism to gain excessive privilege for himself. For both Itanium
architecture and PA-RISC, the TLB is used to control whether a given code
page can allow privilege promotion.
However, there are still some things that must be done differently than a
normal procedure call. An example is that the kernel cannot run on the
user's stack, since it cannot trust that there will be sufficient storage on
that stack (or even that the user is not trying to be malicious and has made
a sys call with a bogus value in the stack pointer). So, at a minimum, the
sys call entrypoint must change stacks. (For Itanium architecture, it must
also change backing stores.)
Secondly, the OS must typically make some kind of annotation that it is now
on the kernel stack and not on the user stack (in case an interrupt occurs
while we're in the sys call). This is generally fairly hard, structurally
for OS's, since most have assumed since the dawn of time that sys calls
would always be done through an interruption (like with a break
instruction), and therefore most assume that the current executing privilege
level is a reliable indicator of whether we're executing on the user or
kernel stack. One could still live with this assumption if one had a way to
atomically raise privilege and change to the kernel stack (without a device
interrupt happening in the middle, that is). I don't know of any
architectures, however, that allow for such a mechanism.
Lastly, note that if the system call in question can be executed without it
making any calls or pushing anything on the stack, then there's no need to
change stacks. This could be possible for very light-weight sys calls, but
in practice I think it's also hard to require of the OS design that such a
sys call not push on the stack.
--
Dale Morris
System Architecture
Hewlett-Packard Co.
Jan C. Vorbrüggen
> on the kernel stack and not on the user stack (in case an interrupt occurs
> while we're in the sys call). This is generally fairly hard, structurally
> for OS's, since most have assumed since the dawn of time that sys calls
> would always be done through an interruption (like with a break
> instruction), and therefore most assume that the current executing privilege
> level is a reliable indicator of whether we're executing on the user or
> kernel stack.
So far, so good.
> One could still live with this assumption if one had a way to
> atomically raise privilege and change to the kernel stack (without a device
> interrupt happening in the middle, that is). I don't know of any
> architectures, however, that allow for such a mechanism.
What does a device interrupt have to do with it?!
The one architecture I know to that level of intimacy, VAX, on a change mode
instruction - being typically CISC, it has that for all four privilege levels
- puts the new (always more privileged) mode into the processor status word
as the current level, switches the stack pointer to the target mode stack
pointer, and starts executing the code at a fixed (given by an exeception
vector) address. No device interrupt anywhere in sight.
Jan
Nick Maclaren
In article <atl0ji$a8j$1...@web1.cup.hp.com>,
"Dale Morris" <dale_...@hp.com> writes:
|>
|> System calls can be done much more like a normal procedure call, if one has
|> a mechanism for promoting to the OS privilege without requiring an
|> interruption. As examples, Itanium architecture has the epc (enter
|> privileged code) instruction for this purpose, and PA-RISC has gate (gateway
|> branch). Obviously, there must be some way to prevent the user from abusing
|> this mechanism to gain excessive privilege for himself. For both Itanium
|> architecture and PA-RISC, the TLB is used to control whether a given code
|> page can allow privilege promotion.
Which was known to be a ghastly way of doing it before 1970. While
it does provide some protection, it is very hard to get right. It
is much better to do the checking the other side of the boundary.
MVT APF used that method, and its security was ghastly - it just
about protected against accidents.
|> However, there are still some things that must be done differently than a
|> normal procedure call. An example is that the kernel cannot run on the
|> user's stack, since it cannot trust that there will be sufficient storage on
|> that stack (or even that the user is not trying to be malicious and has made
|> a sys call with a bogus value in the stack pointer). So, at a minimum, the
|> sys call entrypoint must change stacks. (For Itanium architecture, it must
|> also change backing stores.)
You need a separate stack even for reliable exception handling in
'user' code. For example, you can't handle stack exhaustion
exceptions without one, nor can you diagnose problems where the
user's program has corrupted the stack or stack pointer.
|> Secondly, the OS must typically make some kind of annotation that it is now
|> on the kernel stack and not on the user stack (in case an interrupt occurs
|> while we're in the sys call). This is generally fairly hard, structurally
|> for OS's, since most have assumed since the dawn of time that sys calls
|> would always be done through an interruption (like with a break
|> instruction), ...
Not so. MANY have assumed that since the dawn of time, and MOST
have assumed that recently. But it used not to be so nearly
universal.
|> Lastly, note that if the system call in question can be executed without it
|> making any calls or pushing anything on the stack, then there's no need to
|> change stacks. This could be possible for very light-weight sys calls, but
|> in practice I think it's also hard to require of the OS design that such a
|> sys call not push on the stack.
Well, MVT, MVS and CMS did and do it :-)
But I am not going to justify HOW they did it, which I felt and feel
was a ghastly approach. On modern systems, there is no difficulty
in allowing an application (say) 16 privilege levels, with a separate
stack base and code base (note: NOT necessarily a separate segment)
for each, and a controlled call upwards.
This would allow the writing of reliable debuggers, run-time systems
and all that as well as the lightweight kernel calls. Yes, I really
do mean such things that would NEVER core dump or give up completely
just because you have trashed their 'stub' code. All right, there
are other ways of doing this, but privilege levels give it for free.
Edward A. Feustel
The Prime Computer ca 1975-1995 provided a protected procedure call
instruction which
took care of going to the operating system. I wrote a paper on it for either
ACM OS group or the
Computer Architecture group. Similarly the Prime had an instruction for
doing process exchange.
Ed Feustel
Michael S
several parameters from the user stack to the kernel stack.
glen herrmannsfeldt
"Jan C. Vorbrüggen" <jvorbr...@mediasec.de> wrote in message
news:3DFDA125...@mediasec.de...
> Another aspect that hasn't been mentioned so far: the kernel code has to
> verify the integrity of the arguments being passed to it by the user code.
> This can be a complicated and cycle-hungry issue. VAX has instructions to
> do this, and Alpha emulates these in PAL code - but you end up looking at
> the TLB and/or the associated PTEs in any case.
>
> Of course, Nick is right that some OS calls just shouldn't cross the
> protection
> boundary. Surprise, some OSs (e.g., VMS) allow this to happen - that is,
some
> "syscalls" really _are_ just normal function calls.
Many of the OS/360 I/O macros, such as READ and WRITE, are calls to
user space (and user protection key) routines loaded dynamically by OPEN.
Only at nearly the lowest level is the SVC issued to enter supervisor state
and actually perform the operation. It could be this was done to reduce the
time spent in interrupt routines. It may also have been done to reduce the
size of the OS. As I understand it, the people writing different OS modules
were given budgets of system memory. One way to get around this is to put
as much as possible into user space.
While things have changed over the years, I believe much of OS/390 and z/OS
still does similar operations.
-- glen
glen herrmannsfeldt
"Nick Maclaren" <nm...@cus.cam.ac.uk> wrote in message
news:atl4jf$s4i$1...@pegasus.csx.cam.ac.uk...
>
(someone wrote)
> |> Lastly, note that if the system call in question can be executed
without it
> |> making any calls or pushing anything on the stack, then there's no need
to
> |> change stacks. This could be possible for very light-weight sys calls,
but
> |> in practice I think it's also hard to require of the OS design that
such a
> |> sys call not push on the stack.
>
> Well, MVT, MVS and CMS did and do it :-)
One of the advantages of not having a stack is not having to worry about
changing stacks. The S/360 PSW swap arranges saving the old status
information, and loading new status information, without a stack. The
OS must then have a place to store registers to complete the state change.
-- glen
Joseph Seigh
It's not really a question of stack linkage. IBM standard linkage could be considered
as a linked stack if you want to get technical. I even gimmicked it into a semi-contiguous
stack that was compatible with standard linkage. I used to own VM linkage at one point and
we came *this* close to having a VM linkage (dynamic call) work with standard linkage. But
that's another story.
As far as system calls go, the system call handler saves registers in processor local storage.
It then or at some later point sets up a alternate stack or whatever. How much context you
save and when all depends. Usually you set up a decision tree to optimally decide what
has to be done. This is performance sensitive code and you usually have performance groups
literally counting and timing the code. I used to own VM's program check, svc, and
machine check handlers at one point also. VM's progam call linkage used svc based but
but was changed to branch entry for performance reasons, though you could still do an
svc based program call if you wanted to.
But there's also obligatory stuff that you may have to check because of how you designed the
OS. Signals could be delivered by a processor to processor hardware signal but it may
be more efficient to deliver them via software by having the interrupt handlers check for
pending software interrupts.
You also stuff on IBM machines like cross address space calls which lets you do kernel like
extensions which will have memory protection though maybe not privaleged state. I haven't
used these though. Solaris door calls aren't really the same. They seem to be hyped up
unix domain sockets. I've used these but don't know too much on how they're implemented.
Joe Seigh
glen herrmannsfeldt
"Joseph Seigh" <jsei...@xemaps.com> wrote in message
news:3DFE885E...@xemaps.com...
>
> glen herrmannsfeldt wrote:
> >
(snip)
> > One of the advantages of not having a stack is not having to worry about
> > changing stacks. The S/360 PSW swap arranges saving the old status
> > information, and loading new status information, without a stack. The
> > OS must then have a place to store registers to complete the state
change.
> >
>
> It's not really a question of stack linkage. IBM standard linkage could
be considered
> as a linked stack if you want to get technical. I even gimmicked it into
a semi-contiguous
> stack that was compatible with standard linkage. I used to own VM linkage
at one point and
> we came *this* close to having a VM linkage (dynamic call) work with
standard linkage. But
> that's another story.
Well, in some sense a linked list is similar to a stack, but in the case of
a hardware
stack pointer you have to always be careful that it points to a real stack,
usable
in the current processor state. There is always one current stack, but with
linked
lists you can have as many as you want. Of course there is software
overhead
with linked lists.
-- glen
Nick Maclaren
glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>
>
>Many of the OS/360 I/O macros, such as READ and WRITE, are calls to
>user space (and user protection key) routines loaded dynamically by OPEN.
>Only at nearly the lowest level is the SVC issued to enter supervisor state
>and actually perform the operation. It could be this was done to reduce the
>time spent in interrupt routines. It may also have been done to reduce the
>size of the OS. As I understand it, the people writing different OS modules
>were given budgets of system memory. One way to get around this is to put
>as much as possible into user space.
>
>While things have changed over the years, I believe much of OS/390 and z/OS
>still does similar operations.
Well, so does Unix, actually! The majority of the 'system interfaces'
are done by some quite complex user space library functions. But that
isn't really what I or (I think) Jan Vorbr\374ggen were talking about.
You really don't want to link such code into programs, for many reasons,
but even running it in simple user mode makes it less reliable than a
system call. Nasty corruptions then cause apparently atomic operations
to misbehave. There is also the point that many such operations are
built-in on some systems and emulated on others, or are raised as
exceptions by the hardware.
In most such cases, what you really want is a direct call or interrupt
into the application support library. It is easy to do, has been done
very well in the past, and would fit VERY well with the intentions of
the RISC movement. But it is rarely provided, at least with a decent
specification. And, as has been pointed out, it is NOT true that the
current systems couldn't or even wouldn't use it - at least some would,
and demonstrable successes might cause it to spread.
Nick Maclaren
Joseph Seigh <jsei...@xemaps.com> wrote:
>
>
>
>It's not really a question of stack linkage. IBM standard linkage could be considered
>as a linked stack if you want to get technical. I even gimmicked it into a semi-contiguous
>stack that was compatible with standard linkage. I used to own VM linkage at one point and
>we came *this* close to having a VM linkage (dynamic call) work with standard linkage. But
>that's another story.
Yes, it can be used as a stack, but the fact remains that neither the
hardware nor the operating system provide any kind of a stack for
application use. Those save areas can wander around data segments,
into code segments and out again, and cross task boundaries. They
can be broken, fork, be relinked dynamically and so on. Not very
much like a normal stack :-)
Joseph Seigh
In any operating system you have to maintain the correct register conventions
to make program calls. They may be different but they still have to be adhered to.
Making a IBM linkage call with an invalid register 13 is no less disasterous than
making a stack linkage call with an invalid stack pointer.
Now there may be advantages or disadvantages with one scheme vs. another, but that
has nothing to do with the importance of adhering to linkage conventions.
If someone is trying to make a point unique to stack linkage and interrupt/syscall
handlers, as someone who worked with real ones I'm not clear as what exactly that
point is.
Joe Seigh
Anne & Lynn Wheeler
> As far as system calls go, the system call handler saves registers in
> processor local storage. It then or at some later point sets up a
> alternate stack or whatever. How much context you save and when all
> depends. Usually you set up a decision tree to optimally decide what
> has to be done. This is performance sensitive code and you usually
> have performance groups literally counting and timing the code. I
> used to own VM's program check, svc, and machine check handlers at one
> point also. VM's progam call linkage used svc based but but was
> changed to branch entry for performance reasons, though you could
> still do an svc based program call if you wanted to.
The original cp/67 cp kernel had all internal calls/returns thru SVC
interrupt (code 8 for call, code 12 for return ... made it easy to
branch table). Call would dynamically allocate a new savearea
... thread from the existing savearea ... store various debuging
information, etc. Originally this was fixed pool of 100 saveareas
available for allocation. If you happened to exceed requirement for
100 allocated saveareas ... the system failed (loops, long threads, or
lot of concurrent activity).
I did a number of things as an undergraudate. One was ... when the
available saveareas were exhausted ... call "extend" for another 4k
storage block to make available for additional saveareas. This had
slight downside on debugging. In the original implementation ... it
was possible to easily pick out of contiguous area of storage ... all
currently in use ... and previously used, but available saveareas
(which could provide additional debugging information). Dynamically
extended saveareas were a little bit harder to explain.
The other thing i did as undergraduate was recognize that high
percentage of total time was spent in the SVC call/return processing
... especially the allocation and deallocation of saveareas. Part of
the analysis was determining that a large number of kernel calls were
to "closed" subroutines ... they would perform some function and
always immediately return (w/o making any call). I created two fixed
saveareas in page zero (balrsave & freesave) for use by closed
subroutines and change the call/return sequence for those routines to
do a direct branch & link (BALR) to the routine instead of svc linkage
... and to use the fixed save areas. While the SVC interrupt was more
expensive than BALR instruction ... the majority of the time spent in
the SVC interrupt handler wasn't the instruction itself but the
savearea management. The biggest performance boost came from
eliminating the savearea management.
One more thing that I did (during the summer job I had at BCS) was
introduce the pageable kernel routines. Both CP/67 and VM/370 kernels
ran in "real" addressing mode (but PSW changes to/from virtual
machines would switch in & out of virtual address mode). The issue was
how to allow (low useage) parts of the kernel to page. The process I
creates was to very carefully identity routines that were 4k or less
and make sure that when kernel was initially built, make sure that
they didn't cross 4k page boundary. Then all such routines were placed
by the kernel build routine after a known, fixed address ("CPEND" or
later "DMKCPE"). I then built a dummy address space table for the
kernel.
If the svc call handler was processing a "to" address that was larger
than CPEND ... it would first make a call to page fetch on the
routine's entry address (using the dummy address table) with option to
increment the page lock count, It would then do the standard savearea
stuff and then branch to the translated address (not the original
address). On svc return, the handler would check to see if the svc
interrupt address (where the return was coming from) was larger than
CPEND ... and if so, do a page unlock (decrement page lock count) on
the svc interrupt address (entry and exit had to be in the same 4k
area keeping the page lock counts consistent).
some amount of the stuff I did while an undergraduate ... was
incorporated into the standard cp/67 source code distribution (some of
the fastpath stuff, tty/ascii terminal support, misc. other stuff.
Other of the stuff ... like balr linkages weren't incorporated into
cp/67 distribution until after i joined the science center. The
pageable kernel changes never made it out in the cp/67 distribution
but was part of the standard vm/370 distribution.
random refs:
http://www.garlic.com/~lynn/2001b.html#23 Linux IA-64 interrupts [was Re: Itanium benchmarks ...]
http://www.garlic.com/~lynn/2001l.html#32 mainframe question
http://www.garlic.com/~lynn/2002b.html#44 PDP-10 Archive migration plan
http://www.garlic.com/~lynn/2002n.html#71 bps loader, was PLX
some early performance numbers regarding early kernel changes:
http://www.garlic.com/~lynn/94.html#18 CP/67 & OS MFT14
http://www.garlic.com/~lynn/94.html#20 CP/67 & OS MFT14
Edward A. Feustel
parameters. If I recall, a different instruction that checks arguments must
be run on each argument. Prime checked arguments, changed stacks, etc.
and did the right thing on return.
Ed Feustel
"Michael S" <already...@yahoo.com> wrote in message
news:f881b862.02121...@posting.google.com...
Eric Sosman
>
> You really don't want to link such code into programs, for many reasons,
> but even running it in simple user mode makes it less reliable than a
> system call. Nasty corruptions then cause apparently atomic operations
Perhaps I'm missing the point, but: If a program has
already gone off the rails to such an extent that it's
scribbled all over its own libraries, who cares if it
subsequently "misbehaves?"
Nick Maclaren
In article <3DFF51C2...@sun.com>,
The following are a few reasons why you want to do better:
A) You can finish off writing buffers (especially stderr), so that
there is a sporting chance of locating the problem.
B) You can write more reliable diagnostic and dumping code in the
application, to do even better.
C) You can free up locks and other critical resources, so that
manual intervention is less often needed.
D) (For safety-critical codes.) You can more reliably detect
failure and invoke the fallback code to take over.
Wanting to do any of those is SO old-fashioned, though :-(
glen herrmannsfeldt
>
(snip)
>
> In any operating system you have to maintain the correct register
conventions
> to make program calls. They may be different but they still have to be
adhered to.
> Making a IBM linkage call with an invalid register 13 is no less
disasterous than
> making a stack linkage call with an invalid stack pointer.
>
> Now there may be advantages or disadvantages with one scheme vs. another,
but that
> has nothing to do with the importance of adhering to linkage conventions.
>
> If someone is trying to make a point unique to stack linkage and
interrupt/syscall
> handlers, as someone who worked with real ones I'm not clear as what
exactly that
> point is.
call linkage is one thing. I thought the discussion here was more like
SVC linkage. On some machines, the return address to a call
like SVC is saved on a stack. On S/3x0, it is saved in a specific
memory location and the interrupt handler is responsible for saving
it in the appropriate place. That place better not be in a save
area pointed to by R13.
-- glen
glen herrmannsfeldt
"Anne & Lynn Wheeler" <ly...@garlic.com> wrote in message
news:u65tsn...@earthlink.net...
(snip)
>
(snip)
>
> One more thing that I did (during the summer job I had at BCS) was
> introduce the pageable kernel routines. Both CP/67 and VM/370 kernels
> ran in "real" addressing mode (but PSW changes to/from virtual
> machines would switch in & out of virtual address mode). The issue was
> how to allow (low useage) parts of the kernel to page. The process I
> creates was to very carefully identity routines that were 4k or less
> and make sure that when kernel was initially built, make sure that
> they didn't cross 4k page boundary. Then all such routines were placed
> by the kernel build routine after a known, fixed address ("CPEND" or
> later "DMKCPE"). I then built a dummy address space table for the
> kernel.
>
> If the svc call handler was processing a "to" address that was larger
> than CPEND ... it would first make a call to page fetch on the
> routine's entry address (using the dummy address table) with option to
> increment the page lock count, It would then do the standard savearea
> stuff and then branch to the translated address (not the original
> address). On svc return, the handler would check to see if the svc
> interrupt address (where the return was coming from) was larger than
> CPEND ... and if so, do a page unlock (decrement page lock count) on
> the svc interrupt address (entry and exit had to be in the same 4k
> area keeping the page lock counts consistent).
This sounds similar to the transient SVC area of OS/360, though I
believe that it had more than one, and I think they were 2K.
I hope they don't still have transient SVC's in OS/390.
-- glen
Alex Colvin
>user space (and user protection key) routines loaded dynamically by OPEN.
>Only at nearly the lowest level is the SVC issued to enter supervisor state
>and actually perform the operation. It could be this was done to reduce the
>time spent in interrupt routines. It may also have been done to reduce the
>size of the OS. As I understand it, the people writing different OS modules
>were given budgets of system memory. One way to get around this is to put
>as much as possible into user space.
>While things have changed over the years, I believe much of OS/390 and z/OS
>still does similar operations.
and those OSs were notorious for security holes caused by storing system
data in user space.
--
mac the naïf
Sander Vesik
>
> I'll presume you asked the question thinking about X86. It would be
> fun to look at syscall and interrupt latencies on other architectures
> and/or OS's which share no heritage with PC-DOS, then compare.
Wouldn't the japanese (?) pc98 be one source of insight into this ?
>
> Dale Pontius
--
Sander
+++ Out of cheese error +++
Joseph Seigh
glen herrmannsfeldt wrote:
>
> "Joseph Seigh" <jsei...@xemaps.com> wrote in message
> news:3DFF17C3...@xemaps.com...
> >
...
> > If someone is trying to make a point unique to stack linkage and
> interrupt/syscall
> > handlers, as someone who worked with real ones I'm not clear as what
> exactly that
> > point is.
>
> call linkage is one thing. I thought the discussion here was more like
> SVC linkage. On some machines, the return address to a call
> like SVC is saved on a stack. On S/3x0, it is saved in a specific
> memory location and the interrupt handler is responsible for saving
> it in the appropriate place. That place better not be in a save
> area pointed to by R13.
>
It is about svc linkage. My point was that what the program call linkage is
should not affect how a svc handler operates. Svc linkage and program call
linkage are two different things.
Joe Seigh
Anne & Lynn Wheeler
> This sounds similar to the transient SVC area of OS/360, though I
> believe that it had more than one, and I think they were 2K.
>
> I hope they don't still have transient SVC's in OS/390.
yes ... except instead of having a specific location(s) for loading
transient SVCs ... it was done thru standard paging mechanism. i had
done a lot with transient SVCs also on MFT & MVT performance
enhancements ... you could have some preloaded at boot and fixed in
the kernel (if you had the real storage) and the rest was careful
building of sys1.svclib to optimize disk arm motion. all required good
trace & frequency count of transient SVCs being loaded.
random past references to optimized MFT (& MVT) system builds that
got me three times thruput improvement for specific test job mix.
note that the os/360 thruput optimization work that i did as an
undergraduate was pretty much independent of the cp/67 work (performance
& function).
http://www.garlic.com/~lynn/94.html#18 CP/67 & OS MFT14
http://www.garlic.com/~lynn/94.html#20 CP/67 & OS MFT14
http://www.garlic.com/~lynn/97.html#22 Pre S/360 IBM Operating Systems?
http://www.garlic.com/~lynn/97.html#28 IA64 Self Virtualizable?
http://www.garlic.com/~lynn/98.html#21 Reviving the OS/360 thread (Questions about OS/360)
http://www.garlic.com/~lynn/99.html#93 MVS vs HASP vs JES (was 2821)
http://www.garlic.com/~lynn/2000c.html#10 IBM 1460
http://www.garlic.com/~lynn/2000d.html#50 Navy orders supercomputer
http://www.garlic.com/~lynn/2001.html#26 Disk caching and file systems. Disk history...people forget
http://www.garlic.com/~lynn/2001b.html#23 Linux IA-64 interrupts [was Re: Itanium benchmarks ...]
http://www.garlic.com/~lynn/2001f.html#26 Price of core memory
http://www.garlic.com/~lynn/2001h.html#12 checking some myths.
http://www.garlic.com/~lynn/2001i.html#30 IBM OS Timeline?
http://www.garlic.com/~lynn/2001k.html#37 Is anybody out there still writting BAL 370.
http://www.garlic.com/~lynn/2002b.html#23 Infiniband's impact was Re: Intel's 64-bit strategy
http://www.garlic.com/~lynn/2002b.html#24 Infiniband's impact was Re: Intel's 64-bit strategy
http://www.garlic.com/~lynn/2002c.html#45 cp/67 addenda (cross-post warning)
http://www.garlic.com/~lynn/2002c.html#51 cp/67 addenda (cross-post warning)
http://www.garlic.com/~lynn/2002m.html#3 The problem with installable operating systems
http://www.garlic.com/~lynn/2002n.html#29 why does wait state exist?
http://www.garlic.com/~lynn/2002n.html#71 bps loader, was PLX
--
Anne & Lynn Wheeler | ly...@garlic.com - http://www.garlic.com/~lynn/
Nick Maclaren
Yes and no. That was true in the early days, but that practice had
ben largely eliminated by the time of MVT 20.6 (and perhaps earlier).
There certainly were still places where the supervisor was relying
on user-modifiable code, which were nominally removed by MVS. But,
from that point on, that PARTICULAR loophole was effectively closed.
Nick Maclaren
>should not affect how a svc handler operates. Svc linkage and program call
>linkage are two different things.
A lot of the point of this thread is the question "But why should
that be the case?" There are a lot of advantages in handling the
two very similarly, and very few disadvantages.
Note that you can do the register saving BEFORE you raise privilege
in the SVC case, which removes the need for a separate stack in the
single-threaded case. In the multi-threaded case, you have to deal
with argument handling safely, so even a separate stack doesn't
help all that much.
So WHY should they be entirely different?
glen herrmannsfeldt
>
> In article <3DFF51C2...@sun.com>,
> Eric Sosman <Eric....@sun.com> writes:
> |> Nick Maclaren wrote:
> |> >
> |> > You really don't want to link such code into programs, for many
reasons,
> |> > but even running it in simple user mode makes it less reliable than a
> |> > system call. Nasty corruptions then cause apparently atomic
operations
> |> > to misbehave. [...]
> |>
> |> Perhaps I'm missing the point, but: If a program has
> |> already gone off the rails to such an extent that it's
> |> scribbled all over its own libraries, who cares if it
> |> subsequently "misbehaves?"
>
> The following are a few reasons why you want to do better:
>
> A) You can finish off writing buffers (especially stderr), so that
> there is a sporting chance of locating the problem.
>
> B) You can write more reliable diagnostic and dumping code in the
> application, to do even better.
>
> C) You can free up locks and other critical resources, so that
> manual intervention is less often needed.
>
> D) (For safety-critical codes.) You can more reliably detect
> failure and invoke the fallback code to take over.
>
> Wanting to do any of those is SO old-fashioned, though :-(
All true. I thought, though, the question was whether there was
any C standard related reason. On a reasonable OS that frees
the storage given to a program, I don't see any reason.
I know MVS does free storage.
-- glen
Joseph Seigh
Nick Maclaren wrote:
>
> In article <3DFF784F...@xemaps.com>,
> Joseph Seigh <jsei...@xemaps.com> wrote:
> >
> >
> >
> >It is about svc linkage. My point was that what the program call linkage is
> >should not affect how a svc handler operates. Svc linkage and program call
> >linkage are two different things.
>
> A lot of the point of this thread is the question "But why should
> that be the case?" There are a lot of advantages in handling the
> two very similarly, and very few disadvantages.
>
> Note that you can do the register saving BEFORE you raise privilege
> in the SVC case, which removes the need for a separate stack in the
> single-threaded case. In the multi-threaded case, you have to deal
> with argument handling safely, so even a separate stack doesn't
> help all that much.
>
> So WHY should they be entirely different?
>
You have to switch stacks anyhow if the kernel needed to do program calls.
The kernel can't use the process stack since it is unprotected and a malicious
program could alter saved kernel register values.
I don't think having the svc handler save registers buys you anything. It would make
the svc instruction more complicated and you'd have access exceptions, an
interrupt throwing an interrupt.
If you are writing your own interrupt handler, you can specifiy whatever svc calling
convention you wanted. So you could say the svc doesn't save registers except for
the stack pointer. The calling program would have to save its own registers beforehand
and restore them afterward. The handler would have to set the registers to innocuous
values on return anyhow, or you could return via a protected stub to do the restore
outside of priveleged state.
Joe Seigh
calling program would have do it
Nick Maclaren
glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>
>All true. I thought, though, the question was whether there was
>any C standard related reason. On a reasonable OS that frees
>the storage given to a program, I don't see any reason.
>
>I know MVS does free storage.
Ah, I see. No, there isn't, nor Fortran. And the operating system
doesn't need to free storage, anyway, as far as the standard goes.
If anything goes badly wrong, the behaviour is undefined, and that
means anything can happen (Armageddon, demons flying out of your
nose, or Bill Gates donating his Microsoft shares to the FSF).
glen herrmannsfeldt
> In article <YrML9.509$qF3.278@sccrnsc04>,
> glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
> >
> >All true. I thought, though, the question was whether there was
> >any C standard related reason. On a reasonable OS that frees
> >the storage given to a program, I don't see any reason.
> >
> >I know MVS does free storage.
>
> Ah, I see. No, there isn't, nor Fortran. And the operating system
> doesn't need to free storage, anyway, as far as the standard goes.
> If anything goes badly wrong, the behaviour is undefined, and that
> means anything can happen (Armageddon, demons flying out of your
> nose, or Bill Gates donating his Microsoft shares to the FSF).
Well, to me the point of a protected mode OS is that it prevents
you from destroying the system, or other programs, either
accidentally or intentionally. You are still allowed to destroy your
own data, including data stored on disk.
-- glen
Chris Torek
>> What "single interrupt mechanism" is that?
In article <atkhgl$c8h$1...@pegasus.csx.cam.ac.uk>
Nick Maclaren <nm...@cus.cam.ac.uk> writes:
>I am sure that you are asking that one rhetorically!
Actually, I was wondering if you were referring to some specific
hardware's specific instruction(s), or what.
On the V9 SPARC, hardware interrupts are turned into these rather
interesting UPA packet thingies, carrying two 64-bit words of data.
System calls, on the other hand, use a software trap instruction.
Thus, syscalls are invariably different from actual hardware
interrupts (and the CPU hardware uses different "special" registers
for each). The BSD/OS V9-sparc port (currently in limbo) actually
made use of the TL=0 vs TL>0 "two separate trap tables, in sequence"
thing so that any user->kernel transition was known to *be* a
user->kernel transition, at that point.
Of course, the CPU hardware always dumps the pipe on all traps --
including software traps -- adding an extra 13 or so cycles to
begin with ... and then there was the CPU bug I discovered where
the result of an ALU calculation would change places with the value
read from %tpc or %tnpc (I forget which) if you did not carefully
store in an otherwise-unnecessary way on the kernel side...
--
In-Real-Life: Chris Torek, Wind River Systems (BSD engineering)
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)
Domain: to...@bsdi.com http://67.40.109.61/torek/ (for the moment)
(you probably cannot email me -- spam has effectively killed email)
Anne & Lynn Wheeler
> One more thing that I did (during the summer job I had at BCS) was
> introduce the pageable kernel routines. Both CP/67 and VM/370 kernels
> ran in "real" addressing mode (but PSW changes to/from virtual
> machines would switch in & out of virtual address mode). The issue was
... and for a little bit of thread drift ... later when I did the
vm370 resource manager ... somewhat related:
http://www.garlic.com/~lynn/2001e.html#45 VM/370 Resource Manager
http://www.garlic.com/~lynn/subtopic.html#fairshare
http://www.garlic.com/~lynn/subtopic.html#bench
http://www.garlic.com/~lynn/subtopic.html#dumprx
there were other bits and pieces besides paging & scheduling
algorithms ... there were bits and pieces of other stuff like i redid
the kernel serialization primitive that eliminated a lot of the
timing-dependent failures that had been occuring. I also created a
dummy virtual memory table for each logged on user (in addition to
their normal virtual address space tables). the kernel tended to have
a lot of control blocks for each process (including their virtual
memory tables themselves) .... for inactive processes i would copy a
bunch of these tables into the process's dummy address table and use
that to page the tables out to secondary storage (freeing up the fixed
storage). This was somewhat analogous to the system dummy page table
used to page kernel code.
Jeff Kenton
>
>
> Now since we've got 1 routine handling everything we've got to look up the
> routine we really want from a vector or something and then call it, but
> this still seems pretty cheap. A total cost (including disabling
> interrupts) of less than 20 instructions I would think. This is much less
> than the cost typically attributed to a user/kernal mode switch (several
> thousand clock cycles or more is the type of number I've heard).
>
> What am I missing? Do some of these instructions (e.g. changing the
> protection level) take a very long time to execute?
>
I'm coming to this discussion a little late, but here's another data point.
Motorola 88000 RISC. Hardware does very little, software had to do nearly
everything. I once calculated worst case exception handling, and figured it was
4000-5000 instructions. This included draining a 3 stage memory pipeline and
the floating point pipeline (at least 3 stages, I don't remember exactly). FP
could require handling denormalized numbers, which took a lot of work. You
never really saw this particular worst case combination, but you couldn't do
anything in 20 instructions. You couldn't even save the registers in 20
instructions; there were 32 of them.
Bottom line -- it depends on the processor. This was Nick Maclaren's point.
--
-------------------------------------------------------------------------
= Jeff Kenton Consulting and software development =
= http://home.attbi.com/~jeffkenton =
-------------------------------------------------------------------------
Zack Weinberg
>
>This is the point where most OSs are doing wrong. It isn't necessary to
>save all GP registers. If you see a system call just like a normal
>procedure call with a different protection domain then you have to save
>no more registers than the ones you save during a regular procedure
>call. The content of the temporary registers is saved by the caller and
>a change not relevant after the call. The content of registers which
>have to be preserved over a procedure call are saved by the called
>function if they are used in this function.
You can't allow information to leak out of the higher-privileged
domain, though, which means at the very least you have to clear
all of the temporary registers on kernel exit. If there aren't
many of them compared with the complete register file, I suppose it
might indeed be a win to do that. (especially if there's a load-zero
instruction that doesn't touch memory)
zw
Jan C. Vorbrüggen
> >save all GP registers. If you see a system call just like a normal
> >procedure call with a different protection domain then you have to save
> >no more registers than the ones you save during a regular procedure
> >call.
Yup. I'd even say you need an agreed-upom kernel ABI - this is not your
random piece of code, you can require a little discipline and consistency
from your syscall programmer 8-|...
> You can't allow information to leak out of the higher-privileged
> domain, though, which means at the very least you have to clear
> all of the temporary registers on kernel exit.
Those registers need to be restored anyway to the value they had before
the (sys)call, except for those architecturally defined to return results.
Not a problem in general, although one needs to keep such issues in mind.
Jan
Zack Weinberg
>On 19 Dec 2002 01:02:49 -0500, Zack Weinberg <za...@panix.com> wrote:
>
>> You can't allow information to leak out of the higher-privileged
>> domain, though, which means at the very least you have to clear
>> all of the temporary registers on kernel exit.
>
>relevant information otherwise this is just random information which is
>not of any use. On the other side clearing a few registers does cost
>less performance than saving and restoring them.
In general all information known to the kernel has to be assumed
security-relevant. (At least with your typical monolithic kernel
design.) You would have to prove, for each temporary register that
you wanted to leave uncleared, that that register had never been
set to any value that could possibly reveal a piece of information
that would otherwise be inaccessible to the lower-privilege caller.
And this is a moving target, because you want to write as much of
the kernel as possible in a high-level language, which means you
don't know where the compiler decided to put data.
What it boils down to is, not clearing all the temp registers is a
huge amount of work for very little benefit. Keep in mind that on
modern chips the *trap instruction alone* takes several hundred cycles.
zw
Jan C. Vorbrüggen
> > random piece of code, you can require a little discipline and consistency
> > from your syscall programmer 8-|...
>
> access to system calls is implemented and hidden by the shared libc and
> may change from system version to system version without binary
> compatibility problems.
I was talking about the ABI after the mode switch to kernel, i.e., the
interface of the syscall dispatcher with the actual syscall routines.
Pretending that you can hide the syscall ABI in a user-mode library is
nonsense and dangerous from a security point of view (cf. Windows NT kernel
interface).
> No, they don't have to be restored. They are defined to be destroyed by
> the called function so the application using the ABI cannot depend on
> that registers containing the same values after the call.
You're assuming caller-saves in your ABI. Last I heard, callee-saves has
complementary advantages and disadvatages, and at least some systems I know
well 8-) use callee-saves. For me, this has the advantage that the knowledge
of which registers need to be saved is where it is generated: in the
implementation of the system routine itself.
> They may be cleared if they may have contained a security relevant
> information. But this can be done at the places where critical information
> is handled
Of course.
Jan
Dennis M O'Connor
> Zack Weinberg <za...@panix.com> wrote:
> > You can't allow information to leak out of the higher-privileged
> > domain, though, which means at the very least you have to clear
> > all of the temporary registers on kernel exit.
>
> relevant information otherwise this is just random information which is
> not of any use.
It isn't random information, and even if it isn't the secrets
you are using, it may leak information that may allow
someone to get your secrets more easily. You'd be
surprised at how sophisticated cryptanalysis is.
--
Dennis M. O'Connor dm...@primenet.com
"We don't become a rabid dog to destroy a rabid dog,
but we do get a bit rude."
Sander Vesik
>> >This is the point where most OSs are doing wrong. It isn't necessary to
>> >save all GP registers. If you see a system call just like a normal
>> >procedure call with a different protection domain then you have to save
>> >no more registers than the ones you save during a regular procedure
>> >call.
>
> Yup. I'd even say you need an agreed-upom kernel ABI - this is not your
> random piece of code, you can require a little discipline and consistency
> from your syscall programmer 8-|...
I guess one problem here is that compilers don't support being told
'pleas use only these regsters' very well, so kernels for the most part
end up using the normal ABI internaly. The other issue is that unless you
applied this to only specific parts of the kernel it would probably
be a net loss instead of saving.
>
> Jan
Andy Isaacson
Emil Naepflein <Emil.Na...@philosys.de> wrote:
>On 20 Dec 2002 02:42:29 -0500, Zack Weinberg <za...@panix.com> wrote:
>> What it boils down to is, not clearing all the temp registers is a
>> huge amount of work for very little benefit.
>
>registers. So you
It looks like your post got truncated; sorry if my response is at odds
with what you would have said.
The point you seem to be missing is that the kernel *must* set *every*
register to a known value (zero, or the previous value) at the syscall-exit
point. If it doesn't do that, then the caller can access the value that
was not overwritten, and it is very likely that a security breach would
occur as a result.
Given that you need to set every register, it's easier to just *do* the
save/restore, rather than having caller-saves registers which the kernel
has to modify *anyways*. If you're careful (see the Linux people) you
ensure that your save-frame fits into a single (or small N) cacheline(s),
and you don't even pay any extra memory transactions to do the
save/restore.
There's an interesting thread on Linux-kernel going on right now about
syscall enter/exit; the subject is "Intel P6 vs P7 system call performance"
and the thread is several hundred entries long. Linus had a bunch of
interesting measurements to share towards the beginning (around December
16-18).
<URL:http://groups.google.com/groups?threadm=200212090830.gB98USW05593%40
flux.loup.net>
-andy
Andy Glew
Lynn: will you ever write a book? Please?!?
Michael S
> "Emil Naepflein" <Emil.Na...@philosys.de> wrote...
> > Zack Weinberg <za...@panix.com> wrote:
> > > You can't allow information to leak out of the higher-privileged
> > > domain, though, which means at the very least you have to clear
> > > all of the temporary registers on kernel exit.
> >
> > You only have to do that in cases where they may have contained security
> > relevant information otherwise this is just random information which is
> > not of any use.
>
> It isn't random information, and even if it isn't the secrets
> you are using, it may leak information that may allow
> someone to get your secrets more easily. You'd be
> surprised at how sophisticated cryptanalysis is.
In other words: only paranoids survive...
Nick Maclaren
In article <f881b862.02122...@posting.google.com>,
Well, yes. But it is critical not to fall into the common trap
of spending too much time bolstering one aspect of security while
ignoring more important ones or, worse, making the system so hard
to administer that practical security suffers.
The matter of not passing register contents down a security
level has VERY rarely been the cause of a security breach - I
have heard of only a couple of cases, and they were ages back.
There are FAR more important issues to deal with.
The reason that it is worth clobbering all registers is that it
is too expensive (in effort) to check whether they might contain
something sensitive. So it is easier just to assume the worst,
and does little harm.
If you really need to STOP information leaking through security
levels, you can't start with any modern CPU architecture or
operating system and hope to get far. For example, the sharing
of caches, TLBs etc. allows a malicious program to find out
really quite a lot about other programs running on the same CPU.
And heaven help you with HypeThreading!
Anne & Lynn Wheeler
that sounds too much like hard work.
at one time i had accumulated large number of draft research reports
.... working with professional tech writer and it was very painful
process. making it even more painful we would periodically submit them
to company approval process (approving that they could be subbmitted
for publication) and get turned down. finally after a number of years
the tech writer retired and sent me all of his files of stuff we had
been working on. there was some conjecture that i wasn't getting
approval for publication because of some impression that i had been
associated with something called tandem memos. random refs:
http://www.garlic.com/~lynn/2001g.html#5 New IBM history book out
http://www.garlic.com/~lynn/2001g.html#6 New IBM history book out
http://www.garlic.com/~lynn/2001g.html#7 New IBM history book out
http://www.garlic.com/~lynn/2001j.html#31 Title Inflation
http://www.garlic.com/~lynn/2002k.html#39 Vnet : Unbelievable
http://www.garlic.com/~lynn/2002o.html#73 They Got Mail: Not-So-Fond Farewells
some years later .... an executive told me that they could forgive me
for being wrong but they could never forgive me for being right.
and for something completely different (but similar):
http://www.amazon.com/exec/obidos/ASIN/0316881465/qid%3D1037135329/sr%3D11-1/ref%3Dsr_11_1/104-1355456-9727942
http://www.garlic.com/~lynn/subtopic.html#boyd
--
Anne & Lynn Wheeler | ly...@garlic.com - http://www.garlic.com/~lynn/
internet trivia 20th anv: http://www.garlic.com/~lynn/rfcietff.htm
Sander Vesik
>
> The reason that it is worth clobbering all registers is that it
> is too expensive (in effort) to check whether they might contain
> something sensitive. So it is easier just to assume the worst,
> and does little harm.
>
> If you really need to STOP information leaking through security
> levels, you can't start with any modern CPU architecture or
> operating system and hope to get far. For example, the sharing
> of caches, TLBs etc. allows a malicious program to find out
> really quite a lot about other programs running on the same CPU.
> And heaven help you with HypeThreading!
>
Well, you can just run the programs inside different instances of
Bochs and thus make most of the details opque. But really, if you
need reliable separation you have to use separate hardware (say
a blade server) and isolate compuntication (swithces with vnets
and ipsec).
>
> Regards,
> Nick Maclaren,
> University of Cambridge Computing Service,
> New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
> Email: nm...@cam.ac.uk
> Tel.: +44 1223 334761 Fax: +44 1223 334679
--
Douglas Siebert
>If you really need to STOP information leaking through security
>levels, you can't start with any modern CPU architecture or
>operating system and hope to get far. For example, the sharing
>of caches, TLBs etc. allows a malicious program to find out
>really quite a lot about other programs running on the same CPU.
>And heaven help you with HypeThreading!
What's wrong with flushing caches and TLBs on context switches? If you
want that level of security you are paying plenty of performance penalties,
this one is lost in the noise. I see no reason not to use an off the
shelf CPU, even with all the restrictions that you'd have to impose it'd
perform better than a custom design with miniscule volumes that'd be at
least several years behind in both design and process. After all, it
isn't as though you'd be allowing execution of arbitrary code, so the
issues that make x86 not fully virtualizable don't matter, etc.
--
Douglas Siebert dsie...@excisethis.khamsin.net
"Suppose you were an idiot. And suppose you were a member of Congress.
But I repeat myself." -- Mark Twain
Terje Mathisen
> some years later .... an executive told me that they could forgive me
> for being wrong but they could never forgive me for being right.
LOL!
I had to read this one out loud for my wife. :-)
It reminds me of a Heinlein book (Lazarus Long?):
"Suffer not a true prophet to live" (or words to that meaning).
Terje
PS. Here in Norway Christmas Eve is the big day, not Christmas Day, so
for all those of you celebrating at this time of the year:
Merry Christmas!
--
- <Terje.M...@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching"
Nick Maclaren
Douglas Siebert <dsie...@excisethis.khamsin.net> wrote:
>nm...@cus.cam.ac.uk (Nick Maclaren) writes:
>
>>If you really need to STOP information leaking through security
>>levels, you can't start with any modern CPU architecture or
>>operating system and hope to get far. For example, the sharing
>>of caches, TLBs etc. allows a malicious program to find out
>>really quite a lot about other programs running on the same CPU.
>>And heaven help you with HypeThreading!
>
>What's wrong with flushing caches and TLBs on context switches? If you
>want that level of security you are paying plenty of performance penalties,
>this one is lost in the noise. I see no reason not to use an off the
>shelf CPU, even with all the restrictions that you'd have to impose it'd
>perform better than a custom design with miniscule volumes that'd be at
>least several years behind in both design and process. After all, it
>isn't as though you'd be allowing execution of arbitrary code, so the
>issues that make x86 not fully virtualizable don't matter, etc.
Hmm. Flushing the level 3 cache every time you flip context briefly
to service an event is a bit painful - you are better off with separate
machines.
Christian Bau
nm...@cus.cam.ac.uk (Nick Maclaren) wrote:
> In article <au8oig$ptu$1...@sword.avalon.net>,
> Douglas Siebert <dsie...@excisethis.khamsin.net> wrote:
> >nm...@cus.cam.ac.uk (Nick Maclaren) writes:
> >
> >>If you really need to STOP information leaking through security
> >>levels, you can't start with any modern CPU architecture or
> >>operating system and hope to get far. For example, the sharing
> >>of caches, TLBs etc. allows a malicious program to find out
> >>really quite a lot about other programs running on the same CPU.
> >>And heaven help you with HypeThreading!
> >
> >What's wrong with flushing caches and TLBs on context switches? If you
> >want that level of security you are paying plenty of performance penalties,
> >this one is lost in the noise. I see no reason not to use an off the
> >shelf CPU, even with all the restrictions that you'd have to impose it'd
> >perform better than a custom design with miniscule volumes that'd be at
> >least several years behind in both design and process. After all, it
> >isn't as though you'd be allowing execution of arbitrary code, so the
> >issues that make x86 not fully virtualizable don't matter, etc.
>
> Hmm. Flushing the level 3 cache every time you flip context briefly
> to service an event is a bit painful - you are better off with separate
> machines.
Could you give an example of a machine where a program cannot access
another programs real memory, but can (sometimes) access the other
programs data through cache memory? For example, on a PowerPC it should
be impossible, and I would consider it a bad design flaw if it were
possible.
Casper H.S. Dik
>Could you give an example of a machine where a program cannot access
>another programs real memory, but can (sometimes) access the other
>programs data through cache memory? For example, on a PowerPC it should
>be impossible, and I would consider it a bad design flaw if it were
>possible.
I don't think it's possible but what Nick might be referring to is
covert channels; on a shared system with caches, a process can communicate
with another process by alternating between cache busting and non-cache
busting behaviour.
Casper
Nick Maclaren
In article <3e085b42$0$138$e4fe...@news.xs4all.nl>,
That is one method, but there are a couple of more interesting ones;
I have used a simple form of the first to break xdm's security
(not that doing so is hard!)
You install a spy program, which uses memory and measures the
performance (intelligently). You train it by doing some known
tasks, and now look for those patterns repeating. You now have
some good information on what someone else is doing :-)
More aggressively, is that you run a program that runs a system
completely out of a particular resource and then fire off an
attack. The system then has timing or buffer size problems dealing
with it, and has to start junking data or killing processes - and
it is very likely to choose wrong!
A variant of this was traditionally used on mainframes to avoid
being charged, and worked very well. There are easy variants on
most systems to ensure that your attack is logged as a fact but all
useful evidence is lost.
Note that these approaches will necessarily have some teeth on
any architecture that does not reserve enough of all critical
resources (cache, TLBs etc.) for emergency. Using them effectively
needs a lot of knowledge, so they are rarely used nowadays - at
least not in the "public" attacks.
And so on ....
Sander Vesik
It gets worse if somebody thought that tlb miss, disk access etc timings
and frequency could be used as a sources of entropy and wrote /dev/random
accordingly.
>
> Casper
Terje Mathisen
> "Emil Naepflein" <Emil.Na...@philosys.de> wrote...
>
>>Zack Weinberg <za...@panix.com> wrote:
>>
>>>You can't allow information to leak out of the higher-privileged
>>>domain, though, which means at the very least you have to clear
>>>all of the temporary registers on kernel exit.
>>
>>You only have to do that in cases where they may have contained security
>>relevant information otherwise this is just random information which is
>>not of any use.
This is correct, however...
> It isn't random information, and even if it isn't the secrets
> you are using, it may leak information that may allow
> someone to get your secrets more easily. You'd be
> surprised at how sophisticated cryptanalysis is.
Dennis is totally right that _everything is_ security relevant, so you
have to save/restore or clear all regs anyway.
I.e. why is it up to the OS to block and/or virtualize the
TimeStampCounter on all modern x86 cpus (since the original Pentium)?
This is because it was shown many years ago that simply having a
high-resolution system-wide timer was enough to intentionally leak
information between two processes that should be in separate security
domains.
OTOH, when your OS is WinNT++, the potential risk of cross-process back
channel communication is totally lost in the noise of all the _real_
security problems. :-(
Terje
Robert A Duff
> Dennis is totally right that _everything is_ security relevant, so you
> have to save/restore or clear all regs anyway.
Why? It seems to me that if you can prove that the value of a given
register at the end of a system call is a pure function of the input
data, it is not security relevant. And therefore, the OS need not
restore the old value (or zero it or whatever) -- presuming it's
returning to the same process. The OS would simply document that
"so-and-so register is destroyed". Am I wrong?
For example, suppose a register is used to loop through a buffer whose
address was passed in. It ends up pointing one past the end of the
buffer. How could that be a security hole?
If you say, "It is wise to assume everything is security relevant,
because it's too hard to prove otherwise (and too hard to maintain the
proofs)" I could believe that. But that's a weaker statement
than what you attributed to Dennis above.
> I.e. why is it up to the OS to block and/or virtualize the
> TimeStampCounter on all modern x86 cpus (since the original Pentium)?
What do operating systems actually do about that, these days?
> This is because it was shown many years ago that simply having a
> high-resolution system-wide timer was enough to intentionally leak
> information between two processes that should be in separate security
> domains.
>
> OTOH, when your OS is WinNT++, the potential risk of cross-process back
> channel communication is totally lost in the noise of all the _real_
> security problems. :-(
Indeed.
- Bob
Casper H.S. Dik
>For example, suppose a register is used to loop through a buffer whose
>address was passed in. It ends up pointing one past the end of the
>buffer. How could that be a security hole?
Suppose a user finds a bug in the kernel which allows modification
of arbitrary data at kernel addresses, possessing interesting kernel
addresses may help. It might just the extra little bit you need.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
Robert A Duff
> Robert A Duff <bob...@shell01.TheWorld.com> writes:
>
> >For example, suppose a register is used to loop through a buffer whose
> >address was passed in. It ends up pointing one past the end of the
> >buffer. How could that be a security hole?
>
> Suppose a user finds a bug in the kernel which allows modification
> of arbitrary data at kernel addresses,
Then you're already doomed. ;-)
>... possessing interesting kernel
> addresses may help. It might just the extra little bit you need.
I suppose, but you could find those addresses other ways (reading the
source code, reading the machine code...).
Anyway, my example was not a kernel address. It points one past the end
of a buffer in the user-mode process' address space. I still say, "How
could that be a security hole?"
- Bob
Alex Colvin
>of a buffer in the user-mode process' address space. I still say, "How
>could that be a security hole?"
I think the general feeling is that it's too hard to determine whether a
register caould contain something secure. It's an unusual kind of dataflow
analysis where youre interested in preventing some difinition from
reaching any use.
Assume the kernel service code doesn't use a temporary register, but
sometimes calls subroutines which, in turn, occasionally call others. Any
of the called routines might use the register for something secret.
Remember, you could prove it can't happen for some code, but a minor
change in a subroutine far away could invalidate the proof. You can either
autoomate the analysis and run it each time (and this can be expensive),
or you can use a global policy such as assigning responsibility to the
caller or callee.
We could make erasing the secret the responsibility of the callee, but for
most internal calls that would be a waste.
It's simpler to make it the responsibility of the caller, in this case the
routine right at the boundary. That way we don't need to do any analysis
and we're less likely to forget.
If a register save/restore is a significant part of your kernel call, you
should rethink the interface. Some CPUs do provide a separate register
bank for interrupts.
--
mac the naïf
Paul Repacholi
> >Anyway, my example was not a kernel address. It points one past
> >the end of a buffer in the user-mode process' address space. I
> >still say, "How could that be a security hole?"
> I think the general feeling is that it's too hard to determine
> whether a register caould contain something secure. It's an unusual
> kind of dataflow analysis where youre interested in preventing some
> difinition from reaching any use.
And don't forget to double check all possible error paths as well!
There have been cases where a call was `secure', unless you crafted
the right error, then you had the world on a plate handed back to you!
And you WILL double check this with all future edits as well...
--
Paul Repacholi 1 Crescent Rd.,
+61 (08) 9257-1001 Kalamunda.
West Australia 6076
comp.os.vms,- The Older, Grumpier Slashdot
Raw, Cooked or Well-done, it's all half baked.
EPIC, The Architecture of the future, always has been, always will be.
Terje Mathisen
> Terje Mathisen <terje.m...@hda.hydro.com> writes:
> If you say, "It is wise to assume everything is security relevant,
> because it's too hard to prove otherwise (and too hard to maintain the
> proofs)" I could believe that. But that's a weaker statement
> than what you attributed to Dennis above.
OK, your version is more precise, even though the net effect is the same.
>>I.e. why is it up to the OS to block and/or virtualize the
>>TimeStampCounter on all modern x86 cpus (since the original Pentium)?
>
> What do operating systems actually do about that, these days?
Afaik, all OSs available on x86 cpus leave it in the default state, i.e.
freely available to user mode processes.
Nick Maclaren
Paul Repacholi <pr...@prep.synonet.com> wrote:
>al...@world.std.com (Alex Colvin) writes:
>
>> >Anyway, my example was not a kernel address. It points one past
>> >the end of a buffer in the user-mode process' address space. I
>> >still say, "How could that be a security hole?"
>
>> I think the general feeling is that it's too hard to determine
>> whether a register caould contain something secure. It's an unusual
>> kind of dataflow analysis where youre interested in preventing some
>> difinition from reaching any use.
>
>And don't forget to double check all possible error paths as well!
>There have been cases where a call was `secure', unless you crafted
>the right error, then you had the world on a plate handed back to you!
>And you WILL double check this with all future edits as well...
Yes, precisely. The other aspect is that this loophole is trivial to
close and prove you have closed, if you clobber ALL registers. Closing
the timing- and location-dependent leaks is another kettle of fish. If
you clobber only the 'security critical' registers, then you are making
this leak as tricky to deal with as the timing- and location-dependent
leaks.
But that doesn't mean that one is more important than another, because
that depends on the overall system design.
Robert Myers
>
> You're assuming caller-saves in your ABI. Last I heard, callee-saves has
> complementary advantages and disadvatages, and at least some systems I know
> well 8-) use callee-saves. For me, this has the advantage that the knowledge
> of which registers need to be saved is where it is generated: in the
> implementation of the system routine itself.
>
One obvious disadvantage of callee-saves is that it extends a chain of
trust of arbitrary length that you can trust the register data once you
get it back, thus making debugging an extraordinarily exciting task.
Otherwise, why is this not standard practice?
RM
Andy Isaacson
Emil Naepflein <Emil.Na...@philosys.de> wrote:
>On 22 Dec 2002 16:40:07 GMT, a...@pirx.hexapodia.org (Andy Isaacson)
>wrote:
>> The point you seem to be missing is that the kernel *must* set *every*
>> register to a known value (zero, or the previous value) at the syscall-exit
>> point. If it doesn't do that, then the caller can access the value that
>> was not overwritten, and it is very likely that a security breach would
>> occur as a result.
>
>info on exit. What I mean is that it isn't necessary to do it twice.
>Most OS now save the complete state with all registers on system call
>entry and restore it during exit. But most compilers do also callee
>saves for register state are used during a function. The system call
>entry code calles i.e. a C function which on entry saves the registers
>used in this function the the stack and restores them before return. So
>Even you do no complete save/restore you have the same register state.
>So why doing something twice?
>Sure there a reasons for this, i.e. debugging. But for this probably are
>also other solutions possible.
The cost of doing a complete register save/restore is extremely small.
There might be a few cycles to save here, but it's so tiny that I doubt
anyone is wasting their time attacking it -- there are much bigger
fish to fry in the syscall path.
[snip some stuff]
>Why do you think does a simple system call like getpid() on UNIX systems
>does take so long?
On Linux at least, a lot of the time is spent executing the int 0x80,
according to the thread on linux-kernel.
>One of the reasons is the save/restore of the complete state, the other
>is the generic system call code that checks thousand things.
I guess I don't understand what your goal is. If you just want to make
getpid() fast, write a wrapper that saves it in userland. Doesn't get any
faster than that. If you want to make ALL system calls fast, there's no
point in "optimizations" that only save you time on the trivial getpid()
style calls.
In fact, putting on my benchmarker hat, it's kinda *nice* that getpid()
follows the same code path as all the other system calls. It makes it
a lot easier to establish a "speed of light" measurement for system call
performance. (Of course you have to watch out for people optimizing to
the benchmark; if a big customer made decisions based on getpid() speed
then you'd see implementors doing all kinds of insane hacks to improve
getpid() speed without showing any improvement in other system calls.)
Luckily lmbench is still low-profile enough that you don't see vendors
hacking for better lmbench scores.
-andy
Terje Mathisen
> In fact, putting on my benchmarker hat, it's kinda *nice* that getpid()
> follows the same code path as all the other system calls. It makes it
> a lot easier to establish a "speed of light" measurement for system call
> performance. (Of course you have to watch out for people optimizing to
> the benchmark; if a big customer made decisions based on getpid() speed
> then you'd see implementors doing all kinds of insane hacks to improve
> getpid() speed without showing any improvement in other system calls.)
The link to the linux.kernel discussion was an eye-opener for me.
To me, it seems quite likely that Linux could get multiple different OS
call paths, some of them (like getpid) being _very_ low overhead indeed.
Stephen Fuld
"Terje Mathisen" <terje.m...@hda.hydro.com> wrote in message
news:aupjgj$t0n$1...@vkhdsu24.hda.hydro.com...
> Andy Isaacson wrote:
> > In fact, putting on my benchmarker hat, it's kinda *nice* that getpid()
> > follows the same code path as all the other system calls. It makes it
> > a lot easier to establish a "speed of light" measurement for system call
> > performance. (Of course you have to watch out for people optimizing to
> > the benchmark; if a big customer made decisions based on getpid() speed
> > then you'd see implementors doing all kinds of insane hacks to improve
> > getpid() speed without showing any improvement in other system calls.)
>
> The link to the linux.kernel discussion was an eye-opener for me.
>
> To me, it seems quite likely that Linux could get multiple different OS
> call paths, some of them (like getpid) being _very_ low overhead indeed.
I can't speak to Linux, but another OS I know of formally divides all OS
calls into three classes. Assynchronous ones return before they are
complete (basically Assynch I/O). The other classes, which comprise almost
all of the requests, are called synchronous and immediate. The immediate
ones are performed by the OS in a fast path. This is helped by the
existance of a separate register set for OS use during interrupts. This
eliminates the need for saving and restoring user registers, etc.
--
- Stephen Fuld
e-mail address disguised to prevent spam
Nick Maclaren
Terje Mathisen <terje.m...@hda.hydro.com> wrote:
>Andy Isaacson wrote:
>> In fact, putting on my benchmarker hat, it's kinda *nice* that getpid()
>> follows the same code path as all the other system calls. It makes it
>> a lot easier to establish a "speed of light" measurement for system call
>> performance. (Of course you have to watch out for people optimizing to
>> the benchmark; if a big customer made decisions based on getpid() speed
>> then you'd see implementors doing all kinds of insane hacks to improve
>> getpid() speed without showing any improvement in other system calls.)
>
>The link to the linux.kernel discussion was an eye-opener for me.
>
>To me, it seems quite likely that Linux could get multiple different OS
>call paths, some of them (like getpid) being _very_ low overhead indeed.
The optimal implementation of getpid is a single load instruction
(assuming that you don't hold it in a register). There is no reason
why such information as the pid (and even the time and resource usages)
cannot be held in a location that is readable to the program but
updatable by the kernel. But, even if it isn't, I agree that the path
can be VERY short.
Sander Vesik
>
> To me, it seems quite likely that Linux could get multiple different OS
> call paths, some of them (like getpid) being _very_ low overhead indeed.
>
Getpid is a function call - there is no reason why any but the first
would need to make a system call, and in principle it could be part of
startup environment a process gets handed anyways...
> Terje
Patrick Schaaf
>Some architecturs do the save implicitly during system call
>instruction processing (i386),
Note that this mode is not neccessarily used by contemporary
operating systems.
>others like mips give the
>choice what to save to the implementer.
x86 does the same. There's task gates, trap gates, call gates, interrupt gates.
Much choice for the implementer. And now there's SYSENTER, etc.
There's even one form (I think it was call gates) which copies a
descriptor-specified amount of data from user stack to kernel stack.
Most of the special forms are considered slower than doing it by hand,
nowadays, and they are consequently avoided.
See the linux-kernel thread, referenced some postings ago. It is about
implementing SYSENTER, which is done by the end of the thread, as far
as I can gather. From the numbers given there (e.g. some lmbench runs),
it appears that the SYSENTER path needs about half the time of the
traditional interrupt gate based syscall entry.
The avoided cost, they wrote on linux-kernel, is in not reloading
segment registers, with all their assorted checks.
best regards
Patrick
Anne & Lynn Wheeler
Emil Naepflein <Emil.Na...@philosys.de> writes:
> My goal is to reduce the system call overhead to a minimum for any
> system call. One part of the overhead is the complete save/restore of
> the register set. On architectures where there are only a few registers
> it doesn't matter much. Some architecturs do the save implicitly during
> system call instruction processing (i386), others like mips give the
> choice what to save to the implementer.
> save/restore at entry/exit and rely on callee saves/restores.
one of the things i played with very early on in cp/67 was not
fiddling the floating point registers (just the general purpose regs)
... unless there was task switch .... aka kernel made no use of them.
For some fastpaths ... also tried to just perform the necessary reload
of only general regs that fastpath interrupt handling had fiddled.
charlie tried doing something similar with control register 0. CR0 was
the virtual address table register on the 360/67. Every time into any
kernel interrupt handler it was reloaded ... even tho the value could
never have changed. also since 67's TLB was only one address space ...
and didn't have logic to see if the reload was changing any value ...
it forced a flush of all TLB entires. So he removed the reload ... and
the system started crashing. They spent a couple weeks diagnosing the
problem until they found a bug in the '67 hardware ... which had gone
undiscovered for something like six years. On page fault interrupt
from program state into supervisor state ... the hardware would reset
all the TLB entries to zero but not clear the TLB entry valid bit
(effectively mapping virtual address page zero to real kernel page
zero). This was the first time that a interrupt handler had been tried
w/o reload of CR0 (which forced invalid for all TLB entries).
in the past
http://www.garlic.com/~lynn/2002p.html#56 cost of crossing kernel/user boundary
the above description of changes to the svc interrupt handler for
savearea management & misc. other .... reduced the time spent in the
svc call interrupt processing from about 270microseconds to less than
70microseconds (including the interrupt and the return). Of course the
BALR change (for those applicable calls) collapsed it to a couple
microseconds (not counting the actual register save & restore
instructions).
--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/
Internet trivia 20th anv http://www.garlic.com/~lynn/rfcietff.htm
Casper H.S. Dik
>Terje Mathisen <terje.m...@hda.hydro.com> wrote:
>>
>> To me, it seems quite likely that Linux could get multiple different OS
>> call paths, some of them (like getpid) being _very_ low overhead indeed.
>>
>Getpid is a function call - there is no reason why any but the first
>would need to make a system call, and in principle it could be part of
>startup environment a process gets handed anyways...
But you'd need to make "fork()" and ilk call getpid() and vfork()
caused additional trouble>
Andi Kleen
> system call. One part of the overhead is the complete save/restore of
> the register set. On architectures where there are only a few registers
> it doesn't matter much. Some architecturs do the save implicitly during
> system call instruction processing (i386), others like mips give the
Umm, that's not correct. The standard syscall mechanisms used by
the common x86 OS (LCALL to interrupt gate, int 0x80, SYSENTER/SYSCALL)
do only save a minimum of registers - the stack pointer and the old
instruction pointer and old EFLAGS, including related segment registers
(SS and CS)
The only call that would possibly do a full register set switch would
be a long jump to a TSS (x86 hardware task switching), but to the best
of my knowledge no existing x86 operating system uses this anymore
(very early linux and early OS/2 did)
The saving of all the GPRs and the FPU state is fully up to the
operating system.
Of course with 7 4 byte GPRs and only ~28 byte of state (+
possibly DS and ES) it is in the "it doesn't matter matter much
on modern CPUs" range.
The Linux x86-64 implementation which has 15 8byte GPRs to worry about indeed
uses a partial stack frame (only save the C callee clobbered registers for
system calls unless a full frame is needed).
Funnily when the mechanism was designed the x86-64 ABI still
had a roughtly half/half split of caller/callee saved registers,
which made it worthwhile, but with later ABI tuning it got more
callee clobbered registers which made its gain smaller and smaller
(current split is 9:6)
On RISC with 32+ registers and IA64 this technique is very widely used.
>
> getpid() was only one example because on most system it shows the
> overhead of the generic system call path. That getpid() and other system
> calls which only read status information can be implemented so that the
> information can be provided by calling them only once and caching the
> value, or by mapping this information directly into the user space is
> clear and implemented on various systems.
That only works when you have a reliable way to invalidate the data
on fork() - otherwise the child will see wrong data
Also it is not useful for Unix variants like Linux that support clone
with different threads getting different pids.
-Andi
Nick Maclaren
Andi Kleen <fre...@alancoxonachip.com> wrote:
>>
>> getpid() was only one example because on most system it shows the
>> overhead of the generic system call path. That getpid() and other system
>> calls which only read status information can be implemented so that the
>> information can be provided by calling them only once and caching the
>> value, or by mapping this information directly into the user space is
>> clear and implemented on various systems.
>
>That only works when you have a reliable way to invalidate the data
>on fork() - otherwise the child will see wrong data
>
>Also it is not useful for Unix variants like Linux that support clone
>with different threads getting different pids.
That's slightly overstating the case. POSIX threads, for example,
carry sufficient baggage that they need a separate data area; few
modern thread models are lightweight enough that they can avoid
that, though this used not to be the case. So the technique is
almost universally applicable today.
Whether it is worth designing it into the data model is a different
matter.
Sander Vesik
>
>>Terje Mathisen <terje.m...@hda.hydro.com> wrote:
>>>
>>> To me, it seems quite likely that Linux could get multiple different OS
>>> call paths, some of them (like getpid) being _very_ low overhead indeed.
>>>
>
>>Getpid is a function call - there is no reason why any but the first
>>would need to make a system call, and in principle it could be part of
>>startup environment a process gets handed anyways...
>
> But you'd need to make "fork()" and ilk call getpid() and vfork()
> caused additional trouble>
I didn't think of vfork - my bad 8-(
As such a change is essentialy changing the way certain part of process
state is handled, other functions dealing with process state will need
to change. It is probably easier to just make it a data oage that is
readbale from user space and gets copied as needed (this will complicate
vfork implementation slightly). On the benefit side a whole bunch of
information query functions/syscalls become exteremely simple leaf
functions instead.
>
> Casper
Andy Glew
> implementing SYSENTER, which is done by the end of the thread, as far
> as I can gather. From the numbers given there (e.g. some lmbench runs),
> it appears that the SYSENTER path needs about half the time of the
> traditional interrupt gate based syscall entry.
Can someone repost, or mail me, the pointer to the linux-kernel thread?
I apparently skipped that.
I may have gone a little bit too far with SYSENTER/SYSEXIT.
My goal was to make something that could execute in the same
time as one or two ALU instructions, in some hypothetical
not-yet-implemented system with full privilege renaming.
To that end, I stripped out everything unnecessary from SYSENTER,
including the return linkage. I observed that the transition to the
kernel is usually done inside a library, DLL or .lib.so,
so the user return address was already on the stack. In that sense,
having the SYSENTER/SYSEXIT instruction save it's return address
is redundant - if there is only one such shared library entry point,
always at a fixed address, then the kernel can know where to return to.
But, this is stretching a bit too far. AMD's more conventional
SYSCALL/SYSRET are just plain more familiar. They save a return linkage.
This allows SYSCALL/SYSRET to be inlined; it allows more than one
DLL to make a given system call. It means that the kernel is more
independent of DLLs. It means that different languages can have
their own DLL libraries, instead of everyone having to adapt to
the DLL format of the main language. {Ironically, so far as I know,
most people still go to a single DLL or .lib.so.}
Theoretically, a SYSENTER/SYSEXIT could be a little bit faster than
SYSCALL/SYSRET. But on present machines, with no privilege renaming,
there's negligible difference.
Mea culpa: I plead guilty to excessive aggressiveness.
Amusing anecdote: after a visit to an OS vendor, I was hanging around
chatting with a friend who introduced me to a guy who was just implementing
SYSENTER/SYSEXIT and SYSCALL/SYSRET support.
He complained that it was hard to guarantee recoverability in
SYSENTER/SYSEXIT
for something like machine check, and that he had had to use a gross kluge
to obtain the required atomicity. (I can't remember exactly what the
problem
was - just the usual sort of atomicity problem.)
I pointed out that a task gate provided nearly everything necessary [*].
He said something like "Umm, yeah, that would work. Well, I've already
gotten this gross kluge for Intel SYSTENTER/SYSEXIT working.
But I can use the same trick for AMD SYSCALL/SYSRET."
I.e. I inadvertently got the AMD code optimized more than the Intel code.
I don't know if Intel ever got that fixed.
---
[*] For recoverability, you need an interrupt that makes no assumptions
about the state in which the processor is running. It cannot make
assumptions such as "you are running in 32 bit protected mode, with
caches and TLBs enabled", etc. It needs to load everything on which the
handler code implicitly depends. Task gate is about as close as x86 gets.
Conversely, for efficiency, you don't want to load anything that is
already loaded.
I'm still looking to get a generic mechanism that handles both
into the instruction set.
Nick Maclaren
In article <g2TQ9.5800$O84.25...@newssvr14.news.prodigy.com>,
"Andy Glew" <andy_gle...@yahoo.com> writes:
|>
|> Amusing anecdote: after a visit to an OS vendor, I was hanging around
|> chatting with a friend who introduced me to a guy who was just implementing
|> SYSENTER/SYSEXIT and SYSCALL/SYSRET support.
|> He complained that it was hard to guarantee recoverability in
|> SYSENTER/SYSEXIT
|> for something like machine check, and that he had had to use a gross kluge
|> to obtain the required atomicity. (I can't remember exactly what the
|> problem
|> was - just the usual sort of atomicity problem.)
|> I pointed out that a task gate provided nearly everything necessary [*].
|> He said something like "Umm, yeah, that would work. Well, I've already
|> gotten this gross kluge for Intel SYSTENTER/SYSEXIT working.
|> But I can use the same trick for AMD SYSCALL/SYSRET."
|>
|> about the state in which the processor is running. It cannot make
|> assumptions such as "you are running in 32 bit protected mode, with
|> caches and TLBs enabled", etc. It needs to load everything on which the
|> handler code implicitly depends. Task gate is about as close as x86 gets.
|> Conversely, for efficiency, you don't want to load anything that is
|> already loaded.
|> I'm still looking to get a generic mechanism that handles both
|> into the instruction set.
Yes. This happens at many levels. One reason that I don't like
too much optimisation here is that it often means that you can do
certain recovery operations only when the privileged side of the
boundary. In turn, that means that the author of the unprivileged
side cannot do certain things unless the author of the privileged
side has exported a suitable primitive. It also encourages code
bloat on the privileged side (i.e. putting things there that really
should be unprivileged).
For example, there is absolutely no reason why an application
shouldn't be able extend its own instruction set by emulation. It
is obvious that no privilege should be needed, as no other thread
is affected and it doesn't need to do anything that it couldn't
do anyway by another method. Yet it is often impossible to do
this fully transparently without privilege.
Patrick Schaaf
>> See the linux-kernel thread, referenced some postings ago. It is about
>> implementing SYSENTER, which is done by the end of the thread, as far
>> as I can gather. From the numbers given there (e.g. some lmbench runs),
>> it appears that the SYSENTER path needs about half the time of the
>> traditional interrupt gate based syscall entry.
>Can someone repost, or mail me, the pointer to the linux-kernel thread?
>I apparently skipped that.
Here's the point where Linus started coding:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.2/index.html#234
Note, especially, the following discussions between Linus and Ulrich
Drepper (glibc maintainer), fleshing out the details over some days.
The result is in Linus' current development kernel, and there's a glibc
available from the following link:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.3/0624.html
best regards
Patrick
Joseph Seigh
Patrick Schaaf wrote:
>
> "Andy Glew" <andy_gle...@yahoo.com> writes:
...
> >Can someone repost, or mail me, the pointer to the linux-kernel thread?
> >I apparently skipped that.
>
> Here's the point where Linus started coding:
>
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.2/index.html#234
>
> Note, especially, the following discussions between Linus and Ulrich
> Drepper (glibc maintainer), fleshing out the details over some days.
>
> The result is in Linus' current development kernel, and there's a glibc
> available from the following link:
>
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.3/0624.html
>
Invites a comparison to sausage making. Ugh. Brings back bad memories of
work on another OS.
Was this a linux/intel thread all along? I though it was a theoretical discussion
of whether kernel calls had to cost as much as they did.
Anyway there appear to be multiple issues involved, some of which you can
do something about, some that you can, and some where it doesn't matter.
Stack switching. I have a hard time believing this can be an issue. We're
talking about at most a few instructions.
Hardware error checkpointing. This is so errors that happened in user mode don't
get expressed later in kernal mode where you may have to assume kernel damage
even though there may be none. Not much you can do about the overhead from this.
It's built into the hardware.
I'm assuming we're talking about things that require privilege level promotion.
Joe Seigh
Terje Mathisen
> "Andy Glew" <andy_gle...@yahoo.com> writes:
>
>
>>>See the linux-kernel thread, referenced some postings ago. It is about
>>>implementing SYSENTER, which is done by the end of the thread, as far
>>>as I can gather. From the numbers given there (e.g. some lmbench runs),
>>>it appears that the SYSENTER path needs about half the time of the
>>>traditional interrupt gate based syscall entry.
>
>
>>Can someone repost, or mail me, the pointer to the linux-kernel thread?
>>I apparently skipped that.
>
>
> Here's the point where Linus started coding:
>
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.2/index.html#234
>
> Note, especially, the following discussions between Linus and Ulrich
> Drepper (glibc maintainer), fleshing out the details over some days.
This was amazing, really: Changing a _very_ basic assumption of the OS,
and get it to work reliably and efficiently in a matter of hours/days.
:-)
BTW: ANDY GLEW, PLEASE EMAIL ME DIRECTLY!
Andy Glew
discussion
> of whether kernel calls had to cost as much as they did.
I certainly did not mean to make this into an Intel specific thread.
SYSENTER/SYSEXIT is just, so far as I know, the most extreme attempts
to make the raw instruction fast, without going all the way to
privilege renaming, etc.
System calls are just function calls with a privilege change.
They should be no more expensive that calling a library function.
They really aren't all that hard to make run fast.
It's like running in molasses.
> Anyway there appear to be multiple issues involved, some of which you can
> do something about, some that you can, and some where it doesn't matter.
>
> I'm assuming we're talking about things that require privilege level
promotion.
Yep. I think renaming the privilege level is a definite requirement,
and/or predicting the privilege level to be entered, if the given interface
can go to multiple privilege levels.
Similarly, if interrupt blocking is required, interrupt mask renaming.
> Stack switching.
> Hardware error checkpointing.
Stack switching is easy enough, but coupled to error fencing
Anne & Lynn Wheeler
> That is standard on most systems today because the processors support
> this by generating a trap when the registers were not saved and a new
> task wants to access floating point registers.
the place that i first saw such hardware assist was for the registers
in the 3090's optional vector processor.
Nick Maclaren
Andy Glew <andy_gle...@yahoo.com> wrote:
>
>System calls are just function calls with a privilege change.
>They should be no more expensive that calling a library function.
>
>They really aren't all that hard to make run fast.
>It's like running in molasses.
Yes, precisely.
Similarly, interrupts into user space (e.g. floating-point traps
and external signals) need be no more expensive than a function call
plus an unpredicted branch. But they would have to be designed in
as first order facilities to achieve that.
Actually, an external signal can be delivered almost for free if you
are prepared to delay it until the current pipeline drains. If this
sort of really lightweight communication were more widely available,
we might see some of the high-communication programming paradigms
taking off.
But starting with system calls seems good to me ....
Andy Glew
That was my intention: start with SYSENTER/SYSEXIT
( or maybe SYSCALL/SYSRET, :-( )
and then eventually get around to optimizing interrupts.
Unfortunately, eventually has not yet arrived.
Some observations about optimizing interrupts and system calls:
(a) people sometimes try to optimize the most important special
cases of interrupts, but do not provide a mechanism that works
in general. This leads to godawful kluges for the general case,
which ironically often hurt the special cases.
I call this "Berkeley 80% solutions", in memory of BSD
and Bill Joy.
I prefer first to make them correct, then make them fast.
(b) people often obsess about reducing the register saves and restores,
but forget about the pipeline effects, and then the OS overhead.
Andy Glew
wrt you dissing me, SYSENTER, NMI, and task gate.
http://www.uwsg.iu.edu/hypermail/linux/kernel/0212.2/0234.html
[comp.arch discussion]
> > See the linux-kernel thread, referenced some postings ago. It is about
> > implementing SYSENTER, which is done by the end of the thread, as far
> > as I can gather. From the numbers given there (e.g. some lmbench runs),
> > it appears that the SYSENTER path needs about half the time of the
> > traditional interrupt gate based syscall entry.
>
> Can someone repost, or mail me, the pointer to the linux-kernel thread?
> I apparently skipped that.
>
> I may have gone a little bit too far with SYSENTER/SYSEXIT.
I haven't yet read the linux-kernel thread, but apparently Linus
was dissing the designer of SYSENTER/SYSEXIT - me - for
the fact that it neither changes the kernel stack
nor blocks NMI.
This is undoubtedly the problem that the OS developer I described
in an earlier post was having.
The problem arises because there is a window in which NMI is not blocked
where the user stack is current, and where, if NMI uses a conventional
interrupt
gate, the NMI would be delivered on the untrusted user stack. Potentially a
security
hole. This happens because such gates switch to a special stack if the
interrupt
occurs in user mode, but not if the interrupt occurs in kernel mode;
so if the interrupt occurs while the kernel has a user stack pointer, blam!
Conventional interrupts are masked, but NMIs are not; NMIs are, after all,
non-maskable!
As I described in an earlier post, I believe the correct fix is to use
the x86's task gate, which *always* changes to a new stack dedicated
to the interrupt. The x86's task gate is ugly, but it is the closest
thing to a generic mechanism. There MUST (or, at least, strong should)
be an interrupt mechanism that does not assume that the state being
interrupted is consistent. Else, at the very least, you have no chance
of recovering from machine check errors that do things like corrupt
a kernel stack linkage stored in memory. One of the uses of NMI is
for external machine check like error reporting, watch dog timers, etc;
it should use the safe mechanism.
Yes, it is possible to kluge something that makes NMIs after syscalls
work in most situations. But, the most general case requires an interrupt
that carries its stack.
---
Why did I make SYSENTER behave this way?:
As mentioned earlier, I was trying to strip everything possible out of
SYSENTER.
Manipulating the stack is one thing not strictly necessary. There was a
known,
documented, approach for handling NMIs - task gates - but many OS developers
seemed to try hard to avoid using them because they are butt-ugly.
Me, I hoped to them better eventually.
In retrospect, I probably pushed too hard on SYSTENTER, for no immediate
benefit.
The alternative approach is to block NMIs briefly. But that is a slippery
slope.
---
By the way, I started scanning the linux-kernel thread,
and apart from Linus complaining about NMI and SYSENTER,
there was a lot of discussion about calling convention,
when and what registers need to be saved, etc.
It's just an example of what I want a generalized task gate like mechanism
to do: provide a generic way of getting from any state to any other state
that is safe against interrupts, machine checks, signals, etc.
It may be possible to do this all in macroinstructions, but at the least
it should be proven to be safe, and documented.
There have ben a lot of security holes caused by people
getting this not quite right.
First get it right. Then make it fast.
Andi Kleen
"Andy Glew" <andy_gle...@yahoo.com> writes:
>
> The problem arises because there is a window in which NMI is not blocked
> where the user stack is current, and where, if NMI uses a conventional
> interrupt
> gate, the NMI would be delivered on the untrusted user stack. Potentially a
> security
> hole. This happens because such gates switch to a special stack if the
> interrupt
> occurs in user mode, but not if the interrupt occurs in kernel mode;
> so if the interrupt occurs while the kernel has a user stack pointer, blam!
Are you sure you're talking about the SYSENTER that is currently
in Intel/AMD CPUs ? It has an MSR that sets the ring0 stack pointer
that is set by SYSENTER. It is AMD's SYSCALL that doesn't change
the stack in the instruction.
Changing the stack forcingly has a few problems:
- you need to move RSP away before doing the syscall, stealing
another register (EBP in Linux's case). That costs you one argumet
that could be passed in registers.
- the OS has a per process/thread kernel stack. The MSR is global
however, so it has two choices: keep the MSR constant and use a dummy stack
per CPU and then switch to the real stack in software, or alteratively
rewrite the MSR on each context switch. Context switch is an even
hotter path than system calls, so anything slowing it down is bad.
Unfortunately WRMSR tends to be rather costly in most x86
(50-100 cycles) on the last benchmarks I did.
I think Linus' original implementation had a per cpu stack that
showed an one instruction window for NMIs seeing a non process
stack.
Current Linux/i386 rewrites the MSR on context switch now.
IMHO a bad trade off, because it slows down the context
switch.
Given that I think the current implementation of SYSENTER
with its force stack register rewriting is bad.
I actually prefer SYSCALL because of that
Another problem in SYSENTER that SYSCALL also avoids is EFLAGS
handling. The OS has to go through some goops to make sure TF
is really off. The x86-64 SYSCALL has a MSR for a EFLAGS mask
which avoids this nicely (Athlon SYSCALL has it fixed, but to
an useful default mask at least)
-Andi (who implemented SYSCALL support code for linux/x86-64, so
spent some time thinking about all the issues and may be a bit biased.
Fortunately x86-64 has some extensions to avoid task gates and make
it more efficient)
Nick Maclaren
In article <yAaR9.2761$093.21...@newssvr21.news.prodigy.com>,
"Andy Glew" <andy_gle...@yahoo.com> writes:
|> > But starting with system calls seems good to me ....
|>
|> That was my intention: start with SYSENTER/SYSEXIT
|> ( or maybe SYSCALL/SYSRET, :-( )
|> and then eventually get around to optimizing interrupts.
|>
|> Unfortunately, eventually has not yet arrived.
|>
|> Some observations about optimizing interrupts and system calls:
|>
|> (a) people sometimes try to optimize the most important special
|> cases of interrupts, but do not provide a mechanism that works
|> in general. This leads to godawful kluges for the general case,
|> which ironically often hurt the special cases.
|> I call this "Berkeley 80% solutions", in memory of BSD
|> and Bill Joy.
|> I prefer first to make them correct, then make them fast.
Gods, YES! Having had to code some tests to determine whether this
or that case has been got right, and then having tried to find SOME
way of fixing up the mess, I support you 100%.
It isn't uncommon for the fixup code to be 100 times as large and
10 times as slow (even on average) than fixing the problem properly.
But it is done by someone lower down the food chain :-(
|> (b) people often obsess about reducing the register saves and restores,
|> but forget about the pipeline effects, and then the OS overhead.
Yes. One reason I don't like saving registers in dribs and drabs
is that it is common to glitch the pipeline every time you restore
any, because the code uses one of the restored registers shortly
afterwards. And, of course, by the time you return, the first-level
cache has often had enough action to have replaced those entries by
others.
I have had a couple of heated arguments where people were obsessed
by optimising the calling sequence at the expense of making the
return sequence much slower. One or two seriously said that calls
are much more common than returns!
Andy Glew
> in Intel/AMD CPUs ? It has an MSR that sets the ring0 stack pointer
> that is set by SYSENTER. It is AMD's SYSCALL that doesn't change
> the stack in the instruction.
I am talking about Intel SYSENTER, which I admit blame for,
but I also admit that I haven't looked at in a long time,
and also that it changed in minor ways after I left.
Linus complained that there was a one cycle window after
SYSENTER in which NMIs were delivered inconsistently.
I was trying to remember why that was. Apparently
I should shut up and refresh my memory.
Things page back in slowly. The canonical privilege in x86
is actually the RPL of the stack segment, so unless you are
going to create a new mechanism you have to have an SS.
Similarly, you need a CS and EIP. So having an ESP is natural
- and I occasionally do things for aesthetics. Moreover,
I was hoping to move towards a place where CS:EIP and SS:ESP
where co-renamed.
> Another problem in SYSENTER that SYSCALL also avoids is EFLAGS
> handling. The OS has to go through some goops to make sure TF
> is really off. The x86-64 SYSCALL has a MSR for a EFLAGS mask
> which avoids this nicely (Athlon SYSCALL has it fixed, but to
> an useful default mask at least)
EFLAGS is just yet another thing to manipulate, that's not necessary
in many situations. You can handle TF by (a) linking in a special syscall
stub when debugging, and (b) having code in the debug handler.
The debug handler can, again, be made safe using a task gate.
I wanted to avoid manipulating EFLAGS, because my goal was
to make SYSENTER as fast as possible - what AMD calls a
fast path instruction. SYSENTER manipulated CS:EIP and SS:ESP.
That's two register writes if co-renamed, already a slightly CISCy
instruction.
(I wanted to avoid changing SS:ESP, so that it could be just a
single register write if CS:EIP were co-renamed, but microcode\
thought that would violate invariants about SS RPL they wanted to maintain.)
Adding EFLAGS would make a third register writing, moving
it further away from my goal of SYSENTER being implementable
as fast as, say, a 128 bit register transfer.
All this being said, while it is certainly POSSIBLE to not manipulate
EFLAGS if the software is written appropriately, some software did
not want to get written that way. Like I said: I was probably a bit too
aggressive with SYSENTER/SYSEXIT: if I learned anything from
that, it is that is often better to add stuff that is not, strictly
speaking,
necessary, in order to make software's life easier.
Of course, the ideal would have been to co-develop the software and
SYSENTER. Obviously impossible for LINUX: SYSENTER was defined
in 1992, but LINUX only got around to using it last year, 2001.
Even for the OSes that we worked closely with, there was a big delay.
Another caution: when you deal with multiple OSes, knowing that
something is acceptable to one OS group does not mean that it is
acceptable to a different OS group.
Linus Torvalds
Andy Glew <andy_gle...@yahoo.com> wrote:
>
>I may have gone a little bit too far with SYSENTER/SYSEXIT.
Yes.
>My goal was to make something that could execute in the same
>time as one or two ALU instructions, in some hypothetical
>not-yet-implemented system with full privilege renaming.
That's an admirable goal. HOWEVER, it's pretty much a _pointless_ goal,
if it then means making other things slower.
>To that end, I stripped out everything unnecessary from SYSENTER,
>including the return linkage. I observed that the transition to the
>kernel is usually done inside a library, DLL or .lib.so,
>so the user return address was already on the stack.
That is true, but it means, for example, that sysexit cannot be used for
any other system call return path. In particular, _every_single_OS_
needs to return to other points for things like starting a new execution
image (execve()) or taking some kind of signal.
Which means that you saving the return linkage means that now the _user_
has to save the return linkage, and put it in a place where the OS can
access it to change it. Or, in the case of the Linux implementation, we
end up saying "screw it", and just use "iret" instead.
So your two-cycle implementation caused signals to instead of two cycles
take several _hundred_ cycles just for the iret.
And yes, signals are a performance issue.
> In that sense,
>having the SYSENTER/SYSEXIT instruction save it's return address
>is redundant - if there is only one such shared library entry point,
>always at a fixed address, then the kernel can know where to return to.
That's what Linux does. But when it doesn't work, it sucks. However, it
doesn't suck _too_ badly, because it still optimizes for the common case.
>But, this is stretching a bit too far. AMD's more conventional
>SYSCALL/SYSRET are just plain more familiar. They save a return linkage.
>This allows SYSCALL/SYSRET to be inlined; it allows more than one
>DLL to make a given system call. It means that the kernel is more
>independent of DLLs. It means that different languages can have
>their own DLL libraries, instead of everyone having to adapt to
>the DLL format of the main language. {Ironically, so far as I know,
>most people still go to a single DLL or .lib.so.}
Yes, SYSCALL/SYSRET is better in this respect. HOWEVER:
>Amusing anecdote: after a visit to an OS vendor, I was hanging around
>chatting with a friend who introduced me to a guy who was just implementing
>SYSENTER/SYSEXIT and SYSCALL/SYSRET support.
>He complained that it was hard to guarantee recoverability in
>SYSENTER/SYSEXIT
>for something like machine check, and that he had had to use a gross kluge
>to obtain the required atomicity. (I can't remember exactly what the
>problem was - just the usual sort of atomicity problem.)
Machine checks are irrelevant, they aren't performance critical, and you
_want_ to reset the whole machine, so something like a task gate is
acceptable for them.
However, NMI's are _not_. And SYSCALL/SYSRET is totally broken wrt NMI,
unless SYSENTER/SYSEXIT.
I can live with SYSENTER/SYSEXIT - as you say, you can force a stable
return point for the majority of all cases. So optimize for that, fair
enough. It's inconvenient, and the fact that sysenter/sysexit destroys
other information too (eflags - VM86 mode info etc) makes it _really_
inconvenient. It should have GP'd, but whatever.
HOWEVER, what syscall/sysret does is unforgivable.
NMI's are performance-critical, since the only really valid use for them
tends to be profiling. And profiling should be _fast_ dammit. Task
gates are simply not a good idea.
> I'm still looking to get a generic mechanism that handles both
>into the instruction set.
If sysenter had just saved the few things it crapped on (eflags, esp,
eip, cs, ss) on the stack, it would be perfect. Or at least GP'd on the
known illegal cases (cs != SYSENTER_USER_CS, ss != SYSENTER_USER_SS,
vmflags & VM86).
Throwing information away is a BAD BAD BAD thing.
Linus
Rupert Pigott
news:av4pso$1r0$1...@penguin.transmeta.com...
[SNIP]
> Throwing information away is a BAD BAD BAD thing.
Amen.
John F. Carr
Andy Glew <andy_gle...@yahoo.com> wrote:
>
>I certainly did not mean to make this into an Intel specific thread.
>SYSENTER/SYSEXIT is just, so far as I know, the most extreme attempts
>to make the raw instruction fast, without going all the way to
>privilege renaming, etc.
>
>
>System calls are just function calls with a privilege change.
>They should be no more expensive that calling a library function.
On the original RS/6000 the svc/svcr instruction pair took about
four cycles, compared to about zero for a direct call and return.
Notably, that was less time than a pair of pipeline drains. There
was a vectored system call (caller selects one of a large number
of entry points) that could be used to reduce overhead for simple
calls. I think atomic compare-and-swap was implemented as a fast
system call.
PowerPC removed the vectored system call instruction. I don't know
how fast svc is these days.
--
John Carr (j...@mit.edu)
Anne & Lynn Wheeler
j...@mit.edu (John F. Carr) writes:
> On the original RS/6000 the svc/svcr instruction pair took about
> four cycles, compared to about zero for a direct call and return.
> Notably, that was less time than a pair of pipeline drains. There
> was a vectored system call (caller selects one of a large number
> of entry points) that could be used to reduce overhead for simple
> calls. I think atomic compare-and-swap was implemented as a fast
> system call.
compare&swap is supposed to be atomic ... non-interruptable and
provide serialization in an smp environment. when original
compare&swap was proposed (for 370) ... challenge by the owners of the
370 architecture was to come up with various methodologies for using
compare&swap in uniprocessors environments ... which was born the
description for using it for various kinds of "safe" queue updating in
multi-threaded applications. the 6000/rios didn't have compare&swap
primitive and didn't have multiprocessing support ... the only thing
needed for simulating compare&swap for multi-threaded applications was
to prevent interrupts. the fast simulation in the svc handler was a
way of getting all interrupts turned off from application mode during
the simulation of the instruction.
old trivia ... and of course compare&swap mnemonic was chosen because
they are charlie's initials.
Andy Glew
> has to save the return linkage, and put it in a place where the OS can
> access it to change it. Or, in the case of the Linux implementation, we
> end up saying "screw it", and just use "iret" instead.
>
> So your two-cycle implementation caused signals to instead of two cycles
> take several _hundred_ cycles just for the iret.
>
> And yes, signals are a performance issue.
What were you using before SYSENTER/SYSEXIT?
Why did adopting SYSENTER/SYSEXIT mean that
you had to use a slower than the previous mechanism
for signals?
Yes, I agree that signals are a performance issue.
Next on my agenda[*] is to make interrupts
and interrupt returns faster. They don't need
to be as slow as they are... but, interrupts and
interrupt returns are more complicated tha
system calls and system call returns, so they
should be slower.
> However, NMI's are _not_. And SYSCALL/SYSRET is totally broken wrt NMI,
> unless SYSENTER/SYSEXIT.
I'm having trouble parsing this.
> NMI's are performance-critical, since the only really valid use for them
> tends to be profiling. And profiling should be _fast_ dammit. Task
> gates are simply not a good idea.
I sympathize with the use of NMIs for profiling.
I've used them that way myself.
Let me point out, however, that NMIs are also traditionally
used for external machine checks - watchdog timers.
Yes, external machine checks are infrequent, but they
still need to be correct.
I think the issue here is:
* profiling interrupts should be fast
* profiling interrupts use NMIs
* external machine checks also use NMIs
* external machine checks using NMIs need to be
able to handle inconsistent machine states
* profiling interrupts can run on a system that
is overall correct and not consistent - you are
not worried about profiling inconsistent states,
but mainly correct execution.
(Caveat: get deeply into reliability and you
will be profiling the machine check recovery handlers.)
* one big problem with SYSENTER/SYSEXIT and SYSCALL/SYSRET
is that they made an inconsistent state that NMI needs to be able
to handle common in a correctly operating OS.
> > I'm still looking to get a generic mechanism that handles both
> >into the instruction set.
>
> If sysenter had just saved the few things it crapped on (eflags, esp,
> eip, cs, ss) on the stack, it would be perfect. Or at least GP'd on the
> known illegal cases (cs != SYSENTER_USER_CS, ss != SYSENTER_USER_SS,
> vmflags & VM86).
If SYSENTER/SYSEXIT had implemented this saving, and these checks,
with microcode, SYSENTER/SYSEXIT would be not much faster than IRET.
The checks, in particular, were difficult.
If I had been allowed to build special purpose hardware to implement the
checks,
that objection would have ben removed. The one thing that hardware can do
faster
than software/microcode is such funky checks. But nobody wanted to spend the
hardware to make this unproven idea faster.
Q: was speeding up system calls and system call returns a good idea?
Is it a net win? If you and enough other people think so, then there will
be justification for improved, and more convenient, versions of
SYSENTER/SYSEXIT
and SYSCALL/SYSRET. If we are lucky we can retire the old ones...
if we are unlucky, we will have to keep the old ones around.
Similarly, generating justification for faster interrupts and interrupt
returns.
It is easy to design an ideal syscall/ret and interrupt/iret.
The hard thing is getting them built.
I gave up on getting Intel's committees to jump to the
ultimate answer quickly. I resigned myself to incremental
evolution, occasionally taking steps that sem locally retrograde
with an eye on the deeper valley over the hill.
AMD is able to move quicker in such things,
but still can't jump to the ultimate answer - it is, after all,
a company.
===
[*] I freely admit that I have an agenda.
I want to make the cost of crossing privilege domain boundaries
as inexpensive as possible, because it is the right thing to do.
Even if present day workloads don't always show the need,
the software press is full of problems that can be solved by
cheaper privilege domains. The companies that meet this
need will do better in the market place - eventually.
The problem is not going overboard, and spending too much
hardware on stuff that there is not at present demand for.
You can't lead the market by too much - but you have to be
able to respond when the market asks, and not have to go
off and study a new need for years before satisfying it
if your competitors are ready to respond immediately.
I have other agendae:
* improving memory copies
* making linked data structure codes run fast
(as well as similar agendae in software development
that I am further away from getting anywhere with).
I'm willing to admit to these agendae in a public forum because
they are personal agendae, not necessarily those of my employers
past or present, and because anyone who worked with me
at Intel, Motorola, or Gould already knows that I have them.
They've been posted on my web page since I first had a web page.
I'm not worried about twigging Intel to my agendae,
because, frankly, Intel will just set up a task force
or committee to study it, and they don't have enough
people smart in these things to sit on such a committee,
so the non-smart people will control it.
******
Lest it not be totally obvious:
This post is my personal opinion. It admits to some of my personal agendae
for improving computer architecture, which I hope to put into practice
in my work, but it does NOT reflect the official position or plans of my
employers, past, present, or future.
Andy Glew
> they are charlie's initials.
You've said this before.
What was Charlie's full name?
Anne & Lynn Wheeler
"Andy Glew" <andy_gle...@yahoo.com> writes:
> You've said this before.
> What was Charlie's full name?
past answers (compare&swap mnemonic ... includes pointers to POP and
programming notes):
http://www.garlic.com/~lynn/99.html#89 FIne-grained locking
http://www.garlic.com/~lynn/2001f.html#41 Test and Set (TS) vs Compare and Swap (CS)
somewhat aside .... during project athena days (half funded by dec,
half funded by ibm) .... jerry saltzer was director ... charlie was
asst. director (or whatever they called the position) on the ibm side
... and i forget who it was on the dec side?
when we were doing ha/cmp ... we subcontracted much of the software
development to company that started out as some number of people
having worked at project athena (all within half mile or so
of tech sq). other past trivia
http://www.garlic.com/~lynn/2000d.html#68 "all-out" vs less aggressive designs
in the above, i watched as they pried the letters off the building
... i keep thinking that i needed to ask the guy if I could have one
of the letters.
other stuff from by-gone days:
http://www.garlic.com/~lynn/subtopic.html#545tech
Michael S
>
> [*] I freely admit that I have an agenda.
> I want to make the cost of crossing privilege domain boundaries
> as inexpensive as possible, because it is the right thing to do.
> Even if present day workloads don't always show the need,
> the software press is full of problems that can be solved by
> cheaper privilege domains. The companies that meet this
> need will do better in the market place - eventually.
>
> The problem is not going overboard, and spending too much
> hardware on stuff that there is not at present demand for.
> You can't lead the market by too much - but you have to be
> able to respond when the market asks, and not have to go
> off and study a new need for years before satisfying it
> if your competitors are ready to respond immediately.
>
I probably know 100 times less than you about system calls. However I
also have an opinion. In my opinion, you would better invest you
energy improving performance of existing system call mechanisms,
especially call gate, instead of inventing a new ones. Interrupt gates
and trap gates are not intended for system calls. Call gates are. If
call gate/far return pair would have a fair performance (not
necessarily blazingly fast) there would be no need for anything else.
BTW, you don't have to optimize for general case. Optimization for the
common case is good enough. Optimization for the common case = caching
one segment descriptor per privilege level per segment register.