FOS: Final Operating System
The DeerBear
I'm thinking to develop a new operating system which will be
quite different from the others you can find around now.
This system differs from the others in that it can simulate any
known operating system on Earth by using an "execution driver" which
interprets the executable code of an application and transforms it in
machine code to be executed natively.
In this way you could even load Windows DLLs, keeping the
whole system active and have - say - Linux and Windows running at the
same time.
I have no previous experience in operating systems development, thus I'd
need LOTS of practical help to get started in first instance.
This operating system would also have several new IPC technologies, like
what I call "bubbles", that stay suspended in air with no particular owner
and can
then be made exploding, being also able to generate other bubbles.
This technique would allow a new sprint in AI computing, I think.
I would need references to device programming, tech sheets, etc...
Any good pointer appreciated.
TIA,
Andrew
P.S. If someone wants to jump up, just email me
Filip Janiszewski
> known operating system on Earth by using an "execution driver" which
> interprets the executable code of an application and transforms it in
> machine code to be executed natively.
>
> In this way you could even load Windows DLLs, keeping the
> whole system active and have - say - Linux and Windows running at the
> same time.
:)))
Good Luck..
Spike
wrote:
Well, this seem's quite *big*... What programming language do you
know?
//Spike
The DeerBear
> Well, this seem's quite *big*...
It doesn't *seem* big... it *IS* big <g>.
> What programming language do you know?
C/C++,Pascal,Intel Assembler... the whole thing would run
under Intel architecture.
> //Spike
Andrew
The DeerBear
> iedll.dll error at 0x00000000. memory cannot be written. Rebooting
> Linux. :)))
LOL
Come on, could be a *very* nice project... but I have *no* experience in
building operating systems... well, yeah, I know much of the theory. but
it's
unrealistic to think I can build an OS from scratch without real experience
on the
topic.
> Good Luck..
Thnx, will need it :-)
Andrew
Spike
wrote:
>Spike wrote:
For a project this big I recommend C/C++.
Great tutorials can be found at www.osdever.net,
http://www.mega-tokyo.com/osfaq2/index.php?WriteYourOwnOS,
http://www.nondot.org/sabre/os/articles.
Start reading :)
btw, Are you going to make your own bootsector or will you use a
ready-made (like GRUB)? What do you know about OSD?
//Spike
The DeerBear
> For a project this big I recommend C/C++.
I was oriented on this choice as well
> Great tutorials can be found at www.osdever.net,
Thnx, I'll have a look at it
> http://www.mega-tokyo.com/osfaq2/index.php?WriteYourOwnOS,
Uh? Will look at this, also
> http://www.nondot.org/sabre/os/articles.
And to this too :-)
> Start reading :)
Yeah! ^_^
> btw, Are you going to make your own bootsector or will you use a
> ready-made (like GRUB)?
I'm unsure about this. Grub could be a good choice, but developing my own
may
well be an interesting choice... I'll probably start with GRUB and later
on - maybe -
develop my own.
> What do you know about OSD?
On the practical side, nothing, I never applied my "knowledge", mainly
because of
self-inconfidence <g>.
On the theory side, I know scheduling, file systems, threading, device
drivers, etc.
But - again - I have NO practical experience and the project is quite huge.
I don't expect, however, to finish it so soon <g>.
I know it'll require some years to get to a nicely shaped point.
> //Spike
Andrew
Tim Robinson
news:bpkvva$1pl1mo$1...@ID-170738.news.uni-berlin.de...
> Come on, could be a *very* nice project... but I have *no* experience in
> building operating systems... well, yeah, I know much of the theory. but
> it's
> unrealistic to think I can build an OS from scratch without real
experience
> on the
> topic.
You've summed it up very well. You've got no experience in OS development,
yet you want to build an OS, which can run programs from every other OS,
from scratch. In effect, you want to rewrite every other OS.
My advice is: aim low to begin with. Come up with a really simple design.
Begin to implement it -- it will be much harder than you think. You will be
learning as you go along. After a few weeks, you will have learned enough to
realise that your original design is rubbish. Start again. Repeat the
process until you get so demotivated that you give up, or until you reach
enlightenment.
There are plenty of OS development resources around nowadays, and places
such as alt.os.development, web boards and IRC channels are full of
experienced and not-so-experienced OS developers who may decide to answer
some of your questions. But nobody is going to tell you exactly what to do.
Nobody will sit down and teach you for free.
In summary: OS development is *really* *hard*, and you'll need to prepare
for plenty of hard work, on your own, with only the Intel manuals to guide
you.
--
Tim Robinson (MVP, Windows SDK)
http://www.themobius.co.uk/
w m r
Mike
Teemu Voipio
> Hello,
>
> I'm thinking to develop a new operating system which will be quite
> different from the others you can find around now.
That's what most of us are thinking of.
> This system differs from the others in that it can simulate any known
> operating system on Earth by using an "execution driver" which
> interprets the executable code of an application and transforms it in
> machine code to be executed natively.
Sounds like Windows Subsystems.. except you can't really simulate "every
known operating system on Earth" unless you also emulate the hardware they
were designed to run on.
Some of the operating systems don't even make any sense on a PC.
Anyway, this concept is not a new one. Most full operating systems CAN
indeed emulate other operating systems. The question is mostly the work
needed to do this, and the resulting performance.
> In this way you could even load Windows DLLs, keeping the whole system
> active and have - say - Linux and Windows running at the same time.
Wine can load Windows DLLs and executables on top of Linux. There is (or
at least used to be) LINE, which could run Linux binaries on top of
Windows. Some BSD's run Linux binaries as well (AFAIK). The problem is,
there are operating systems that have so different basic concepts, that
while you CAN emulate them, you might not gain any advantages to a regular
emulator.
> I have no previous experience in operating systems development, thus I'd
> need LOTS of practical help to get started in first instance.
You should start small, then expand as you become more confident. You
should also start with smaller goals. If you really want to support
binaries from several other operating systems, you should study their
architecture and try to find common concepts.
> This operating system would also have several new IPC technologies, like
> what I call "bubbles", that stay suspended in air with no particular
> owner and can
> then be made exploding, being also able to generate other bubbles.
Forget the real world. Words such as "bubbles" and "exploding" only serve
to confuse you and others. Only when you have designed something, you
might use such words to explain it to others.
> This technique would allow a new sprint in AI computing, I think.
Yeah, right.
It'll probably also bring peace to all the people of the world..
--
- Teemu Voipio <tvo...@cc.hut.fi>
Nick Roberts
> ...
> This operating system would also have several new IPC technologies, like
> what I call "bubbles", that stay suspended in air with no particular owner
> and can then be made exploding, being also able to generate other bubbles.
>
> This technique would allow a new sprint in AI computing, I think.
> ...
Would you be willing to say more about these bubbles, please? What problem
do they solve? How do they work?
--
Nick Roberts
Bryan Bullard
> Hello,
>
> I'm thinking to develop a new operating system which will be
> quite different from the others you can find around now.
>
> This system differs from the others in that it can simulate any
> known operating system on Earth by using an "execution driver" which
> interprets the executable code of an application and transforms it in
> machine code to be executed natively.
>
> In this way you could even load Windows DLLs, keeping the
> whole system active and have - say - Linux and Windows running at the
> same time.
reality aside.
what would be the purpose of this? how would this be beneficial?
are you suggesting that there would be so type of interoperability between
emulations?
-bryan
Aaron Gray
This is not an April 1st joke.
Well take a look a Boch and see if you think you could do better if not give
up now.
Boch is a Virtual Machine which will run x86 code. A JIT from one MC to
another, well thats never been done. Even if you could it would be nightmare
code.
Aaron
Aaron Gray
> do they solve? How do they work?
I am a fraid they follow quantum laws and may burst, seemingly at random :)
Aaron
Daniel Raffler
> such as alt.os.development, web boards and IRC channels are full of
> experienced and not-so-experienced OS developers who may decide to answer
> some of your questions. But nobody is going to tell you exactly what to
> do.
> Nobody will sit down and teach you for free.
Hi Tim,
I didn't know that there are IRC-Channels about OS development. Can you
give me some adresses ?
Thanks,
Daniel Raffler
Spike
>> what I call "bubbles", that stay suspended in air with no particular
>> owner and can
>> then be made exploding, being also able to generate other bubbles.
>
>
>Yeah, right.
>It'll probably also bring peace to all the people of the world..
It actually sounds a little like GP (Genetic Programming) :)
//Spike
Tim Robinson
news:opryz4ay...@news.t-online.de...
> I didn't know that there are IRC-Channels about OS development. Can you
> give me some adresses ?
Sure, irc.freenode.net #osdev is the one where I usually go. On freenode.net
there's also #OSDevelopment, though that's generally less busy.
KVP
> > I'm thinking to develop a new operating system which will be
> > quite different from the others you can find around now.
> > This system differs from the others in that it can simulate any
> > known operating system on Earth by using an "execution driver" which
> > interprets the executable code of an application and transforms it in
> > machine code to be executed natively.
> > In this way you could even load Windows DLLs, keeping the
> > whole system active and have - say - Linux and Windows running at the
> > same time.
> > I have no previous experience in operating systems development, thus I'd
> > need LOTS of practical help to get started in first instance.
> > This operating system would also have several new IPC technologies, like
> > what I call "bubbles", that stay suspended in air with no particular owner
> > and can
> > then be made exploding, being also able to generate other bubbles.
> > This technique would allow a new sprint in AI computing, I think.
> > I would need references to device programming, tech sheets, etc...
> > Any good pointer appreciated.
> This is not an April 1st joke.
> Well take a look a Boch and see if you think you could do better if not give
> up now.
> Boch is a Virtual Machine which will run x86 code. A JIT from one MC to
> another, well thats never been done. Even if you could it would be nightmare
> code.
Actually there is one implementation. The transmeta cpus (now produced by
ibm), contain an x86 to transmeta jit. The core components of that software
were developed by Linus Torwalds (who was working for transmeta that time).
The same cpus could be loaded with jits for ia64, amd64 or ppc.
Viktor
ps:
It is not a nightmare code, even the intel P4 cpus have jits in them.
(the trace cache is exactly what its name suggests, a jit compiler's
code memory)
Stefanus Du Toit
> Boch is a Virtual Machine which will run x86 code. A JIT from one MC
> to another, well thats never been done. Even if you could it would
> be nightmare code.
Go have a look at qemu. http://www.google.com/search?q=qemu&btnI=1
There are some good links at the bottom too. Bochs is not the end-all
be-all of virtual machines.
--
Stefanus Du Toit; http://3.141592.org/; gpg 4bf2e217; #include <iostream>
template<int i,int j=i-1>struct I{I(){if(i%j)I<i,j-1>();else I<i-1>();}};
template<int i>struct I<i,1>{I(){std::cout<<i<<'\n';I<i-1>();}};template<
>struct I<1,0>{I(){}};int main(){I<50>();}/* Use -ftemplate-depth-5000 */
sam kayley
"The DeerBear" <rain...@tin.it> wrote in message
news:bpkspi$1p2toj$1...@ID-170738.news.uni-berlin.de...
> what I call "bubbles", that stay suspended in air with no particular owner
> and can
> then be made exploding, being also able to generate other bubbles.
>
> This technique would allow a new sprint in AI computing, I think.
(possibly included in the database) to recognise items in the database and
create new items as a result?
Carnac
message news:bplek0$1q926c$1...@ID-176968.news.uni-berlin.de...
> This is not an April 1st joke.
>
> Well take a look a Boch and see if you think you could do better if not
give
> up now.
>
> Boch is a Virtual Machine which will run x86 code. A JIT from one MC to
> another, well thats never been done. Even if you could it would be
nightmare
> code.
Bochs apparently doesn't run actual machine code as such. If it does, then
it's doing a pretty poor job of it.
cr88192
"The DeerBear" <rain...@tin.it> wrote in message
news:bpkspi$1p2toj$1...@ID-170738.news.uni-berlin.de...
yes, I get drunk, go on comp.arch, and write a post under the general
assumption that you can make the transitors small enough and use weird
configurations/logic to accomplish data time travel and other "magical"
stuff.
but, otherwise, I state up front that I don't know what I am talking about
and that it probably wont work...
oh yes, people on this group probably remember a few times when I was saying
stupid crap about physics, or ai's, or whatever...
if I stick to what I know, I have something all least.
all is not to claim I am sober right now though...
Stephane Hockenhull
>
> Any good pointer appreciated.
>
- dont give up,
- do it tiny steps by tiny steps,
- dont create a web site for it,
- dont give up,
- dont involve other ppl in until your vision is whole and working,
- try as many of your concepts as you can under other OSes before
starting your kernel,
- dont give up,
- make frequent backups (get a CD burner if you dont have one) and date
them,
- expect to start over a lot because you'll be extremely lucky if you
get things right the first time on your own, if you do, buy a lottery
ticket, its that lucky,
- dont give up,
- keep notes, keep a lot of notes, you'll eventually have to take a
break and will forget quite a few things about your code which is
excellent, that way you'll re-read your code and your concepts with a
fresh mind and discover flaws and possible improvements,
- dont give up,
- talk about it with people, it helps to go from concept to
implementation, but dont say "its gonna do this" but "I wanna make it do
this" and "but I have no idea how yet" or "I'm gonna make it do this by
<algorithm>" (see point about the web site)
- if you're about to start over something, make a backup, you might
realize that you werent so wrong the first time but only made a little
mistake here and there,
- dont expect quick results,
- dont give up.
I dont give many explanations about the why of the advice, I'd rather
let you figure it out and not take those for absolute truth.
as for documentation and tutorials, do a google search for "operating
system development" and synonyms,
get some books/docs on the PC architecture,
knowing about other computer archs wont hurt either,
check out some open source OS source code (but give credit to the
author(s), be nice) for working code (books have plenty of errors, dont
trust JUST ONE of them, and hardware circuits have plenty of bugs and
quirks).
look for "helppc", "pcgpe" (if you can run it on a slow PC, otherwise
you'll still have the TXT), "faqsys".
http://www.sandpile.org/
the thing is, there are many parts of an "OS":
-the kernel
-the drivers
-the abstraction
-the user-space applications and tool programs
you could actually take for example a linux or freebsd kernel and
drivers, and build an abstraction layer on top of it in a way that it
would act as a totally different operating system which would save you
the work of building your own kernel and drivers if all you want is test
some abstraction concepts like those IPC bubbles you mentioned.
which btw sound like a cool idea, and I'd love to hear about it *ONCE*
you've made a proof that it works or doesnt by implementing it with some
test programs with real-world application and documenting it.
(a theory is just a theory until its proven to be right or wrong)
and even if its proven to fail or to have flaws document it, there'll
always be some good things about it.
the way I see it there are 4 possibilities:
- it works: awesome!
- it works but has flaws: cool, it can always be improved or might
simply have a limited usefullness which is ok since there is no absolute
solution.
- it doesnt work at all: good, so now that you/we know it doesnt, you/we
can try something else.
- you never finish it and give up: now that would be sad.
just keep that in mind.
oh, and have fun: its a hobby after all.
--
Stephane Hockenhull
Tim Robinson
news:ymDvb.21585$aT.1...@news-server.bigpond.net.au...
> Bochs apparently doesn't run actual machine code as such. If it does,
then
> it's doing a pretty poor job of it.
Bochs parses x86 machine code completely in software. I've found its
processor emulation to be pretty good, but its hardware emulation to be
pretty bad.
Aaron Gray
> > to another, well thats never been done. Even if you could it would
> > be nightmare code.
>
> Go have a look at qemu. http://www.google.com/search?q=qemu&btnI=1
Interesting, will have to have a look at that.
> There are some good links at the bottom too. Bochs is not the end-all
> be-all of virtual machines.
I just used it as an example if 'The DearBear' can follow that code then he
stands a chance of doing his own machine emulation :)
>
> --
> Stefanus Du Toit; http://3.141592.org/; gpg 4bf2e217;
Strange URL
> #include <iostream>
> template<int i,int j=i-1>struct I{I(){if(i%j)I<i,j-1>();else I<i-1>();}};
> template<int i>struct I<i,1>{I(){std::cout<<i<<'\n';I<i-1>();}};template<
> >struct I<1,0>{I(){}};int main(){I<50>();}/* Use -ftemplate-depth-5000 */
Whats all this ?
Aaron
Teemu Voipio
> The DeerBear wrote:
>>
>> Any good pointer appreciated.
>>
[snip]
> - make frequent backups (get a CD burner if you dont have one) and date
> them,
[snip]
> - if you're about to start over something, make a backup, you might
> realize that you werent so wrong the first time but only made a little
> mistake here and there,
I'd say, install and learn to use RCS or CVS or Subversion or some other
version control, then keep backups of your repository. This way you can
try things and go back any number of steps without needing to find the
right backup, and you can get "diffs" to see what you have changed.
Update your repository every time you change something and get it to
compile and boot/pass basic test. I usually commit changes to repository
even after writing single function, at least if I've managed to test that
they seem to work. That usually means a commit cycle of about 15-60
minutes, sometimes less.
It's also good idea to write a script that does a fresh checkout of last
version in the repository, and builds it. Run this every time you've done
something "bigger". If your working version compiles and the fresh
checkout does not, your repository is not up to date. You should have
everything you need to build a working kernel (maybe even floppy image) in
the repository, including build scripts like Makefiles (not created one
though, since you want to recreate them to test your repository).
Once you have a single command checkout&build it's easy to add a crontab
entry (or something similar) that creates nightly builds if you want them
later.
Third advantage is that if you use a system that support multiple
developers, it's easier to add other people to the system. The repository
is also logical place to keep a log of changes.
When you restart the project, DO NOT REMOVE the old repository. Create
another one instead. This way you can always checkout code from the old
version to check how you did things in some previous iteration.
> you could actually take for example a linux or freebsd kernel and
> drivers, and build an abstraction layer on top of it in a way that it
> would act as a totally different operating system which would save you
> the work of building your own kernel and drivers if all you want is test
> some abstraction concepts like those IPC bubbles you mentioned.
I'd also suggest testing/debugging any non-trivial support functions for
the actual kernel on top of some existing system. If your malloc works
with your text-editor or web-browser, it's more likely to work with your
kernel..
> which btw sound like a cool idea, and I'd love to hear about it *ONCE*
> you've made a proof that it works or doesnt by implementing it with some
> test programs with real-world application and documenting it. (a theory
> is just a theory until its proven to be right or wrong) and even if its
> proven to fail or to have flaws document it, there'll always be some
> good things about it.
..to which I'd add, document your ideas before coding them, and if you
find out they suck, add that to documentation too, so you won't make the
same mistake again. The reasons why (and how) something failed are very
helpful when trying to find another, better design.
Almost nothing here is specific to operating systems development ofcourse,
but the more complex the task, the more important good development
practices become.
Apurva Mehta
[snip]
>
>> #include <iostream>
>> template<int i,int j=i-1>struct I{I(){if(i%j)I<i,j-1>();else
>> I<i-1>();}}; template<int i>struct
>> I<i,1>{I(){std::cout<<i<<'\n';I<i-1>();}};template<
>> >struct I<1,0>{I(){}};int main(){I<50>();}/* Use -ftemplate-depth-5000
>> >*/
>
> Whats all this ?
That is just a fancy C++ program that uses templates to display all prime
number less than 50. The special thing is that all the calculations are
done at compile time because of templates. It is pretty interesting code
if you have a liking for advanced programming techniques.
Regards,
- Apurva
w m r
> version control, then keep backups of your repository. This way you can
> try things and go back any number of steps without needing to find the
> right backup, and you can get "diffs" to see what you have changed.
... or just use VMS
Mike
Teemu Voipio
Actually, I really fail to see why Unix-likes or Windows doesn't provide
versioning filesystem. I think there might be some "research" type
filesystems for various Unix and Unix-like kernels, but generally, the
need for separate repository is one of the more annoying "features".
How does it work on VMS? Any resources like APIs or research papers about
this, or even keywords like what they call it..
KVP
> On Mon, 24 Nov 2003 07:26:40 -0800, w m r wrote:
> >> I'd say, install and learn to use RCS or CVS or Subversion or some
> >> other version control, then keep backups of your repository. This way
> >> you can try things and go back any number of steps without needing to
> >> find the right backup, and you can get "diffs" to see what you have
> >> changed.
> > ... or just use VMS
> Actually, I really fail to see why Unix-likes or Windows doesn't provide
> versioning filesystem. I think there might be some "research" type
> filesystems for various Unix and Unix-like kernels, but generally, the
> need for separate repository is one of the more annoying "features".
The problem is that file deletion and rename can't be tracked with a basic
versioning filesystem. For example both cdroms and ntfs supports multistream
files, where the each stream can be a version of a file. It is not as
optimal as differential storage, where only the newer sectors get stored,
but both can be used for basic versioning. In this case an external file
must be used to store valid files, so deletion can occure with the deletion
of a file from this valid files list. A good solution for ntfs is to use the
hidden streams of directories, and store directory dumps into these. It also
allows the storage of file creation/access/update info on a per version
basis.
Viktor
ps:
There are fs mounting plugins for ntfs, that can display version trees as
directories, and hide the versioning system from the applications that use
them.
The DeerBear
> Would you be willing to say more about these bubbles, please? What
> problem do they solve? How do they work?
The idea is simpe: they should be a sort of "free threads" without a
specific
owner that can accomplish a task and be called from any process running,
being created and destroyed at will with no specific rules.
Each bubble should be able to generate other bubbles independently,
alot like CreateProcess in Windows...
Andrew
The DeerBear
> I just used it as an example if 'The DearBear' can follow that code
> then he stands a chance of doing his own machine emulation :)
I think you missed my whole point... I don't aim to "emulate" system, but to
make all of them native <g>.
Andrew
The DeerBear
<snip>
Plus, they could either be some "data repository" or even code executers...
This kind of thing could be used to make "software agents" that work
throughout the system.
Cheers,
Andrew
Tim Robinson
Sounds like a DPC in Windows NT, or a bottom-half in Linux.
The DeerBear
> Sounds like a DPC in Windows NT, or a bottom-half in Linux.
Would you please elaborate on that?
TIA,
Andrew
xby
You are taking upon yourself a huge project, and I hope that you
have the strength, the stamina, and the perseverance to build it into
successful completion. However, no matter what you do and how
successful you are, it is NOT going to be final! Either you or someone
else will come up with something better. Therefore, before you start,
save yourself the future embarrassment, eliminate the word 'final'
from the name, and choose an appropriate name that reflects the
functionality of the end product.
"The DeerBear" <rain...@tin.it> wrote in message news:<bpkspi$1p2toj$1...@ID-170738.news.uni-berlin.de>...
> Hello,
>
> I'm thinking to develop a new operating system which will be
> quite different from the others you can find around now.
>
> This system differs from the others in that it can simulate any
> known operating system on Earth by using an "execution driver" which
> interprets the executable code of an application and transforms it in
> machine code to be executed natively.
>
> In this way you could even load Windows DLLs, keeping the
> whole system active and have - say - Linux and Windows running at the
> same time.
>
> I have no previous experience in operating systems development, thus I'd
> need LOTS of practical help to get started in first instance.
>
> This operating system would also have several new IPC technologies, like
> what I call "bubbles", that stay suspended in air with no particular owner
> and can
> then be made exploding, being also able to generate other bubbles.
>
> This technique would allow a new sprint in AI computing, I think.
>
> I would need references to device programming, tech sheets, etc...
>
> Any good pointer appreciated.
>
> TIA,
>
> Andrew
The DeerBear
> Andrew,
> else will come up with something better. Therefore, before you start,
> save yourself the future embarrassment, eliminate the word 'final'
> from the name, and choose an appropriate name that reflects the
> functionality of the end product.
Oh yeah, that's a good point <g>.
Never to write under the push of enthusiasm <g>.
Andrew
Tim Robinson
> > Sounds like a DPC in Windows NT, or a bottom-half in Linux.
>
> Would you please elaborate on that?
DPC stands for Deferred Procedure Call, and are normally used to complete
interrupt processing in a driver. They execute in kernel mode, within any
thread; because they can run in any thread, they could also run in any
process, and therefore can only access kernel data (which is mapped into
every process). DPCs run at a priority slightly higher than every user
thread (they run at IRQL=DISPATCH_LEVEL), so they can be preempted by any
interrupt but not by user code.
I'm not as familiar with Linux, but I belive bottom halves are used for the
same purpose (preemptable non-time-critical post-interrupt processing) and
work in a similar way.
Maxim S. Shatskih
> thread (they run at IRQL=DISPATCH_LEVEL), so they can be preempted by any
> interrupt but not by user code.
Also there can be only 1 DPC currently running on a CPU. The others are sitting
in the queue.
Second part of ISRs (which run with PIC unmasked) and the timer callbacks are
how DPCs are fired.
Note that the large pieces of code are executed in a DPC context - i.e. with a
timer callback on stack or with a DpcForIsr on stack. For instance, the whole
TCP receive path up to sockets.
DPCs (and any code called by them) cannot wait on sync objects and cannot touch
anything pageable.
DPCs have no latency guarantees. The DPC queue is fired under the following
conditions:
- the CPU is idle
- if the context switch decision is made (if the thread waits)
- from a timer ISR tick, and not always - the probability is governed by
IdealDpcRate kernel parameter.
So, if you have a thread which consumes 100% of CPU (no idle time) and it is
the only running thread in the machine (no context switches) - then you can
have large DPC latency, and the OS effectively treats the thread as a
more-priority task then a DPC. This is to avoid starving of all processing by
crazy device driver who schedules a DPC after DPC. Nevertheless, the DPCs
cannot be preempted by any thread.
It is theoretically possible to not use DPCs/bottom halves in the OS design at
all, having only threads and ISRs (or even the rudimentary ISR which only
awakens some thread - "the ISR thread). This is better for realtime OSes, and
in this case you will only need a spinlock per IRQ and a global dispatcher
spinlock - all other sync primitives are mutexes.
--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
ma...@storagecraft.com
http://www.storagecraft.com
KVP
Irq handling can be made as simple calling a kernel function. In one my
kernel prototypes, each hardware irq got its own counter. The kerel ran
with interrupts enabled. The irq handler had two modes of operation: it
was always incrementing one of the irq counters with an atomic operation,
so an irq could't get lost because the kernel was busy, and if the irq came
from user mode (the kernel was inactive on entry), then it enters the kernel
(otherwise just returns). This way the timer irq was able to interrupt
a user mode thread, and the kernel was able to process any irqs when it
arrived to the scheduler phase. (actually the sheduler was just unblocking
irq handler threads until their counter reached zero, otherwise the counters
are left growing) The user mode side was a simple recv_irq(irqn); call, and
the irq handler thread just had to loop and wait for irqs.
On the kernel side it is more complex, but makes writing device drivers much
easier. It also has the advantage that the irq priorities are handled by the
scheduler. (based on thread priorities)
The scheduler was accounting a timer tick (quantum) to a thread when it
was interrupted by the system timer irq. Otherwise if the thread got
switched out, it was able to keep the time it used up from the current
tick. This results in a percision of timer_irq_time/timeslice_time.
If a thread got switched in as a result of preemption and the cause was
not a timer irq, then the 1st tick should be dropped, so the kernel /
irq handler code can't eat up the time of a thread. This could be made
better if the kernel could use a higher precision timer, that can be
read without using irqs. (so the actual clock cycles could be accounted)
However, currently there is no timer or counter in most modern x86 pcs
that remain at a fixed rate regardless of the various system frequencies.
My actual solution was to only schedule threads on the end of a task
timeslice or if the active task blocked itself. It gave me worse
granularity, but resulted in much simplier code. (all I needed is
just a counter to count the timer irqs until the end of the current
task and I just added one to this count down counter if a thread
got switched in because another one got blocked, so the next timer
irq was not accounted for that thread)
Viktor
ps:
If someone has a better solution for this task timeslice jitter, I would
be glad if it can be shared with this group.
Maxim S. Shatskih
> so an irq could't get lost because the kernel was busy,
Why counter? Just setting the bit is enough. Set the bit when the IRQ was first
caught, and reset after it was handled.
With PCI, even this is not necessary. The PIC will interrupt the CPU over and
over, till the CPU will do something to the card so it will deassert INTA#.
Counting IRQs... looks really strange.
BTW - task timeslice jitter does not care, since I don't think anybody will
need exact precision in this.
KVP
> > so an irq could't get lost because the kernel was busy,
> Why counter? Just setting the bit is enough. Set the bit when the IRQ was
> first caught, and reset after it was handled.
Because I acknowledge the irq at the pic before it gets hadled by a user
mode irq hander, and the pit timer don't need an ack to fire an interrupt
again, so irqs can be lost. This makes sure that the system timer is kept
in a consistent state, no matter how long the handler thread was busy.
(this is needed above 100% cpu load)
> With PCI, even this is not necessary. The PIC will interrupt the CPU over and
> over, till the CPU will do something to the card so it will deassert INTA#.
> Counting IRQs... looks really strange.
It is a very simple method used on microcontrollers, where the irq handlers
are placed into the main application loop, so several timer irqs can pass
before the system has time to handle them.
> BTW - task timeslice jitter does not care, since I don't think anybody will
> need exact precision in this.
In a realtime system it does matter. And it is also required to be fair to
all threads. (two threads running the same code at the same priority should
get the same cpu time)
Viktor
Maxim S. Shatskih
> mode irq hander, and the pit timer don't need an ack to fire an interrupt
> again, so irqs can be lost. This makes sure that the system timer is kept
> in a consistent state,
This is needed for a timer IRQ only, not for any other IRQs.
> It is a very simple method used on microcontrollers, where the irq handlers
> are placed into the main application loop, so several timer irqs can pass
> before the system has time to handle them.
With PCI (level interrupts):
- the cards pulls INTA# down when any interrupt conditions are pending on it,
and if the interrupts are enabled on it (the card-specific enable, each PCI
card must have this).
- it tri-states INTA# if the following complex condition is wrong
- so, the "open collector" OR for IRQ sharing, the wire is pulled up by a
resistor if no cards have interrupt conditions (or have interrupts disabled).
- when the PIC senses the wire at down, it raises its internal masks and pings
the CPU to execute the ISR
- on ISR return, the CPU sends EOI to the PIC
- if the INTA# wire is still down at this moment - then the PIC immediately
sends another ISR.
- the kernel's ISR must call all driver's ISRs registered for this vector
number
- the driver's ISR immediately returns FALSE if the interrupt is not from this
card
- otherwise, it must either disable interrupts on the card or clear the pending
interrupt condition, and then do the rest of processing (or queue the rest of
processing).
- so, when the kernel is done with calling ISRs, the INTA# wire must go back up
- when the "queue rest of processing" will complete, it must re-enable
interrupts on a card specific way - in case if the ISR have disabled it. This
is how SCSI miniports work in Windows, but there are other approaches.
ISA's edge interrupts are absolutely other, and they cannot be shared.
KVP
This is why I like edge triggered mode better. This way, the main irq
dispatcher thread can receive an irq, and send it out to the driver threads.
When the pci bus is set to level triggered mode, then the pci irqs must be
masked off by the kernel before dispatching them. This way the kernel can see
them as edge triggered ones (because when all drivers completed their checks,
the main irq dispatcher thread blocks on the irq, and the kernel enables its
irq line again)
Viktor
Maxim S. Shatskih
??? can it be set to some other mode, given the open-collector nature of PCI
INTA# lines, and given the fact that the card pulls INTA# down till the
condition will be cleared?
KVP
> ??? can it be set to some other mode, given the open-collector nature of PCI
> INTA# lines, and given the fact that the card pulls INTA# down till the
> condition will be cleared?
The old x86 irq controller has two modes, level triggered and edge triggered.
If you set it to treat pci irqs as edge triggered, then it only throws an
irq when the irq lines become active. This way if no irq is pending, then
the handler thread can be blocked. The handler thread just have to check the
line state before blocking, and if it's still active, then it must dispatch
an irq condition again. When the line finally goes inactive, it can block
again until an irq arrives again. (this saves the kernel from taking an
interrupt gate more than once for a bunch of irqs)
Viktor
ps:
If the controller doesn't support edge triggered mode, then the irq in
question must be masked off in the controller, and reenabled again when
the irq line is inactive. This prevents repeated irqs, and allows the
scheduling of the irq handlers to take place in the scheduler instead
of the system's interrupt logic.
Maxim S. Shatskih
triggered.
> If you set it to treat pci irqs as edge triggered,
Are you sure this is possible?
> the handler thread can be blocked. The handler thread just have to check the
Why having an overhead of a full thread (own stack, for instance) for an ISR?
KVP
> > triggered.
> > If you set it to treat pci irqs as edge triggered,
> Are you sure this is possible?
Yes, the old pic has many modes of operation:
-edge/level triggered
-8085/80x86 cpu support
-single/chained operation
(most of it is not used anymore, but that doesn't mean it isn't there)
Tehere are two ways to do what I described:
-either set the controller to edge triggered
-or mask off the irq on the 1st trigger, and unmask it if the line
becomes inactive again (this can be checked in software)
> > the handler thread can be blocked. The handler thread just have to check
> Why having an overhead of a full thread (own stack, for instance) for an ISR?
Because in a modern microkernel, there are no isrs in the kernel itself, but
they are normal user mode threads. The kernel only unblocks a waiting thread
and let it handle the rest. I also ack the pic so other irqs can happen too.
(this flats out the irq priorities, and new priorities come from the kernel
scheduler)
The rule is to use one thread per irq line. For shared irq lines, the main
thread just dispatches the irq to the device driver threads. For nonshared
lines, the main thread is part of the driver. The driver always calls the
irq support library, and the library decides if the irq is shared or not.
In fact, the irq support library is part of the device enumeration/management
system. (the devices are not rquired to be aware if they share an irq or not,
the only thing that they must know is they can be activated anytime, and they
must check if their device issued an irq)
Viktor
ps:
The most common way for a device driver to handle irqs is:
uint irq_handler_thread(void* params)
{
// set device parameters
while(recv_irq(irqn, &count) == K_ERR_OK)
{
// handle irq (optionally send/recv data)
// set new device parameters
}
// set device shutdown parameters
return(0);
}
To make the handler thread exit, the main thread has to close the irq
assignement, which unblocks the waiting thread.
Maxim S. Shatskih
> thread just dispatches the irq to the device driver threads.
If the ISR thread must exist due to microkernel design - then let each driver
to spin the "ISR thread", which will keep most of the time blocked in the
uKernel waiting for a PIC event.
> In fact, the irq support library is part of the device
enumeration/management
> system.
Agree.