[SharpOS Developers] Additional stuff regarding memory
Zachary Gorden
SharpOS Memory Architecture Design
This is a basic outline of the low level memory management system of SharpOS. This does not include the higher level
garbage collection and managed memory schemes, but the basic services that they will depend on.
Virtual Address Space Organization
The virtual address space in SharpOS will follow the traditional 2GB split for kernel and user mode in 32bit mode,
however this is a mere formality, as there does not exist a user mode. It is more for organization purposes so that
the kernel has sufficient reserved space for its own data structures. The only thing prevent applications and other
unauthorized code from accessing the kernel portion will be the managed nature of such code.
Using a separate page table for each appdomain will mean we must change the value in CR3, so we will need to do a
memory context switch. This results in increased overhead and may be avoidable in 64bit mode, but will require
some experimentation in the future. The page table entries describing the kernel range will be shared by all page
tables. Intel processors have support for this so it shouldn't be problematic.
The kernel range will be the traditional upper 2GB of the 4GB virtual space. It will be a series of preallocated
paged and unpaged pools of memory, specifically set aside for usage by the kernel. These preallocated data structures
will need some tweaking in order to ensure there is enough for each. For allocations smaller than a page in size,
there will be a series of stacks of varying sizes. Allocations from them will be done through lockfree atomic
instructions to ensure better throughput. Whether a general purpose pool will be needed or even possible I'll have
to look into, as traditionally you cannot allocate to the heap like you would in user mode.
The lower 2GB reserved for the application is where things get interesting. Because everything the application
does is managed, we might be able to shift almost all responsibility to the GC and company. It requires some thought,
as I don't know what is intended for management of application memory.
Okay, I'm lying. What I really want to know is when people keep saying "GC kicks in when memory is low," what the
blazes do they mean? Are they talking about the GC being aware of physical pages actually running low? Or just this
heap that the GC is supposedly managing, which might as well be a range of virtual addresses that might all be paged
out for all we know? Until that point gets cleared up, I'm stuck as to what I need to provide for the upper level
services.
The tricky part we now have to deal with is the initial mapping of the kernel into virtual address space and putting
together a page table for it. I still don't fully know how I'm going to pull it off, but it will definitely require
some refactoring of the code. I also need to pick a point early enough in the loading where I won't be overwhelmed
with repointing all the pointers.
Physical Memory
This is basically the paging/swapping part. This is actually fairly simple to work out, but does depend on a working
storage stack. However, we can start by putting together the data structures that keep track of physical pages and
make use of the additional values in the structures once we're able to. I honestly haven't given much thought to
which paging algorithm would be best, but there's so much research out there we'll have plenty of help picking one.
Closing Thoughts
First and foremost, I'm hoping that we stick with an object based design. My reason is centered around security, as
applying security when everything is an object is much easier. That might not seem like it has anything to do with
memory, but trust me, it does. But we won't actually deal with it until the basic virtual memory is done.
Sander van Rossen
My responses are meant as positive criticism, and meant to start a
dialog, not to force my opinion onto anyone. I -expect- people to come
up with counter arguments. Hopefully trough this dialog we can come up
with a good design.
That being said:
> Virtual Address Space Organization
>
> The virtual address space in SharpOS will follow the traditional 2GB split for kernel and user mode
> in 32bit mode, however this is a mere formality, as there does not exist a user mode.
> It is more for organization purposes so that the kernel has sufficient reserved space for its own
> data structures. The only thing prevent applications and other unauthorized code from accessing
> the kernel portion will be the managed nature of such code.
Does that still make sense with a micro-kernel? After all, we're not
talking about a monolithic kernel which uses a ton of resources, we're
talking about a lot of smaller interconnected pieces which each use a
relatively small amount of resources...
'kernel' and 'user' mode don't really exist, at least not in the
traditional sense, since we'd use a more detailed rights model to
determine what a process may or may not do.
If you consider the libraries as 'the kernel' then this split would
make more sense.
(We really need better terminology when it comes to this)
Also, we wouldn't want to expose all libraries to all appdomains..
basically only the ones they'll be using, yet we do want to share the
code-memory of libraries between appdomains.
(The memory pages containing the libraries should also be set to
read-only, but i hope that's obvious to everyone here...)
> The kernel range will be the traditional upper 2GB of the 4GB virtual space.
> It will be a series of preallocated paged and unpaged pools of memory,
> specifically set aside for usage by the kernel.
Sounds a bit much for 'the kernel'. Keep in mind all the drivers etc.
will be outside 'the kernel' and inside 'user mode'..
> The lower 2GB reserved for the application is where things get interesting.
> Because everything the application does is managed, we might be able to
> shift almost all responsibility to the GC and company. It requires some
> thought, as I don't know what is intended for management of application memory.
That's sounds rather vague, and we'll need regular heaps for things
like inner-process communication as well.
although arguably, this would be part of 'the kernel'.
Also, we'd want a GC for every process separatly, it would allow much
quicker releasing of the memory
of a process when it shuts down, and would also keep the GC
generations much smaller and easier to manage.
Imagine having to do a system wide garbage collection! You'd have to
freeze the -entire- system and walk trough -all- the objects in -all-
the processes! every time!
> Okay, I'm lying. What I really want to know is when people keep saying "GC
> kicks in when memory is low," what the blazes do they mean?
> Are they talking about the GC being aware of physical pages actually running low?
> Or just this heap that the GC is supposedly managing, which might as well
> be a range of virtual addresses that might all be paged out for all we know?
> Until that point gets cleared up, I'm stuck as to what I need to provide for
> the upper level services.
It's gonna be tricky, especially since we need to mix virtual memory
that's been paged to disk with memory that's used in the garbage
collector..
And we don't want to have to read all the memory back from the page
file to determine what can be collected.
We might be forced to swap whole processes to and from disk instead of
pieces of memory that haven't been used in a while.
Or restrict it to large blocks of memory (but keep some small
book-keeping information in memory to be able to walk trough the
process to see what's still 'used' and what needs to be collected)
As for when to garbage collect.. i suppose there are some thresholds
that when crossed would trigger a collection.. the most obvious ones
are when there's not enough memory available to allocate some memory
for a process, or when the system is (relatively) idle you do a
collection (to prevent a collection occurring when the system is hard
at work), or before (part of) a process is written to the page file
(to prevent memory being written to the page file which isn't actually
used anymore anyway)
We'd need some way to assign and collect pages of memory from
processes as well, and a way to force other processes to do a
collection when memory is full and a process needs more memory (and
hopefully another process would release some memory after a
collection)..
> The tricky part we now have to deal with is the initial mapping of the kernel into
> virtual address space and putting together a page table for it. I still don't fully
> know how I'm going to pull it off, but it will definitely require some refactoring
> of the code. I also need to pick a point early enough in the loading where
> I won't be overwhelmed with repointing all the pointers.
Don't worry to much about the refactoring at this point, we need a
good design first, and only then should we worry about making it
reality..
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-D...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers
grover
Some answers inline below.
> -----Ursprüngliche Nachricht-----
> Von: sharpos-devel...@lists.sourceforge.net
> [mailto:sharpos-devel...@lists.sourceforge.net] Im
> Auftrag von Zachary Gorden
> Gesendet: Dienstag, 5. August 2008 09:39
> An: sharpos-d...@lists.sourceforge.net
> Betreff: [SharpOS Developers] Additional stuff regarding memory
I think we can live with this, however I would not assume that
an AppDomain is a process. This may be, but its not in a traditional
.NET sense. We'd have to do lots of inter-process marshalling in a
lot of common situations if we'd move AppDomains to their own
address space. So here comes the requirement that AppDomains must
be able to share address spaces.
Like I wrote previously there's two parts to this: The actual
allocator (a heap of some sort) and the GC itself. Heaps need
to grow and when that happens a first-gen collection is performed,
so that the crap pile of all temporary objects is freed on the
heap. Then the allocation is tried again - if it fails this time,
the heap must grow and the heap will request new pages. So the GC
does not have much to do with you, its more the heap which needs
to be able to allocate (batches) of pages and free them again.
I see this similar to the VirtualAlloc/VirtualFree functions in
Win32, an interface of that style should be sufficient.
I would think of:
- An alloc
- A free
- A low watermark notification
- An info method to get statistics and page granularity
- Ability to set page protection attributes (read, write, execute,
no-execute)
- Page status: Reserved, Committed, Freed, (Paged, ...)
> The tricky part we now have to deal with is the initial
> mapping of the kernel into virtual address space and putting
> together a page table for it. I still don't fully know how
> I'm going to pull it off, but it will definitely require some
> refactoring of the code. I also need to pick a point early
> enough in the loading where I won't be overwhelmed with
> repointing all the pointers.
Does this have to do with grub? Do we need a different (managed?)
boot loader?
> Physical Memory
>
> This is basically the paging/swapping part. This is actually
> fairly simple to work out, but does depend on a working
> storage stack. However, we can start by putting together the
> data structures that keep track of physical pages and make
> use of the additional values in the structures once we're
> able to. I honestly haven't given much thought to which
> paging algorithm would be best, but there's so much research
> out there we'll have plenty of help picking one.
>
> Closing Thoughts
>
> First and foremost, I'm hoping that we stick with an object
> based design. My reason is centered around security, as
> applying security when everything is an object is much
> easier. That might not seem like it has anything to do with
> memory, but trust me, it does. But we won't actually deal
> with it until the basic virtual memory is done.
I agree with you on this.
Zachary Gorden
A valid point, regarding the reserved space. However, I was thinking that the drivers were also restricted to the upper 2GB with the kernel. Actually, this is the point where you guys tell me how you want to approach this separation of drivers thing. After all, sticking all the drivers into their own address space is not the way to go in my opinion. Minix does this and they suffer a 20% performance penalty in the process. I don't fully understand how Singularity does it, so, yeah. After all, the point of Singularity's design was to avoid that penalty.
I talked to a friend of mine who has a better idea of how GCs work. Apparently GCs do not care whether their heaps are paged out or whether there's a lack of physical pages. It's entirely possible for it to try to sweep heaps that are completely paged out, causing performance issues and etc. Now granted the OS can provide ways of telling whether a page is in memory or not, there are other complications. Anyways, I don't expect some kind of global GC or something. I expect each appdomain to have its own. But since the 2GB range (or whatever it ends up being) for the application's personal usage is separate for each appdomain anyways, then what's the problem with shifting all responsibility to the GC and the managed heap? Again, that's all stuff that needs to be thought about.
Forcing GC across the board really would be an act of desperation, to be honest. But we'll see.
In response to Grover:
Well, originally I was thinking about an all or nothing approach. But to be perfectly honest, appdomains sharing address spaces have almost nothing to do with the work I need to do in the lower levels. I'm just providing the ability to create address spaces for each appdomain. The MM provides the services and people implementing the higher level stuff can worry about keeping things straight. Correct me if there is something I need to take into consideration though.
The stuff you suggested for basic functionality the MM provides through its interface will most likely be present for use by whatever is managing the heap.
In Windows, the bootloader actually handles all of the mapping of the kernel into virtual memory using page tables. It's one reason grub can't natively boot Windows (the other being the registry). Freeloader, the ReactOS bootloader, actually would be capable of doing this too, but we'd need to write the code for it to understand the SharpOS kernel.
grover
You're right. Current GCs tend to trash the disk if they try to
collect a heap that's mostly paged out. I was hoping we can
prevent this kind of behavior by applying some smarts there. Actually
I think the chances are actually good there as pages could be removed
from memory on a LRU scheme - we could try to collect these specifically
to throw them away before paging them out.
Mike
> -----Ursprüngliche Nachricht-----
> Von: sharpos-devel...@lists.sourceforge.net
> [mailto:sharpos-devel...@lists.sourceforge.net] Im
> Auftrag von Zachary Gorden
> Gesendet: Dienstag, 5. August 2008 12:39
> An: sharpos-d...@lists.sourceforge.net
> Betreff: Re: [SharpOS Developers] Additional stuff regarding memory
Sander van Rossen
keeping "book-keeping information" in memory when swapping pages to
disk.
That way you know if blocks of memory (which are currently on disk)
need to be collected or not.
This way you wouldn't even have to touch the memory blocks... if
they're collected, then the page is freed in the page file, if it's
not collected.. nothing happens.
Only problem is that the book-keeping information (which specifies
which object is connected to which, and what block of memory on disk
it uses) costs memory itself.. so it should only be used on reasonably
large blocks.
Trashing to and from disk would *really* decrease the performance of
virtual memory.. and make collections a nightmare performance wise...
Sander van Rossen
<drakeka...@gmail.com> wrote:
> In response to Sander:
>
> A valid point, regarding the reserved space. However, I was thinking that
> the drivers were also restricted to the upper 2GB with the kernel.
> Actually, this is the point where you guys tell me how you want to approach
> this separation of drivers thing. After all, sticking all the drivers into
> their own address space is not the way to go in my opinion. Minix does this
> and they suffer a 20% performance penalty in the process. I don't fully
> understand how Singularity does it, so, yeah. After all, the point of
> Singularity's design was to avoid that penalty.
Right, i was still thinking about a single adress space like solution
like singularity..
> I talked to a friend of mine who has a better idea of how GCs work.
> Apparently GCs do not care whether their heaps are paged out or whether
> there's a lack of physical pages. It's entirely possible for it to try to
> sweep heaps that are completely paged out, causing performance issues and
> etc.
Right, and i was talking about those performance issues, sorry for not
being clear.
> I expect each appdomain to have its own.
> But since the 2GB range (or whatever it ends up being) for the application's
> personal usage is separate for each appdomain anyways, then what's the
> problem with shifting all responsibility to the GC and the managed heap?
define "all responsibility" :)
> Well, originally I was thinking about an all or nothing approach. But to be
> perfectly honest, appdomains sharing address spaces have almost nothing to
> do with the work I need to do in the lower levels. I'm just providing the
> ability to create address spaces for each appdomain. The MM provides the
> services and people implementing the higher level stuff can worry about
> keeping things straight. Correct me if there is something I need to take
> into consideration though.
If multiple appdomains share the same adress space then they would
also share the 2GB limit...
I *suppose* technically it would be possible to start putting those
appdomains into multiple adress-spaces the moment we detect that that
2 GB isn't enough.
Hmm... interestingly, if we could get *that* to work, then we could
then easily extended that to -all- appdomains.
Meaning that as long as all appdomains stay within the 4GB footprint,
we'd only use one address-space...
Each appdomain has it's data already seperated from other appdomains,
so that would definitely help in this scenario..
Also, If the entire 'kernel' is always in memory, then we would also
never have to switch to another adress space when switching back and
from the kernel.
Of course all of this assumes we won't have any kernel rings.