[SharpOS Developers] Virtual Meeting: Memory Manager Design, To Do List, and Overall Compartmentalization

[Adam Stevenson]

Howdy Mike and Everyone,

It is late here, 4:30 am, so I am not going to be up for much longer, but I wanted to get some ideas and some direction out on the table. 

First, when I wake up, and after I go to church (and maybe take a short nap) I am starting a document - which I want to lead to a group publication, on the design strategy for building the SharpOS Memory Manager.  I think we all agree is the next big step in this project.  If I am wrong in this, please correct me and we can redirect our efforts elsewhere. 

I would like to place the document and its contents on the website, which is down again.  This problem has to be solved and stay solved.  XFury, what do we need to do / contribute / find / help with / pray for / donate / find rumpelstiltskin and trade our first born for (well maybe not the first born...) to get the server in working order and for it to stay in working order.  I am thinking since redmine keeps crashing, we can install a separate wiki, and have that as our main page, and then link to redmine under a different hosting account to improve stability?  And what is the current configuration of the server?  I am not asking to cause issues, just so that we can help provide input to solve it and get the problem off of our collective to-do list plate. 

Second, speaking of to do list, I want to gather a list of to-do items that everyone thinks as a group we need to do.  I would then like to enumerate them, and get everyones feedback on them all, and collect the feedback into another document/wiki page "July To-Do List", and assign priorities which we can collectively work on as a group.  I don't feel this to do list has to be fixed in any way or become law, but it will allow us all the ability to see what needs to be done from their different points of views and perspectives.

Ok, with that said, I am going to be heading to bed soon (the underlying them of this email... sleep), but I want to get the "virutal" meeting started.  When I wake up and get back from the morning activities I will try to reply and start organizing responses.  Till then, please everyone that reads this, send in your perspective and issues - it is time for this online meeting to take shape. 

Oh one more thing, since I want to go to bed soon (sleep...) , I am going to save a lot of my thoughts for the memory management till later, but I would like for us to start making a list of sources from which we can all reference to aid us in our design of the memory manager(s) (I am thinking we can make simpler ones and build on the implementations as we go, to help keep momentum going in the project - and to keep "market share" - really I am not competitive at all ;-) )  Right now I am using the following references, but I know there is a TON of more material out there.  As we cannot collect it all, I was wondering what you all see as the first set of reference material we all need to be up to speed on?  So far what I am using for reference is the following:

1.  The Minix Book - Operating Systems Design and Implementation (3rd Ed)  by Tanenbaum and Woodhull. 

2.  Memory Management - Algorithms and Implemnation in C/C++ by Bill Blunden

3.  Garbage Collection - Algorithms for Automatic Dynamic Memory Management (really for the next stage, but it helps)

4.  The Intel microprocessors : 8086/8088, 80186/80188, 80286, 80386, 80486, Pentium, Pentium Pro processor, Pentium II, Pentium III, and Pentium 4 : architecture, programming, and interfacing

I also would like to find some good papers on the material - I need to scan the references in Memory Management (#2), and I would like good recommendations for any other really good x86 books and working with its memory protection parts; any one have the links to the formal documents on the stuff? 

I know there a good couple of websites out there on OS design, anyone have the links to  those handy so we can push them there - they are burried somewhere on my other computer, but someone has to have them more handy than I, and links to Memory Management?

Any other internet link / source / paper / book / piece of bedtime reading material that needs to be read by all of us to get memory management off of our plates?

Ok, with that rant said, please all, hit me with your replies and I will see you when my cerebral cortex is working again.  Remember that the forum is open, lets get what needs to be done out currently out into discussion so we can work as a International Code Hacking Commando Team and get it done. 

Adam to bed he said....

On Sun, Jul 20, 2008 at 2:23 AM, grover <sha...@michaelruck.de> wrote:

Hi Adam,

something like that could work, but there is still the problem that you have to locate the object in memory in a heap, which will later on be maintained by the GC. There are some low-level services, which IMHO just can't be wrapped in object instances - as they are essentially singletons. Memory management should be split into five pieces IMHO:

1. A physical memory manager, which gives out 4KB pages of physical address space

2. A process memory manager, which manages virtual memory and acquires physical pages on demand 

3. A pager, which moves virtual pages between disk and physical memory

4. A heap manager, which is responsible for the actual object allocation

5. A GC which returns regions to the heap

At least #1,3 are singletons with respect to system lifetimes. #2,4,5 are per-process singletons.

Maybe we could think about your design of using adapter or strategy objects to wrap the access to these services in your design? This way the adapters could implement the interfaces you need and abstract away the access to the actual service implementations.

Mike

Am 20.07.2008 um 00:34 schrieb Adam Stevenson:

Howdy Mike,

I am thinking what we need to do is figure out what format the objects are going to need to look like in memory when a memory manager / GC are in place.  This way, when the MM and GC come online during the boot up, they can "take over" objects that were previously not managed, and the initial memory manager can keep a list of memory allocated that is not yet been assigned to a memory manager - thus making it easy to identify objects.  Think this approach could work?  If so, we just need to figure out how we want to structure the unmanaged memory / objects, so that their structures can be compatible. 

-Adam

On Sat, Jul 19, 2008 at 5:21 AM, Matthijs ter Woord <matthijs...@gmail.com> wrote:

Ah, you guys are trying to instantiate the memory manager. Hmm..

 

 

From: sharpos-devel...@lists.sourceforge.net [mailto:sharpos-devel...@lists.sourceforge.net] On Behalf Of grover
Sent: zaterdag 19 juli 2008 11:31

To: sharpos-d...@lists.sourceforge.net
Subject: Re: [SharpOS Developers] Memory manager designand overallcompartmentalization

 

Matthijs, you're right. However it still doesn't solve the problem that you want to new a memory manager without having a memory manager to allocate one from ;) SharpOS doesn't have a GC with the current memory management facilities either. I see it more of a structural problem than a technical one and thus I see it more like a design issue.

 

Mike

 

---

Von: sharpos-devel...@lists.sourceforge.net [mailto:sharpos-devel...@lists.sourceforge.net] Im Auftrag von Matthijs ter Woord
Gesendet: Samstag, 19. Juli 2008 11:04
An: sharpos-d...@lists.sourceforge.net
Betreff: Re: [SharpOS Developers] Memory manager designand overallcompartmentalization

An option is maybe this:

 

Make a fake memory manager, which only is able to allocate (ie, it never takes back memory). For most basic operations you don't use too much objects, so you can live without a GC for a while too. This way you can start on Object support, then interface support, while not having issues with GC and memory manager.

 

We do it that way in Cosmos. We don't have a memorymanager capable of freeing memory, neither do we have a GC (we have the code in place to plug one in, but don't have one. Ralf just started on this)….

 

From: sharpos-devel...@lists.sourceforge.net [mailto:sharpos-devel...@lists.sourceforge.net] On Behalf Of grover
Sent: zaterdag 19 juli 2008 10:16
To: sharpos-d...@lists.sourceforge.net
Subject: Re: [SharpOS Developers] Memory manager design and overallcompartmentalization

 

Hi Adam,

 

the AOT does support interfaces. There were some bugs, but they shouldn't stop us there. There's only one problem: For interfaces to work you need object instances and new/GC... Which causes us a kind of chicken & egg problem. At least at the level Zachary is working at, we need to stay with static classes. I understand your goals, but unless you can give me a hint how you want to manage it without the chicken&egg situation I don't have any idea.

 

Mike

 

---

Von: sharpos-devel...@lists.sourceforge.net [mailto:sharpos-devel...@lists.sourceforge.net] Im Auftrag von Adam Stevenson
Gesendet: Freitag, 18. Juli 2008 06:44
An: sharpos-d...@lists.sourceforge.net
Betreff: Re: [SharpOS Developers] Memory manager design and overallcompartmentalization

Howdy Zachary and All,

Yes, I believe that we do need to do a good restructruring.  I have a bunch of ideas I have been hashing out in my sandbox, but I there are parts that I am not sure exactly what is going on (some parts get cryptic, any understand the process of preparing the image fully?).  I first think we need a high level diagram of all the major components, so we can start figuring out where we can make plugable modules.  From my own project work, and my experience with biologcial systems, we need to slowly evolve this and not break it.  (I know duh). 

First does thing support interfaces yet?  Because I don't see much of them in the code base.  If not, I say we work on making a list of abstract classes that we can start keying in as modules.  I know Grover has been working on a lot of this, and we need to get his input on the AOT plugable architecture.  Anyone going to be around Sunday afternoon (CST) - or most anytime Monday (minus 3:00 to 5:00 CST)?  I can sit down online with anyone and start diagraming some of this stuff and work a bit in team mode - identify document, and review.  And then we can send out whatever documents we get done via the list serve, and continue to move on ot the next.  Anyone up for it?

As for what I have been working on is a bunch of class processing stuff, and memory structuring of components.  I want to build an architecture that you could say is "fractal" - thus allowing for the same archtiecture whether a JIT is being ran on top of a JIT (pretty dang inefficient) or is actaul the JIT.  To do this, everything needs to be turned into an interface, to allow for components to shift up and down.  I have started looking into how to structure this namespace and working on a prototype base, but it is not compable yet (on my to do list for next week).  But bottom line, this is still a long way off from where the AOT and SharpOS is now, so this is by now means a fully workable design that SharpOS can be converted too.  Plus there are some differences in concepts too with the idea of runtime and OS that are going to have to be hashed out. 

So personally, we need to look at all of us "newbies' fully understanding what we have, and second we slowly turn this thing modular and make a list of all the modules and what their requirements are, so the code base can be evolved in small parts versus changing one thing causes a bunch of ripple effects throughout the code.  My guess, a lot of this modularity already exists, but we just need to uncover dividing points.  So back to Sunday and Monday for me. Anyone game?  And thoughts on how we want to start breaking up this code base into smaller understandable pieces?  Oh Zachary, I got both those books on Memory Management and Garbage Collection - need to finish them, but if you have concepts to reference, just let me know page numbers.

Adam Stevenson

Graduate Student
Department of Biology
Texas A&M University
College Station, TX 77843, USA

On Wed, Jul 16, 2008 at 10:03 AM, Zachary Gorden <drakeka...@gmail.com> wrote:

Just a clarification, when I say memory manager, I mean from the page tables to the physical memory table to the allocations.  It seems that people keep confusing what I am referring to whenever I say MM.  This is mostly an informational thing to let people know what's going through my head and the fact that I'm not dead and still working/planning.

The memory manager as it stands suffers from a somewhat awkward design that makes it difficult to correct certain items.  Not exactly anyone's fault, as SharpOS needed a MM and that's what it got.  The gist is, trying to fix the issues in here and/or optimizing it won't do much in the long term, which is one reason you guys haven't seen much activity from me.

I've been mostly consulting a few ROS developers on how to bootstrap a page table based MM, the initial loading and stuff, so that's the explanation for the lack of code.  For the time being, I've gotten most of the conceptual things explained and will be writing out some of my own code as well as examining just what kind of support for paging exists right now.  And actually, in the process of trying to look at the current code, another thought occurred to me.  Currently, the ADC is, to put it simply, horribly organized.  It's basically all of the AD code in one spot, without any actual segmentation.  That is definitely going to screw with us down the line, and it is likely to not be limited to the ADC.

Because of the above reasons, I am most definitely going to need to create a branch (once I figure out how), as obviously doing surgery on trunk is going to result in incredible instabilities and breakages.  But as far as the restructuring goes, I think a lot of us would benefit from such changes, including people already working with their own branches.  I'd like to hear from other people on the restructuring idea, as I know I'm not the only one who noticed the issues.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-D...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers

 

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-D...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
SharpOS-Developers mailing list
SharpOS-D...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-D...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers

[grover]

Hi Adam,

 

First of all my contributions to a document list of memory management topics themselves:

 

Since we probably all agree that x86/x64 are the primary targets, we should really study the application notes and manuals from the processor makers:

 

- Paging: 

    - Intel, "TLBs, Paging-Structure Caches, and Their Invalidation", Application Note, April 2007

- Memory Organization

    - Intel, "Intel 64 and IA-32 Architectures Software Developer's Manual Volume 1: Basic Architecture", specifically Chapter 3.3 and following

    - Intel, "Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3A: System Programming Guide, Part 1", Chapters 2.4, 3, 4, 8.4/8.5/8.8, 10

 

And probably the most important:

 

- ISO/IEC 23271, "Common Language Infrastructure (CLI) Partition I: Concepts and Architecture",Second Edition from 2006-10-01, specifically chapter 12.6

 

Next my wishlist for a TODO-List - working this list in order is preferred, but not mandatory. I've most likely missed some things, because this is just my PoV.

 

- We need a roadmap

- We really need to define a system architecture, how we want things to work in the end. I suppose this should be in a Wiki, where we can all edit and track changes.

- We need to define an operation model for x86/x64 specifically - regarding memory model, protection boundaries etc.

- I believe we need to implement really good boot and processor init sequences first

   - Since it is 2008, we should be friendly regarding Virtualization Extensions - we should be able to run as guests under a Virtual Machine Monitor on supporting processors

   - We really need to get Interrupts working, all other approaches just consume time without any benefits

   - Some of this is already done, but needs to be documented - sequence diagrams would be nice for others to jump in later on - ideally hyperlinked to respective source files

- We need to define a programming model for the core operating system

- We really need to obey object oriented principles, such as seperation of concerns, testability etc.

- We need to define the APIs of core OS services so we can use them even if they're just mocks at first

   - Actually for testability I'd prefer having mock implementations of all of these services

- We need to define and support a runtime execution engine

   - Without availability of proper runtime metadata we're always going to beat around the bush and not really get to a point or even use the benefits we could gain by the runtime

     (I'll provide a lot in this regard, but I do need support after I've implemented as much as I need there.)

   - We need to support exceptions so that we can stop writing a C-style operating system in a more advanced language

- Since we're working on our own native code compiler(s) we need a lot more tests there. I don't think we'll every be happy with any compiler if we continue the way we're running now.

   - Ideally we'd try to get 100% code coverage here

- I'd like us to work closely with Ensemble on this. After all we all have limited resources. I think we should focus on a common kernel with Ensemble/SharpOS being distributions of it.

- For the JIT we need some people who work on emitting native x86 code from the intermediate representation. I REALLY NEED SUPPORT HERE GUYS.

 

To get to something, we need to do all of the above. It doesn't make sense to work on anything more advanced or higher up in layers, unless the basement's been built.

 

I'm going to eat my lunch. I'll be in IRC later on, but I'm still busy writing my thesis paper - only about half way through there - and my deadline is aproaching closer.

 

Mike

 

[Adam Stevenson]

Howdy All,

Ok, keeping this thing going, for the last 4 hours, Mike (grover) and I have been hashing out ideas and direction on the project.  We believe we got some sound direction figured out that we want to layout in front of you all, but we are going to first spend the next two days getting the report together, and send it out.  If you want to take a look at what we discussed, it is all on the IRC chat.  Feel free to take a look and give us feedback, else in about two days, we are going to be sending out a direction document asking you all for you feedback on it, so we get this boat out of hte mud. 

That is all for now,

Adam

[William Lahti]

Adam, the website problem on my end is solved. The problem which was
causing 100% redmine downtime is gone. However, redmine has always had
occasions of downtime... I do know how to remedy the problem when it
happens but since I am not always available I would like to delegate
authority to a trusted dev who *is* active. Any volunteers?

[William Lahti]

Another option is to upgrade redmine, which hasn't been done since we
installed it. Any takers there?

On Sun, Jul 20, 2008 at 8:37 PM, William Lahti <xfur...@gmail.com> wrote:
> Adam, the website problem on my end is solved. The problem which was
> causing 100% redmine downtime is gone. However, redmine has always had
> occasions of downtime... I do know how to remedy the problem when it
> happens but since I am not always available I would like to delegate
> authority to a trusted dev who *is* active. Any volunteers?
>

--
fury

long name: William Lahti
handle :: fury
freenode :: xfury
blog :: http://xfurious.blogspot.com/

[grover]

I've just looked into the issue. We need to update ruby to 1.8.6 and
we could go to redmine 0.7.3 from 0.6.1 - sounds like major changes to
me. Additionally we may need to update rails unless its part of the
ruby package.

I'm not the most profound user of these shared hosting accounts - if
we have a backup I could try my luck.

Mike

[Sander van Rossen]

... speaking of which, it seems to be down again -sigh-

On Mon, Jul 21, 2008 at 10:59 AM, grover <sha...@michaelruck.de> wrote:
> I've just looked into the issue. We need to update ruby to 1.8.6 and
> we could go to redmine 0.7.3 from 0.6.1 - sounds like major changes to
> me. Additionally we may need to update rails unless its part of the
> ruby package.
>
> I'm not the most profound user of these shared hosting accounts - if
> we have a backup I could try my luck.
>
> Mike

-------------------------------------------------------------------------

[Sander van Rossen]

it seems to be up again now, nvm.

Anyway, another thing we need to do is get the IRC bot up and running again.
Last time i tried it it crashed, i didn't have time to figure out what
was going on though.
The logs are pretty valuable since it allows everyone to get back into
the game pretty quickly and easily after being away for a while.
Which seems to happen to a lot of contributors fairly often *cough* ;)

[grover]

Mike

 

---

Von: sharpos-devel...@lists.sourceforge.net [mailto:sharpos-devel...@lists.sourceforge.net] Im Auftrag von Adam Stevenson

Gesendet: Sonntag, 20. Juli 2008 12:24
An: sharpos-d...@lists.sourceforge.net
Betreff: [SharpOS Developers] Virtual Meeting: Memory Manager Design,To Do List, and Overall Compartmentalization

[Zachary Gorden]

While I personally do not mind close cooperation with Ensemble or even Cosmos, I'm not sure why we should go the route of having the same kernel.  There may be certain design decisions in various components that we may not agree upon and I would prefer not to be tied down with the need to accept a compromise for both projects.  I'd prefer the freedom for both projects to go whichever path each one wants.  Now granted I expect a lot of similarities, but I also expect some major differences in design philosophy.  Personally, I think maintaining compatible interfaces would be a more fruitful approach, as it allows both projects to do whatever they want in the internal components while still preserving the ability to swap entire components without the need to do massive rewrites.
 
I also accept that there may be completely identical components, worked on by both sides, in and out of the kernel.  I just don't see a reason that such sameness be a priority.

 

Sidenote: I think we should all stop resending the entire chain of messages (or at least I should) since we're hitting the 50KB limit.

[Zachary Gorden]

While I personally do not mind close cooperation with Ensemble or even Cosmos, I'm not sure why we would go the route of having the same kernel.  There may be certain design decisions in various components that we may not agree upon and I would prefer not to be tied down with the need to accept a compromise for both projects.  I'd prefer the freedom for both projects to go whichever path each one wants.  Now granted I expect a lot of similarities, but I also expect some major differences in design philosophy.  Personally, I think maintaining compatible interfaces would be a more fruitful approach, as it allows both projects to do whatever they want in the internal components while still preserving the ability to swap entire components without the need to do massive rewrites.

 

I also accept that there may be completely identical components, worked on by both sides, in and out of the kernel.  I just don't see a reason that such sameness be a priority.

[Sander van Rossen]

Any update on this? Is there any plan currently to upgrade redmine?
having to retry 5-10 times before an issue i'm modifying is actually
modified is getting rather tiresome...

On Mon, Jul 21, 2008 at 10:59 AM, grover <sha...@michaelruck.de> wrote:

> I've just looked into the issue. We need to update ruby to 1.8.6 and
> we could go to redmine 0.7.3 from 0.6.1 - sounds like major changes to
> me. Additionally we may need to update rails unless its part of the
> ruby package.
>
> I'm not the most profound user of these shared hosting accounts - if
> we have a backup I could try my luck.
>
> Mike
>
> Am 21.07.2008 um 02:40 schrieb William Lahti:
>
>> Another option is to upgrade redmine, which hasn't been done since we
>> installed it. Any takers there?

-------------------------------------------------------------------------

[Bruce Markham]

Chriss was the one that originally set up redmine (it was rather involved, I believe.) But as far as I know, Will is the only one with access to the server...

[grover]

I have not received a reply on the backup status. The other issue is that
someone with better Unix skills should do this job. So there's no updates
from me.

> -----Ursprüngliche Nachricht-----

> Von: sharpos-devel...@lists.sourceforge.net
> [mailto:sharpos-devel...@lists.sourceforge.net] Im

> Auftrag von Sander van Rossen
> Gesendet: Dienstag, 29. Juli 2008 15:44
> An: sharpos-d...@lists.sourceforge.net
> Betreff: Re: [SharpOS Developers] Virtual Meeting: Memory

> Manager Design,To Do List, and Overall Compartmentalization
>