Transition from assembly to C

[Mercury Thirteen]

Due to the sudden possibility of Night becoming a cross-platform kernel, I am revisiting the possibility of making Night primarily C-based. If this is done, the assembly part of the kernel will basically become a program loader for the kernel executable itself, which will be written in C. What executable format should we use? I believe most C kernels use COFF format, but I'm open to feedback on that.

To accomplish this change, we will need to determine what stays in the assembly portion and what is coded in the C portion. A few questions I can think of which will need discussed are:

-Where should we initialize hardware?
-Will interrupts be handled fully in C or would they be better left in assembly?
-Does the memory manager function better in C or assembly?

There is a lot of added complexity in doing this, but it may pay off if we attempt a port to the 68K. Or, perhaps a new non-DOS kernel should be made for the 68K (and leave NightDOS as is) which will run whatever GUIs and tools we make for Night DOS. Maybe a completely custom tailored kernel would work better on the 68K computer since at that point there would be no reason to support any vestiges of DOS at all, paving the way for a less hampered design.

[Maarten]

Well, what most people do is just boot in to assembly part. assembly part goes 'immediately' to the C kernel. Then they do almost all stuff there.
Maybe that's a reason to not do that, and do most in assembly... But yeah... Also most documentation is about C and not Assembly. If you search interrupts then you get something simillar to this:

C
C
C
ASM
C
C
C
C
Oh, lets do some ASM, go learn about it and you will know how to do it!
C
C


Not to complain.. :)
So to summarize:
Most people use C for almost everything so this would mean interrupts, hardware initialization and memory is all done in C.

[Maarten]

 Or, perhaps a new non-DOS kernel should be made for the 68K (and leave NightDOS as is) which will run whatever GUIs and tools we make for Night DOS. Maybe a completely custom tailored kernel would work better on the 68K computer since at that point there would be no reason to support any vestiges of DOS at all, paving the way for a less hampered design.

Hahaha! You know where this reminds of? (for me)

 



***joke

[Antony]

Hi,

You can still use assembly in the OS. However in doing so, you lock that specific portion to the platform you're writing for and that portion has to be re-coded for the other platform. When we talked off the board, I gave an example of a good use for assembly in the OS - memcpy. Copying to and from memory locations would be faster in assembly language and the C code would just need to wrap that. The memcpy routine would of course have the normal stack prep and cleanup required for the C language, however, if you are a skilled assembly language coder, that should be easy to overcome with minimal impact.

COFF is supported for m68k but not x86 (unless you build NightDOS with DJGPP). ELF is supported for both and if GRUB becomes a possibility, ELF easily hooks into GRUB.

I guess the next question is, since we are building cross platform, do we still build in DOS, or do we build in Linux/Mac OS/Windows? In any case, this project needs a cross compiler (or 2).

-T

[Antony]

One more thing.

If NightDOS is built properly, the DOS support would be an add-in module that you can just not include. If you look at the diagram I sent, DOS applications make calls to the DOS VM, which in turn calls the NightDOS kernel. If the DOS VM is removed, there should be no side effects other than not having the ability to run DOS software natively. 

This is the best way to build an OS (modular) because components can be linked in (via an API, or drivers) and removed without a lot of work.

-Tony

[Mercury Thirteen]

On Tuesday, May 24, 2016 at 12:40:18 PM UTC-4, Antony wrote:

Hi,

You can still use assembly in the OS. However in doing so, you lock that specific portion to the platform you're writing for and that portion has to be re-coded for the other platform. When we talked off the board, I gave an example of a good use for assembly in the OS - memcpy. Copying to and from memory locations would be faster in assembly language and the C code would just need to wrap that. The memcpy routine would of course have the normal stack prep and cleanup required for the C language, however, if you are a skilled assembly language coder, that should be easy to overcome with minimal impact.

As much as I wanted to make Night the "old-fashioned" way, I don't think it's feasible. Especially with a team working on this and not just me. It seems like this is the best solution, and would let us reuse most of the code between platforms.

COFF is supported for m68k but not x86 (unless you build NightDOS with DJGPP). ELF is supported for both and if GRUB becomes a possibility, ELF easily hooks into GRUB.

I was reading over the details of these formats last night on OS Dev, and it seems ELF is the most modern option. I say we may as well go with that.

 

I guess the next question is, since we are building cross platform, do we still build in DOS, or do we build in Linux/Mac OS/Windows? In any case, this project needs a cross compiler (or 2).

-T

Developing in Windows is an advantage since I'm fairly certain we all run it on our main computers. I have no idea how to build a cross compiler, so I guess that's my first order of business should we all agree to go this route.

[Mercury Thirteen]

It would be cool to see Bird running on 68K hardware lol

[Mercury Thirteen]

So basically, we're building a kernel which runs mainly 32-bit ELF apps, then shifts to V86 mode for legacy DOS apps. Sounds good to me.

[Antony]

Hi,

A lot of what I started in C is in the testing branch. I haven't compiled any of it, and it maybe mostly incomplete because the original direction wasn't to go that way. It was written with GCC in mind, so I will have to try to see if the C code will compile without too many errors.

-T

[Maarten]

Hi!

Recently I was mailing with Mercury, and it seems we are still going to use Assembly. Mercury said:

I'm actually considering going back to assembly for this. I can't see the EPiC computer being done any time soon (although the processor finally arrived the other day!) and even when it is, I want it to have its own kernel, not share something with the x86 architecture.

[Antony]

Hi,

It's all in how you view it. NightDOS or whatever you call it, has it's own kernel. In order to run MS-DOS applications, a MS-DOS subsystem is created (for x86) to run the DOS applications virtually. For EPiC, the DOS subsystem isn't needed, unless you want to run DOS apps there as well, then you need the DOS subsystem plus a virtualizer to translate instructions on the fly from x86 to the native processor for EPiC.

The issue at hand is scope. We (collectively) are looking at the project from too high a perspective. Let's scale it back.

Let's start with booting a kernel. What will that look like? Should the kernel be one consistent code base that (generally) works on all platforms (minus the platform specific bootstrapping) or should there be an entire kernel built specifically for each platform?

If the OS will consist of multiple kernels for multiple platforms, who will manage the x86 platform? Who will manage the EPiC platform? If a new platform presents itself as a viable alternative (ARM for example), will we have to find a SME (subject matter expert) to manage that kernel?

Once these questions are answered, then the work on the kernel can begin. That's major milestone one. Once the kernel is done and completely boots natively on the target platforms with a CLI then for x86, milestone two begins. That is building a DOS subsystem that can run 16-bit and 32-bit MS-DOS applications simultaneously.

Since HXDOS is open source, as is VirtualBox, and QEMU, and even WINE, there may be portions of those projects that we can borrow from to complete the DOS/Windows milestone.

The only other alternative is to take the existing FreeDOS kernel (which is written partially in C already) and figure out how to make it 32-bit.

-T

[Maarten]

One question I can answer for you already. :)
And for that one I refer to my message to the group (the one message you answered on). It says in about the last 2 lines, that each platform gets it's own kernel.

But for that I have another question:
Are we going to make them together? Or just finish one then finish the other one?

A logic answer would be 'yes' on the last question. And after that an also logical thing would be, Night DOS because that one is first set on the list. A year now. :)

[Antony]

Hi,

We *COULD* finish them together if we were building the OS around C. Building in assembly requires learning assembly nuances for each processor (which is needed for OS design anyway). However, since every aspect of the OS is written in assembly rather than C, each aspect has to be written, debugged, tested on x86 for example, then written, debugged, and tested on the next processor and so forth.

How fast this process moves depends on how knowledgeable the team is in assembly language on the target platforms.

Here's an article on why C is used for OS development

https://www.quora.com/Why-are-operating-systems-written-in-C-Why-arent-other-languages-used

Also, look at this

http://stackoverflow.com/questions/580292/what-languages-are-windows-mac-os-x-and-linux-written-in

-T

[Maarten]

I think we are going to stick with Assembly for now. But I heard Mercury, when he has time, also is going to make a plan of the NightDOS kernel. So probably it would make more sense then.

Sorry for the other posts,
Maarten/Martin

[Antony]

This is a discussion. not an argument. I would leave the project if it turns into an argument.

[Maarten]

Sorry, discussion. I editted my post.

[Maarten]

And I edited it again, sorry though.

[Antony]

Hey,

No need to apologize. I'm not upset. Not even close. Let me be clear.

Ultimately, the path that is chosen for NightDOS is Mercury's. As a programmer with a degree in IT, and having done a LOT of research, I preferred and recommended C only because a lot of the work that would be done for the actual kernel could easily move to another platform and become the foundation for another project with minimal work. Mercury is obviously more familiar with assembly language and prefers to code there. Consequently, I more than likely won't be contributing much in the way of actual code because I quit trying to code in Assembly language back in the early 90s. Being that I'm 40 years old with kids and a wife, my role in this project may be best suited to testing.

My goal in this thread and others relating to this was just to bring an alternative viewpoint. Hopefully my tone conveys as much (since you can't see my face) and no one feels slighted or offended.

Cheers,

-T

[Maarten]

Okay! :)
But me being me I am still going to apologize.
So, I'm sorry. My tone wasn't right and it isn't my day today. The reason below, not really something serious. And a bit stupid.

For me the same: It was (and won't ever be) my intention to make someone feel offended. But in this case you already said (if I got it right) you don't feel offended.

THen the reason to not have my day:
My town is actually a really small town (like my country is small too). But the 'gemeente' (piece of goverment, don't know the english word for it) recently chose to just build/rebuild the marketplace and stores all again. The company which is about water, tries to solve an issue with leaks the majority of he street has. Then some sort of health institution, thought they need to change the building. And, one of my neighbours is also changing how his house looks. So I literally hear every minute, machines, hamers, etc... Now I have a headache and I always get negative if I have a headache. It's that simple! :/

[Mercury Thirteen]

I have decided to proceed with Night in pure assembly. Mainly because - as Tony pointed out - I am indeed way more comfortable coding an OS with assembly. A game or word processor is most definitely well suited to C, but (in my mind) the low level and efficient nature of a kernel lends itself to an assembly design. I've looked into mixing the two languages, studied some code and while it does make sense, I just can't wrap my head around mixing C and assembly in this case. And if this is going to be my OS project - meaning something I will be doing a significant part of the work on - then I am compelled to do it in the manner in which I am most comfortable. Even if "most comfortable" doesn't always equal "most intelligent" lol

Tony: In a way, I hate to decide this way because - as you, our only professional programmer, have also pointed out - using C would be a more professional way to go and in some ways be simpler. Hopefully my amateur methods don't frustrate you too much. And I say that with no snark or condescension whatsoever - I'm glad you're here to provide the project with that professional input! I don't code regularly as part of my job, and therefore may not see the tasks at hand as you do.

So, to answer the question posed, my vision is to use a specific kernel for each platform but keep the API platform independent, which should ease program development. Maarten has started working on the memory manager's initialization code, and the next major contribution I make will be getting all the necessary functions in place in the kernel code - even if they do nothing. That way, we all can see what calls will be needed and what needs done - then all that will remain will be to write them.

Wow, it really has been over a year already! Ugh, I hoped to be so much further by now. Oh well, thus is life.

Maarten: I hear you loud and clear on headaches. As a migraine sufferer for most of my life, I'm all too familiar with the bad mood in which that severe pain puts you. I try to stay away from others during those times. For both their benefit and mine lol

[Antony]

I wouldn't advise writing the kernel as one flat assembly file. If you look at the source code for FreeDOS, you'll see two things:

     1. It is written primarily in C

     2. Each portion of the kernel is written as a module that is linked together to produce KERNEL.SYS

If you recall the directory structure I put in one of the branches, it was something along the lines of

NIGHTDOS

    |

    | 

    | 

    +------- mm (memory manager)

    |

    |

    |

    | 

    +-------- fs (file system)

    |

  (so forth)

You can (and I would advise you to) use this structure and in addition do the following, since you're thinking about other platforms:

NIGHTDOS

     |

     |

     +----- common (for common code)

     |

     +---- x86

     |          |

     |          + mm

     |          |

     |         + fs 

     |

     +---- 68k

to keep everything organized.

Next you will want to look into build utilities (like MAKE) to automatically build the kernel (MAKE is a lot better than managing a bunch of batch files to build a project)

[Maarten]

Yeah we can do that. I would use MAKE because I know that one, and I use(d) it. I shouldn't know what other build utillity you need (except LD or similar for the linker and nasm)