Kernel Build

[Antony]

Team,

I've run into a serious snafu. Given the current direction of the kernel source, I am having some difficulties building the C components. There would have to be some serious revamping and it seems that I am the only interesting in building this around C. I think what I will do (as sort of a fork of the project) is build a kernel that is compatible with GRUB. If it is decided later to utilize GRUB for a boot loader, a lot of the functionality will already be built to implement the kernel.

The current kernel is fine, it will just take a lot of work to get a hybrid 16-bit and 32-bit kernel to compile to a COFF file and then link the COFF file with C code. Most examples tend to use ELF32, which presents its own challenges.

Since the kernel is going to remain all assembler, I think from a programming perspective, each part should be built separately and linked together instead of one monolithic source code file. 

So essentially, you would have a series of assembly files and then a linker step. You would also need an extern directive for functions that are not included in assembly. 

It does make things difficult, BUT the added advantage is that each component of the kernel can be unit tested via code that makes test calls to the routines.

[maarten]

Team,

I've run into a serious snafu.

    snafu? never heard of that... but i understand.... :)

 

Since the kernel is going to remain all assembler, I think from a programming perspective, each part should be built separately and linked together instead of one monolithic source code file. 

So essentially, you would have a series of assembly files and then a linker step. You would also need an extern directive for functions that are not included in assembly. 

It does make things difficult, BUT the added advantage is that each component of the kernel can be unit tested via code that makes test calls to the routines.

  what do you mean with each part? we already have three files. also this would mean that it will be unclear to search add things and compile/debug the things. and it's not very handy, and certainly not if we have more people that want to help. it's maybe handy that you can call certain files instead of one big file, but on the other hand you need to explain everything to new members. and if they are new to github then there will be problems as: not finding the files, deleting one file (can also happen with the bigfile but that is easy to have on a second computer) so the others don't work anymore, which one do i need to have? , if forget to compile one file you need to begin the process of compiling again and last but not least you need to add extra text/code in order to get the right file working on the right moment and if that doesn't work because there is an error? we have a batcfile named bsodBATCH.bat so we can do a bsod but then we we may need to add a timer (like the drivers in windows: if the driver doesn't get the HDD ready in 10 seconds there will be shown a bsod) that's an option, but extra code....

you get the point? :p

[Antony]

Hi,

I won't be hard at all. The keyboard code you write would be in keyboard.asm. If there is a call to a function, you just use the extern keyword to reference the function. It makes it easy because as the kernel gets bigger from a code perspective, you have will have to jump around a lot to find things. A support routine that's causing a problem maybe at line 500, but the code calling it maybe at line 1700...

The only difficulty is the build process.

[Antony]

BTW, you do realize that you can't actually use that bsod batch file in the kernel right?

Do some research on Makefiles. We were going to need one anyways. That will solve the build issues.

While at it, take a peek at the source code for the FreeDOS kernel. It consists of multiple files, not just one. 

Since we aren't dealing with DOS per se, you can take a peek at the MS-DOS source as well. You'll see that it consists of multiple files. IO.SYS and MS-DOS.SYS (or IBMDOS and IBMBIO) are comprised of several source files linked together. 

This is why there was talk of COFF (the Common Object File Format) so that all the assembly could be pushed to a bunch of object files and then linked together. Tools like EXE2BIN or EXEFLAT can convert the COFF to a flat binary if necessary...

So basically what would happen is something along these lines:

nasm -f coff kernel.asm

nasm -f coff keyboard.asm

nasm -f coff timer.asm

nasm -f coff console.asm

nasm -f coff memory.asm

nasm -f coff disk.asm

ld -o kernel kernel.o keyboard.o timer.o console.o memory.o disk.o

A linker script may be needed, to set things like the origin and stack space (for LD). I don't know what will happen if you have access to LINK from Visual Studio Express. That would require a trip to MSDN.

On DOS platforms, it may come out as kernel.exe. EXEFLAT may be able to convert to a flat binary, KERNEL.BIN which could be renamed KERNEL.SYS

Here's a link to a Makefile, not for this project, but a basic tutorial. You should be able to configure it to work. However, given the fact that there is BITS 16 and an ORG statement in the current kernel, NASM will complain if you use the -f coff option, which is why I said a linker script would be needed for LD (since the GCC toolchain is being used)

Cheers

[Antony]

Forgot the link - http://cs.smith.edu/dftwiki/index.php/CSC231_Makefile_for_linking_ASM_and_C_Files

[Maarten]

Op maandag 14 september 2015 19:21:50 UTC+2 schreef Antony:

BTW, you do realize that you can't actually use that bsod batch file in the kernel right?

Yes, I did know that. The bsod batch file was made to give an idea on how we could let it look. In groupMe I already wrote that it was only for the idea. Although this is a year ago (I didn't see the subject answer of you) I know that it was made with that idea. :)
 

Do some research on Makefiles. We were going to need one anyways. That will solve the build issues.

While at it, take a peek at the source code for the FreeDOS kernel. It consists of multiple files, not just one. 

Multiple Makefiles? I searched on it a while ago for my own project but it's a bit of a overkill. We use nasm which is fine for now, I think. If we need one, I think it's easier to write one on your own as I need to research first.
 

Since we aren't dealing with DOS per se, you can take a peek at the MS-DOS source as well. You'll see that it consists of multiple files. IO.SYS and MS-DOS.SYS (or IBMDOS and IBMBIO) are comprised of several source files linked together. 

I did that 2 years ago, that's why I am not allowed to help with the FreeDOS kernel.
 

This is why there was talk of COFF (the Common Object File Format) so that all the assembly could be pushed to a bunch of object files and then linked together. Tools like EXE2BIN or EXEFLAT can convert the COFF to a flat binary if necessary...

So basically what would happen is something along these lines:

nasm -f coff kernel.asm

nasm -f coff keyboard.asm

nasm -f coff timer.asm

nasm -f coff console.asm

nasm -f coff memory.asm

nasm -f coff disk.asm

ld -o kernel kernel.o keyboard.o timer.o console.o memory.o disk.o

That's completly right. This I learned for my own BirdOS when I was 12.. In 2014. :)
 

A linker script may be needed, to set things like the origin and stack space (for LD). I don't know what will happen if you have access to LINK from Visual Studio Express. That would require a trip to MSDN.

How do you mean? I do have 2 versions of Visual Studio.
 

On DOS platforms, it may come out as kernel.exe. EXEFLAT may be able to convert to a flat binary, KERNEL.BIN which could be renamed KERNEL.SYS

There are also project which do this when compiling, you simply put them into visual studio. But they sometimes are a bit bad, actually almost always.

Thanks,
Maarten

[Antony]

Hi,

I've started my own branch locally on my machine to attempt to build a linker compatible version of the kernel. My eyes are crossing going through the code, but I'm sure I will figure it out.

Here's the why:

While include files are good, they aren't intended to introduce entire functionality to a program. The general purpose of an include file is to define common variables that can be shared among modules. For example, I would make a GDT.INC that included templates for the global descriptor tables that I would be using, that way I wouldn't have to specify them multiple times. However, the code to manipulate and utilize the GDT would be a separate module that would build into an object file. I could then link that object file into some test code and test the functionality of my code and once complete, it would be added to the makefile and of course since now I'm including module files, I'd have to make a linker script (but that's another topic)

The benefit of the modular method (where everything builds to an obj and the subsequent obj files link together the make the kernel) is although the kernel itself has to be rebuilt on each iteration, there are certain components that over time won't need to be rebuilt (except in the case of make clean, if it exists) and only smaller portions of the code is rebuilt. 

As an example, I notice the kernel is comprised of a few files:

[Antony]

Arrgh!!! Typing on the phone sucks!!!

Anyway, a few files:

* kernel.asm

* pic.asm

* pci.asm

* gdt.asm

...there are more, but for brevity, I won't list them. It seems pretty obvious that kernel.asm for the most part, is no longer going to change. It contains the main routine which calls out to initialization stuff and moves on to other aspects and if certain conditions aren't met it drops to the debugger or a halt (and catch fire) sequence. Therefore kernel.asm could compile to kernel.obj and for probably never change unless some major kernel initialization function(s) change.

pic.asm contains code for the programmable interrupt controller (I assume, I haven't looked at it yet in detail). A cursory glance of that code indicates that it too is pretty self-sufficient (and is code not a bunch of data constructs only) so I would compile that separately place a few GLOBALS in there, add some EXTERN in kernel.asm and make an object file from that. It took can be unit tested (by linking to some test code and running cases against it).

From my understanding, NASM has a pretty high line count - 2,000,000,000 lines of code. However, a smaller build of linked modules will make troubleshooting the build process a lot easier moving forward.

As I find time, I am going to play with the current build in my Linker path and see if I can modularize what exists so far and then figure out a good linker script to pull the kernel together. It will take some time as I am a father of 5 and married going on 7 years.

If you want an idea of what I am looking at from a build perspective, pull the source to any Linux application or tool and run a make on it. If you really want to see a massive build, try like I did in 2015 to build a custom GCC or build OpenWatcom from source.

-T

[Antony]

Did I mention I hate typing on phones? I don't feel like getting my computer...took should have been too in the sentence. I'm going to bed now.

[Mercury Thirteen]

I like the idea of doing compilation this way, but the one sticking point for me is that it not increase the overall size of the kernel at all. If it can meet that requirement, I'm completely good with it. Keep us posted on how your linker tests go!