Official Tool Chain for building NightDOS

[Antony]

The fun thing about building any project with multiple people is deciding on the tools to use.

The present options include (in no particular order)

OpenWatcom and related tools (as it can generate 32-bit code) either in FreeDOS or Windows

Visual Studio Community (formerly Visual Studio Express). The OSDev wiki has tips on how to use Visual C++ for OS development (reference http://wiki.osdev.org/Visual_Studio)

CygWin or MingW (uses GCC)

NASM 

JWASM (which supports multiple object file formats, based on Watcom Assembler, but uses MASM syntax)

It goes without saying that GitHub access is needed.

[Mercury Thirteen]

If we go the route of pure assembly, NASM gets my vote. If we elect to go with C, I'm for OpenWatcom, which will likely come in handy regardless for application development since it's cross-platform and can make the PE executables this kernel expects.

If anyone wants to work on the code directly, let me know and I'll gladly add you to the contributors list in GitHub.

So far as the rest of the development environment, I've been using a VirtualBox with two virtual hard drives - one for booting into FreeDOS for normal use and the other for booting the kernel. I select which one launches at startup using the F12 key during VirtualBox's boot sequence. I use FTPSrv in the FreeDOS environment and FileZilla on the Windows side to get files back and forth to and from the VM. Figured I'd throw that out there in case it gives anyone else some ideas on how to set up their system.

[Mercury Thirteen]

Is C directly capable of producing a bootable kernel.sys?

I think we would have to use a bit of assembly to get things started, then shift control over to C. If I'm correct, then what linker would we use to merge the two?

[maarten]

Op donderdag 23 juli 2015 17:24:07 UTC+2 schreef Mercury Thirteen:

Is C directly capable of producing a bootable kernel.sys?

I think we would have to use a bit of assembly to get things started, then shift control over to C. If I'm correct, then what linker would we use to merge the two?

it can not boot but if you put a bootloader by it you can boot it....

i have it linked....

asm:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 bits 32

section .text

align 4

dd 0x1BAD0002

dd 0x00

dd (0x1BAD0002*0x00)

global start

extern kmain

start:

cli

call kmain

hlt

-----------------------------------------------------------------------------

C:

------------------------------------------------------------------------------

kmain()

{

}

-------------------------------------------------------------------

C:  Casm.o

ASM: Kasm.o

^^ files ^^

-------------------------------------------------------------

LINKER:

-------------------------------------------------------------

OUTPUT_FORMAT(elf32-i386)

ENTRY(start)

SECTIONS

{

. = 0x100000;

.text : {*(.text)}

.data : {*(.data)}

.bss : {*(.bss)}

}

-------------------------------------------------------------------

Link.ld

^^ file ^^

for linking it together:

[CYGWIN]:

i586-elf-ld -m elf_elf_i386 -T link.ld -o kernel.sys [file (First assembly file) in this case KASM.o] [file (then C) in this case Casm.o]

----------------------------------------------------------------------------

Does this help?

Maarten

[Mercury Thirteen]

Yes, it does, thanks!

I'll have to see how I can mimic that using the Watcom linker, but this gives me the basic idea of how to mix different language files.

[maarten]

that's good :D

in the "kmain" function you can put everything you want....

[Mercury Thirteen]

What are the chances we can make this exclusively in assembly? How many of us can even code in assembly? I know Maarten made that protected mode demo, so he can obviously work with it. I can pick my way through it fairly well enough and wouldn't mind getting better. How about any others here? What do you think of going pure NASM?

Call me old fashioned, but as I weigh the two sides of the pure assembly vs. assembly and C decision, I'm leaning more towards assembly despite its tedium.

Pros:
-Allows for insanely fast code - no compiler can match hand optimized assembly in speed.
-Super compact binary.
-One single language for development.
-Easier for folks to contribute as there's no compilers to download or large development environments to configure. Just NASM.
-No mucking about with a C compiler in an attempt to convince it why it should go against its better judgement and output a flat 32-bit binary file with no DPMI fixups or virtual memory hooks.
-We get to hone our assembly skills AND earn old-school street cred! :D

Cons:
-Assembly isn't the most friendly language to use. Personally, though, I think it's worth mastering.
-Resulting code is not highly portable - but do we really need it to be? The NightDOS Kernel - like FreeDOS and MS-DOS before it - is specifically for the x86 architecture. Not to mention that kernels, by definition, are closely tied to the underlying hardware to the point that I would think anyone porting one to another platform would be better off writing a specialized version from scratch designed specially for the platform at hand. We don't need to be as ubiquitous as Linux.

I realize straight assembly isn't the most modern way to complete a project, but I'm okay with that. It seems a bit cheap to me to have to use kernel.sys to load another file to finish running the kernel. It's just not as elegant; it gives the impression we're just loading some sort of DOS extender over a basic kernel instead of making a single solid, sleek product styled and tuned from the ground up. Think of the panache of us not only doing the "impossible" job of creating a 32-bit DOS, but doing it in one single file.

[maarten]

I totaly agree with Mercury there.

I like to add a few things, so here we go:

- MS-DOS 1.0 was also written in assembly. i never looked at it in detail but for me it's a great Operating System of his time.

- I don't know much about C, so i can't help with that. the kmain function above, is token from a youtube Tutorial.... The only thing i know is that it can         acces hardware really good.

- In the Tutorial of Kmain i found you had to Type much for printing one msg to the screen ormaking a Key Function. It struck me hard against C...(< that  was google translate, please forgive me if it is not a good sentence.) I dont like C.. If programming then java, assembly or batch for me...(the only programming languages i can for helping this project.  :)   ) 

Maarten

[Mercury Thirteen]

Granted the terminology is different, but there's not much difference between C and Java structurally. Then again, this is coming from a C programmer who never wrote anything in Java! lol

[Mercury Thirteen]

If we end up going with C, then it seems DJGPP will be a better choice. I'm reading this now which seems to have all the info we need to compile OS independent code (e.g. raw binaries) right from DJGPP. As it turns out, DJ Delorie first decided to port GCC to MS-DOS because he wanted to write an OS with it! :D

[maarten]

Op zaterdag 25 juli 2015 18:06:07 UTC+2 schreef Mercury Thirteen:

On Saturday, July 25, 2015 at 7:30:12 AM UTC-4, maarten wrote:

Granted the terminology is different, but there's not much difference between C and Java structurally. Then again, this is coming from a C programmer who never wrote anything in Java! lol

HAHA!! LOL!! :) 

[maarten]

I find java much more different from C.
maybe from the first look it looks the same but it isn't......

[Mercury Thirteen]

I spent most of the evening looking into how to get a flat binary out of DJGPP. I have succeeded:

First, I downloaded the UNZIP32.EXE program from DJ's site along with the following:

djdev203.zip      DJGPP Basic Development Kit   (Pretty sure this isn't neccessary)
bnu225b.zip       Basic assembler, linker       Needed
gcc473b.zip       Basic GCC compiler            Definitely needed
csdpmi7b.zip      CWSDPMI - DPMI server         (Not sure if any GCC uses this or not - don't think so though)

I sent these into my virtual machine with FileZilla and installed them just as described on his site.

I made a batch file called make.bat containing these two lines:

gcc -ffreestanding -c -o kernel.obj kernel.c
ld -Ttext 0x0600 --oformat binary -o kernel.sys kernel.obj

and then wrote a simple do-nothing kernel.c as follows:

void main(void)
{
 repeat:
 goto repeat;
}

I then used MAKE to run the batch file and got the resulting kernel.sys, 1KB in size. Only a few useful bytes of code (three lines of assembly) are in the beginning of the file, the rest is garbage but doesn't get executed anyway. The size anomaly must be a quirk of GCC.

I copied the kernel.sys over top of the FreeDOS one on my second hard drive, restarted the virtual machine and booted from that second hard drive. The usual "..." from the FreeDOS bootloader appeared in the upper left corner, and the system appeared hung exactly as it should be. Using the built-in debugger of VirtualBox I determined the code was loaded in the proper place and running in a loop as expected.

Voila. A flat 32-bit binary using C.

[Mercury Thirteen]

On Sun, Jul 26, 2015 at 8:12 AM, JAYDEN wrote:

Mercury,nicely done.As for the "Java or C" thing,is Java even a good operating system code?To clarify,shouldn't we write our operating system code in something OTHER than Java?Java is a great programming language for Applications and the "bells and whistles" of the OS,but the actual OS itself should be programmed in C or assembly.

No, we wouldn't want to write any OS components directly in Java. The main reason is that Java doesn't compile down to x86 machine code but instead to byte code designed to be executed in a Java Virtual Machine. Once this OS is functional, though,  we will definitely want to make a JVM for it so that users can run their Java apps here.

[Mercury Thirteen]

On Sun, Jul 26, 2015 at 12:24 PM, Antony wrote:

Hey,

Check out this wiki article on building a cross compiler. 

http://wiki.osdev.org/OS_Specific_Toolchain

GCC can probably be built specifically for NightDOS.

-Tony

Nice find.

[Antony]

If you'd prefer to work in the Windows environment, you can use Cygwin or MinGW as they both include GCC.

[Antony]

I've managed to build a C/C++ compiler for OS development using Cygwin in Windows 8.1 (32-bit). 

Why is this beneficial?

1st, there isn't a native or otherwise available GitHub client that is compatible with FreeDOS. I'm sure no one is in the mood to try to write one either. GitHub software exists for Linux, Windows, and Mac.

2nd, I have been doing a lot of reading, and from what I've read, trying to use a hosted compiler to build an OS can be problematic, as some compilers insist on using their runtime libraries. The cross compiler only has a bare bones library.

[Mercury Thirteen]

Sweet. How does it work? You mean we can develop in C directly in FreeDOS and the compiler generates only code which does not rely on an underlying OS?

[Antony]

It's in Cygwin which only runs on Windows. The output can be pure binary or i686 ELF (which doesn't matter for NightDOS). Loading a .EXE file for DOS or PE .EXE is just a matter of reading the program file and executing the code. 

[Mercury Thirteen]

On Friday, August 7, 2015 at 2:19:27 PM UTC-4, Antony wrote:

It's in Cygwin which only runs on Windows.The output can be pure binary or i686 ELF (which doesn't matter for NightDOS). Loading a .EXE file for DOS or PE .EXE is just a matter of reading the program file and executing the code.

So we can write the entire kernel in C now and output a flat binary image? Nice. Is it in the repo?

[Antony]

Yes, we can develop in C. I just finished the compiler. I will have to do some tests to setup the linker and verify the foundation we need is there. I'm following the steps on the osdev wiki to build it.

Unfortunately, it won't work in FreeDOS, primarily because of its 16-bit nature. However, if you'd prefer Linux over Windows I'll research that.

To answer your question about building the kernel, the rest of the OS can be built in C and linked to the assembly code that has been written or will be written.

[Mercury Thirteen]

Did you customize LD as well? If I remember correctly, it couldn't produce flat binaries... or something like that.

[Antony]

This is where that stage 2 loader comes in. Build a parser to handle the PE EXE format or COFF format. Just jumping past the header info is all you need to do. I will look into it though.

[Antony]

Oh, I forgot, the stage 2 can put the processor into protected mode so the kernel is already there and doesn't have to handle the switch.

[Antony]

According to wiki.osdev.org on the page for rolling your own boot loader, it states

• it is possible to perform more initialization once protected mode is enabled and before kernel is loaded. This will, however, require that you mix 16 bits and 32bits code in a single object file, which can quickly become a nightmare too...

This is why I recommended a two stage loader. So the kernel can be all 32-bit. The stage2 loader can do some initialization before going 32-bit, but the kernel should be all 32-bit with no 16-bit code.Kind of an IBMBIO.COM and IBMDOS.COM.

[Antony]

All,

My testing is nearly complete. I will zip up the GCC toolchain and make it available via my Dropbox. I want to limit the amount of extra that goes into the Git repository because on every pull, you're pulling down all those files. 

In theory (which is another topic) there should be a repository just for code and another, if deemed necessary for the documentation. 

[Antony]

So the good and bad about the toolchain

The version I built will only work with Cygwin installed. I just tried it on a plain vanilla Win 8 install. 

You can use this link ---> https://drive.google.com/file/d/0B85K_c7mx3QjUnZuaFRPWlBIcXM/edit?usp=sharing to download a version of GCC that will work natively on Windows. The link is from this page ---> http://wiki.osdev.org/GCC_Cross-Compiler#Windows_Users at the bottom of the page, look for Windows hosts.

I guess I need to rebuild using MinGW. 

I'll get on that now, or at some point.

[Mercury Thirteen]

On Sat, Aug 8, 2015 at 6:14 PM, Antony wrote:

This is where that stage 2 loader comes in. Build a parser to handle the PE EXE format or COFF format. Just jumping past the header info is all you need to do. I will look into it though.

I understand the simplicity of having an all 32-bit kernel, however that's pretty much what we have now. Yes, there's that small bit of 16-bit jumpstart code in the beginning, but it's there, it works and it doesn't hurt anything. Plus, it allows one to logically follow every step of our code - increased clarity of intent is always a good thing. Also, there's the very real possibility of having to jump back into real mode to switch video modes and such, and for that we'll need to mix the two bit depths anyway. Besides, as I've said before, having one kernel which loads another kernel just looks like a hack. I want one file - a simple replacement for the FreeDOS kernel which can be easily swapped out at will. Nothing complex, no multiple files, etc. That may seem arbitrary or unnecessarily obtuse - and maybe it is - but that's how i want it to be.

[maarten]

Many people jump back to real mode also for the keyboard and mouse.... but I'm first trying in protected mode. After that (plan B) I'm going to try with real mode...

I have a real mode keyboard section in one of my files. Also the most what I did was in real mode...

Maarten

[Antony]

Nah, you just work the ports Maarten. In MASM/TASM/NASM parlance it would be in port and out port, accumulator. In GAS parlance (GNU assembler), the assembler mnemonic is outb an inb (since the keyboard is a byte port). You can wrap an interrupt around this if you'd like to provide some similarity to BIOS Int 16h. I'm assuming you have the ICW and PIC stuff coded already.

[Mercury Thirteen]

It should be nothing more than responding to an interrupt (which one exactly escapes me at the moment, but it's in the interrupts file in the documentation in our repo) by reading the keyboard data port (0x60). There may be a little more involved, but none of it should require a switch to real mode. I know the

On Tuesday, August 11, 2015 at 11:03:20 AM UTC-4, Antony wrote:

Nah, you just work the ports Maarten. In MASM/TASM/NASM parlance it would be in port and out port, accumulator. In GAS parlance (GNU assembler), the assembler mnemonic is outb an inb (since the keyboard is a byte port). You can wrap an interrupt around this if you'd like to provide some similarity to BIOS Int 16h. I'm assuming you have the ICW and PIC stuff coded already.

I got the PIC remapped and interrupt handler system in place as of the last push, so the framework is there.

[Mercury Thirteen]

On Tue, Aug 11, 2015 at 7:52 PM, Jayden wrote:

Should we give users a choice of 16 bit or 32 bit at boot time,like a selection menu?

I see no reason to. If they want to run in 16 bits, they can stick with the traditional FreeDOS kernel.