Feedback & Issues

[Maarten Vermeulen]

We get more and more feedback from others. Well, not loads, but I think enough to make a thread for it. This thread is 'designed' for talking about GitHub Issues, emails and any other feedback we get. 

Feedback of the week:

https://github.com/mercury0x000d/NightKernel/issues/2

This is an issue from someone asking for documentation, so... Do we agree? And if so, who is going to make it. And... Where do we put it? (Documentation Hub?)

It would be good to at least document which compiler we use and what build options.

[Mercury Thirteen]

After looking around other GitHubs (GitHubi? GitHubae? Idk...) and getting some feedback from Tony, I'm thinking maybe I was a bit overzealous in the "code only" approach. But I will say that we need to consciously work to make sure it contains only current, accurate, and useful information and not become a catch all for old files as it did before. If our project starts attracting more eyeballs, we don't need outdated information complicating the efforts of those who may be inclined to assist.

...that all being said...

I'm thinking a single html-based documentation file containing basic info (such as what was requested) should suffice. What do you guys think?

[Antony]

You can create a website from the GitHub repository (or create a separate repository that is the website).

That being said, at this point I think we need to clarify that the kernel isn't necessarily ready to replace the FreeDOS kernel at this point. 

[Antony]

Actually, here's the information here on how to document in one project.

https://blog.github.com/2016-08-22-publish-your-project-documentation-with-github-pages/

[Maarten Vermeulen]

On Saturday, July 21, 2018 at 11:49:05 PM UTC+2, Antony wrote:

You can create a website from the GitHub repository (or create a separate repository that is the website).

We are currently using my crappy website for this, which uses github pages. You can take a look at the better-than-the-rest-of-the-website-page of Night Kernel right here. But when Mercury's website is done, the content will probably be moved to there.

 

That being said, at this point I think we need to clarify that the kernel isn't necessarily ready to replace the FreeDOS kernel at this point. 

Good point, we should.

[Maarten Vermeulen]

On Saturday, July 21, 2018 at 3:12:05 AM UTC+2, Mercury Thirteen wrote:

After looking around other GitHubs (GitHubi? GitHubae? Idk...) and getting some feedback from Tony, I'm thinking maybe I was a bit overzealous in the "code only" approach. But I will say that we need to consciously work to make sure it contains only current, accurate, and useful information and not become a catch all for old files as it did before. If our project starts attracting more eyeballs, we don't need outdated information complicating the efforts of those who may be inclined to assist.

Agree.

 

...that all being said...

I'm thinking a single html-based documentation file containing basic info (such as what was requested) should suffice. What do you guys think?

As long as you see my website as 'html-only' then that's okay. :P

Do you think how the current website looks is okay, or do you want something even simpler?

Sorry for my horrible English...

[Mercury Thirteen]

I closed the open issue. We have documentation now :)

[Antony]

I've skimmed the documentation, I see a slight change that needs to be made in the codebase.

I think all the Mem functions that are in the AT driver should be moved into the core OS. That could probably be useful in more than one place and at some point will need to be included in the C runtime library that will invariably need to be created.

I know how much you "love" when I bring up C, but since FreeDOS is mostly written in C (the kernel) at some point it's going to come into play in the development process.

We'll talk offline on this if you'd like.

-T

[Mercury Thirteen]

On Monday, July 30, 2018 at 12:34:57 PM UTC-4, Antony wrote:

I've skimmed the documentation, I see a slight change that needs to be made in the codebase.

I think all the Mem functions that are in the AT driver should be moved into the core OS. That could probably be useful in more than one place and at some point will need to be included in the C runtime library that will invariably need to be created.

I know how much you "love" when I bring up C, but since FreeDOS is mostly written in C (the kernel) at some point it's going to come into play in the development process.

We'll talk offline on this if you'd like.

-T

I'm not sure which Mem functions you mean... I call the kernel memory management functions from the ATA driver, but (unless I'm missing something) it doesn't contain any itself - the memory functions are all contained in memory.asm. And so far as the fact that FreeDOS was written in C shouldn't matter since, as least in the beginning, we can run a copy of the existing FreeDOS kernel binary inside the memory space of each DOS application using the V86 mode of the processor to arbitrate any I/O port and memory accesses the FreeDOS kernel may make.

If there are complications with that approach, then the DOS API would be implemented as an external driver anyway, and that's where C comes in. Speaking of C, though, do you propose we use GCC or Watcom? I know FreeDOS uses Watcom, but perhaps GCC would be better suited to creating a flat binary?

And you actually made me lol with that "love" comment! :)

[Maarten Vermeulen]

Nice! I think we handled the issue proffesionally. xD

Achievement Unlocked +1000xp

First Issue solved!

[Mercury Thirteen]

WOOHOO! Look at us doing professional things! lol

[Antony]

Hi,

Maybe I misread this part

7/6/2018, 0.12 Update - Functional ATA driver

New

- Added function listings to the beginnings of all files

- Added GDT setting and getting routines to facilitate generation of GDT elements on-the-fly; a first step to multitasking

- Added MemSearchString() - It works for now, but it's not elegant and will definitely need optimized later.

- Added MemCompare(), MemSwapWordBytes(), and MemSwapDWordWords()

I guess I assumed the MemCompare routines were in the ATA driver.

[Mercury Thirteen]

Oh! Ok, that makes sense. I meant that "Functional ATA driver" is the main new thing for this release. That's kind of its title, if you will. But the things listed under "New" are their own separate things.

[Antony]

I made an update to the driver skeleton. I think you will like it. Just added some more dispatch points for the driver especially when dealing with hardware.

[Mercury Thirteen]

On Wednesday, August 1, 2018 at 7:02:37 PM UTC-4, Antony wrote:

I made an update to the driver skeleton. I think you will like it. Just added some more dispatch points for the driver especially when dealing with hardware.

Nice! I'll check it out. 

[Antony]

Since we're looking at more options for the codebase, I created a linker script for LD. I modified the kernel code slightly to build with it and it worked. I haven't pushed that script up. It compiled out slightly larger than the nasm -f bin version (I also set the code to align on a 4096 boundary)

The cool thing about it is if we use this linker script, the code can be written in ASM and C. I opted to build for ELF but the linker converts it to binary mode.

Here's the script for review:

OUTPUT_FORMAT("binary")

ENTRY(kernel)

phys = 0x0600;

SECTIONS

{

  .text phys : AT(phys) {

     code = .;

     *(.text)

     *(.rodata)

     . = ALIGN(4096);

   }

   .data : AT(phys + (data - code))

   {

      data = .;

      *(.data)

      . = ALIGN(4096);

   }

   .bss : AT(phys + (bss - code))

   {

     bss = .;

     *(.bss)

     . = ALIGN(4096);

   }

   end = .;

}

This is how I compiled the kernel (i had to remove the map and org lines since the org information will be in the linker script)

nasm -f elf -o kernel.o kernel.asm

This is what the command line would look like to use the linker:

ld -T link.ld -m elf_i386 -o kernel.bin kernel.o

...anything additional would be added AFTER kernel.o

-T

[Mercury Thirteen]

Is it outputting a flat binary or an ELF executable? And by how much does this method bloat the image?

Edit: I see the answer to one of those now, it still results in a binary image.

[Antony]

A difference of 887 bytes. Some of which may have to do with the alignment on 4K boundaries.

This was built on my Mac running VirtualBox on Cinnamon Linux 64-bit 18.3 using gcc 5.4.0 binutils 2.26.

[Maarten Vermeulen]

>
>
> Here's the script for review:
>
>
> OUTPUT_FORMAT("binary")
> ENTRY(kernel)
> phys = 0x0600;
>
>
> SECTIONS
> {
>   .text phys : AT(phys) {
>      code = .;
>      *(.text)
>      *(.rodata)
>      . = ALIGN(4096);
>    }
>    .data : AT(phys + (data - code))
>    {
>       data = .;
>       *(.data)
>       . = ALIGN(4096);
>    }
>    .bss : AT(phys + (bss - code))
>    {
>      bss = .;
>      *(.bss)
>      . = ALIGN(4096);
>    }
>    end = .;
> }
>

Could be me, but when I used LD, it quit with an error saying that 'phys = 0x0600' is bad syntax. (using the newest version of GCC and LD, I did only test it with my own kernel)

[Maarten Vermeulen]

On Sunday, August 19, 2018 at 11:26:42 AM UTC+2, Maarten Vermeulen wrote:
>
> Could be me, but when I used LD, it quit with an error saying that 'phys = 0x0600' is bad syntax. (using the newest version of GCC and LD, I did only test it with my own kernel)

Oh, blblbplp, that is probably because my linker.ld specified it had to be elf-i386 which probably doesn't support that or anything that has the same thing in mind as the 'org' in Assembly.

[Mercury Thirteen]

I got it to link properly and produce a binary image, which ends up being identical to that created by NASM, with the exception of alignment padding. Viewing the resulting image in a hex editor shows that this padding is solely at the end of the binary image, meaning it does not serve to align individual data items within the kernel image.

This brings up an interesting question: does alignment make any noticeable difference on modern processors? I seem to remember reading in some documentation somewhere that the speed hit which was previously incurred on older CPUs from misaligned accesses has been mitigated.

[Antony]

There is probably more work that needs to be done in the actual kernel code. Nothing in the code IIRC is organized in .code, .data, and .bss therefore those sections would be empty.

[Antony]

Also, check out this kernel comparison (for kernel formats)...https://en.wikipedia.org/wiki/Comparison_of_operating_system_kernels, obviously MS/FreeDOS isn't listed but I would surmise that they are the ONLY flat binary kernel, all the rest seem to be ELF or PE...

Just food for thought

-Tony

[Maarten Vermeulen]

It appears to still be up for debate if we should go for ELF or binary? I just want to say that if this is still up for debate, pointing at the majority doesn't say why we should do it. If we are still unsure, we have to come up with our goals and why ELF will benefit us.

FreeDOS can't load Windows PE files either. It just so happens that Windows PE files have a MZ EXE stub attached to them. However, FreeDOS wouldn't be loading anything. NightKernel is supposed
to replace the FreeDOS kernel and then run it inside a VM86 instance (at least that's what Mercury said)

 

The FreeDOS bootloader can't load us when we are an ELF binary, is what I meant last time. So, how can we run anything inside a VM86 instance if we can't be executed ourselves. We can edit the current bootloader to solve this, which goes against a drop-in replacement in my opinion.
Or we can write a second file which we then have to call kernel.sys so that it can be loaded by the FreeDOS loader, and that file can then load 'JustGiveItAName.sys' (Night Kernel) or something like that. ELF can't be loaded by a bootloader which loads flat binary's.



If we want to be a drop-in replacement of the FreeDOS kernel, that means (in my opinion) we need to leave the FreeDOS bootloader alone.

[Maarten Vermeulen]

it should have been quoted, sorry about that. (it wasn't)

[Antony]

On Monday, August 20, 2018 at 10:28:34 AM UTC-4, Maarten Vermeulen wrote:

It appears to still be up for debate if we should go for ELF or binary? I just want to say that if this is still up for debate, pointing at the majority doesn't say why we should do it. If we are still unsure, we have to come up with our goals and why ELF will benefit us.

It's not so much of a debate as it is looking at the project and re-assessing what the goals and outcomes are. It's a normal process of the Software Development Lifecycle.  Initially, it sounded like a great idea to code NightDOS (now NightKernel) in 100% assembly as a flat binary to be a drop in replacement for FreeDOS. In the time since, a lot of code has been written, contributed, and tested. 

The main benefit to ELF over binary is that it's relocatable, whereas a flat binary has to load into a specific place in memory, an ELF can be loaded anywhere and adjusted to run from that location. The DOS kernel because it was a binary file had built in relocation code that moved the kernel once loaded from one particular spot in memory either lower or higher. With an ELF kernel, that code wouldn't be needed, the kernel can load anywhere and the calls and data would get adjusted to match up. (Incidentally, that's the same process that is used for MZ .EXE files). 

FreeDOS can't load Windows PE files either. It just so happens that Windows PE files have a MZ EXE stub attached to them. However, FreeDOS wouldn't be loading anything. NightKernel is supposed
to replace the FreeDOS kernel and then run it inside a VM86 instance (at least that's what Mercury said)

 

The FreeDOS bootloader can't load us when we are an ELF binary, is what I meant last time. So, how can we run anything inside a VM86 instance if we can't be executed ourselves. We can edit the current bootloader to solve this, which goes against a drop-in replacement in my opinion.
Or we can write a second file which we then have to call kernel.sys so that it can be loaded by the FreeDOS loader, and that file can then load 'JustGiveItAName.sys' (Night Kernel) or something like that. ELF can't be loaded by a bootloader which loads flat binary's.

There are multiple solutions to this:

Solution 1: Take a page from the co-existing Windows NT and DOS/Windows environment, but instead of a bootloader, use GRUB. GRUB understands ELF so it can boot an ELF kernel. GRUB allows booting back into FreeDOS if so desired.

Solution 2: Take a page from the Windows 9x boot process: boot NightKernel with a similarly named KERNEL.SYS that then loads an ELF (or COFF Windows format) executable that contains the kernel.

If we want to be a drop-in replacement of the FreeDOS kernel, that means (in my opinion) we need to leave the FreeDOS bootloader alone.

That is/was an initial goal however, I think we should reassess the goal and scope of the project. Especially since it's no longer JUST a kernel, but now supporting utilities and applications, the primary one being the Virtual Machine Manager that is going to handle making NightKernel look like multiple isolated DOS machines with 2-16MB RAM, dynamic  switching (depending on need) from XMS to EMS, support for VCPI and DPMI 0.9

[Maarten Vermeulen]

I'm sorry in advance.

On Monday, August 20, 2018 at 9:03:34 PM UTC+2, Antony wrote:

It's not so much of a debate as it is looking at the project and re-assessing what the goals and outcomes are.

Yeah...

That is/was an initial goal however, I think we should reassess the goal and scope of the project.

Anyway...

 

It's a normal process of the Software Development Lifecycle.  Initially, it sounded like a great idea to code NightDOS (now NightKernel) in 100% assembly as a flat binary to be a drop in replacement for FreeDOS. In the time since, a lot of code has been written, contributed, and tested. 

 

The main benefit to ELF over binary is that it's relocatable, whereas a flat binary has to load into a specific place in memory, an ELF can be loaded anywhere and adjusted to run from that location. The DOS kernel because it was a binary file had built in relocation code that moved the kernel once loaded from one particular spot in memory either lower or higher. With an ELF kernel, that code wouldn't be needed, the kernel can load anywhere and the calls and data would get adjusted to match up. (Incidentally, that's the same process that is used for MZ .EXE files). 

I didn't know that about ELF, but now the question sounds logical to me.

So, if I understand correctly... More code, project becomes bigger, we need to be more flexible in memory?

There are multiple solutions to this:

Solution 1: Take a page from the co-existing Windows NT and DOS/Windows environment, but instead of a bootloader, use GRUB. GRUB understands ELF so it can boot an ELF kernel. GRUB allows booting back into FreeDOS if so desired.

Pros: We have a ready to go bootloader without us having to do a whole lot about it, it's an okay bootloader.

Con: We aren't a drop-in anymore.

 

Solution 2: Take a page from the Windows 9x boot process: boot NightKernel with a similarly named KERNEL.SYS that then loads an ELF (or COFF Windows format) executable that contains the kernel.

Pros: We can use ELF and we can still stay a drop-in.

Con: More work.

 

That is/was an initial goal however, I think we should reassess the goal and scope of the project. Especially since it's no longer JUST a kernel, but now supporting utilities and applications, the primary one being the Virtual Machine Manager that is going to handle making NightKernel look like multiple isolated DOS machines with 2-16MB RAM, dynamic  switching (depending on need) from XMS to EMS, support for VCPI and DPMI 0.9

We're already in FreeDOS's roadmap as a drop-in replacement for FreeDOS and FreeDOS tweeted about us being a drop-in replacement for FreeDOS. We market ourselves as a drop-in replacement. Even if we stay a replacement for FreeDOS we would break one of the biggest goals, not being a drop-in and using another bootloader (e.g. GRUB). I think that's important to know for when we are going to reasses the project. Now, on the other side we are still in dev, and we can still change.

You probably already noticed, I disagree with solution 1 (I do agree with number 2), this is because it's one of the biggest goals of the project, to be a drop-in replacement. It was part of the project; going away from that goal would be like starting with a simplistic monitor, but ending up with a big complicated TV only for the 1500 extra pixels. No, it isn't. But, if we are re-assessing the goals and scope of the project, I want to make an entire thread for it and get it as clear and detailed as we can. This would, I think, be necessary to keep everything clear and organized.

[Antony]

Hi,

On Monday, August 20, 2018 at 4:00:13 PM UTC-4, Maarten Vermeulen wrote:

I'm sorry in advance.

For what? 

 

On Monday, August 20, 2018 at 9:03:34 PM UTC+2, Antony wrote:

It's not so much of a debate as it is looking at the project and re-assessing what the goals and outcomes are.

Yeah...

That is/was an initial goal however, I think we should reassess the goal and scope of the project.

Anyway...

 

It's a normal process of the Software Development Lifecycle.  Initially, it sounded like a great idea to code NightDOS (now NightKernel) in 100% assembly as a flat binary to be a drop in replacement for FreeDOS. In the time since, a lot of code has been written, contributed, and tested. 

 

The main benefit to ELF over binary is that it's relocatable, whereas a flat binary has to load into a specific place in memory, an ELF can be loaded anywhere and adjusted to run from that location. The DOS kernel because it was a binary file had built in relocation code that moved the kernel once loaded from one particular spot in memory either lower or higher. With an ELF kernel, that code wouldn't be needed, the kernel can load anywhere and the calls and data would get adjusted to match up. (Incidentally, that's the same process that is used for MZ .EXE files). 

I didn't know that about ELF, but now the question sounds logical to me.

So, if I understand correctly... More code, project becomes bigger, we need to be more flexible in memory?

I don't follow. Elaborate.

There are multiple solutions to this:

Solution 1: Take a page from the co-existing Windows NT and DOS/Windows environment, but instead of a bootloader, use GRUB. GRUB understands ELF so it can boot an ELF kernel. GRUB allows booting back into FreeDOS if so desired.

Pros: We have a ready to go bootloader without us having to do a whole lot about it, it's an okay bootloader.

Con: We aren't a drop-in anymore.

 

I think by the time this project is ready to ship, there will be way more than what could be considered "drop in" just by looking at the hardware platform changes. Intel plans to drop the legacy BIOS by 2020. What does that look like for NightKernel? How will FreeDOS work?

 

Solution 2: Take a page from the Windows 9x boot process: boot NightKernel with a similarly named KERNEL.SYS that then loads an ELF (or COFF Windows format) executable that contains the kernel.

Pros: We can use ELF and we can still stay a drop-in.

Con: More work.

 

That is/was an initial goal however, I think we should reassess the goal and scope of the project. Especially since it's no longer JUST a kernel, but now supporting utilities and applications, the primary one being the Virtual Machine Manager that is going to handle making NightKernel look like multiple isolated DOS machines with 2-16MB RAM, dynamic  switching (depending on need) from XMS to EMS, support for VCPI and DPMI 0.9

We're already in FreeDOS's roadmap as a drop-in replacement for FreeDOS and FreeDOS tweeted about us being a drop-in replacement for FreeDOS. We market ourselves as a drop-in replacement. Even if we stay a replacement for FreeDOS we would break one of the biggest goals, not being a drop-in and using another bootloader (e.g. GRUB). I think that's important to know for when we are going to reasses the project. Now, on the other side we are still in dev, and we can still change. 

You probably already noticed, I disagree with solution 1 (I do agree with number 2), this is because it's one of the biggest goals of the project, to be a drop-in replacement. It was part of the project; going away from that goal would be like starting with a simplistic monitor, but ending up with a big complicated TV only for the 1500 extra pixels. No, it isn't. But, if we are re-assessing the goals and scope of the project, I want to make an entire thread for it and get it as clear and detailed as we can. This would, I think, be necessary to keep everything clear and organized.

It doesn't have to be an entire thread, but it does have to be a development consideration. There is a lot to consider that wasn't probably thought about 2-3 years ago when this project was conceived and started. 

This project is still evolving. FreeDOS's goal was to function 100% like MS-DOS. In that, it succeeded. The reason this project started is because the project owner for FreeDOS felt first and foremost that FreeDOS had to maintain it's 16-bit heritage to run classic DOS applications. There were some that were on the thread discussing improvements that would bring FreeDOS to 32-bit, but maintaining legacy prevented those changes.

The initial concept was to write a 32-bit kernel and replace 16-bit KERNEL.SYS with a 32-bit one. Looking at this kernel, and what has to be developed, I'm seeing a kernel, and eventually, all of the device handling will de done with separate device driver files that will be loaded based on the configuration. A very sophisticated SYSCALL feature to call into the kernel for various system calls and a virtual machine manager that will run for each DOS program that a user will want to run that will emulate as much as possible any environment from DOS 3.3x, 5.x, and 6.x depending on the needs of the program, which may require another Windows 3.1 carry over - some type of Program Information File (.PIF) to establish compatibility. 

This moves the project a long ways past dropping in a kernel and moving along. If you think all this can be done in one file, I'm all for it. 

[Maarten Vermeulen]

On Wednesday, August 22, 2018 at 8:34:54 AM UTC+2, Antony wrote:

For what? 

 

On Monday, August 20, 2018 at 9:03:34 PM UTC+2, Antony wrote:

It's not so much of a debate as it is looking at the project and re-assessing what the goals and outcomes are.

Yeah...

That is/was an initial goal however, I think we should reassess the goal and scope of the project.

For pointing that out in a not so nice way.

 

I don't follow. Elaborate.

You actually answered the question already.

 

I think by the time this project is ready to ship, there will be way more than what could be considered "drop in" just by looking at the hardware platform changes. Intel plans to drop the legacy BIOS by 2020. What does that look like for NightKernel? How will FreeDOS work?

Well, FreeDOS will probably stick with the legacy BIOS, that means our (potential) userbase is too.

 

It doesn't have to be an entire thread, but it does have to be a development consideration. There is a lot to consider that wasn't probably thought about 2-3 years ago when this project was conceived and started. 

I think to keep it organized we should start a thread for it. Our official development thread already has like over 380 posts. Simply searching for past info isn't really fun. Yes, you can use the search function, but that shouldn't be our mentality. This thread is for feedback and issues, this is not feedback or an issue but an entire reconsideration of our goals and scope. That may become a large topic very soon. I think it's best to start a new thread for this. Then this topic can stay about Feedback and Issues from e.g. the community. And asking for feedback about a linker script does fit the meaning of this topic, and asking for a reassessment does too. But in my opinion, not the reassessment itself.

 

This project is still evolving. FreeDOS's goal was to function 100% like MS-DOS. In that, it succeeded. The reason this project started is because the project owner for FreeDOS felt first and foremost that FreeDOS had to maintain it's 16-bit heritage to run classic DOS applications. There were some that were on the thread discussing improvements that would bring FreeDOS to 32-bit, but maintaining legacy prevented those changes.

Yes, the general reply of everyone at that day in 2015 was "Why? There already is a FreeDOS-32".

 

The initial concept was to write a 32-bit kernel and replace 16-bit KERNEL.SYS with a 32-bit one. Looking at this kernel, and what has to be developed, I'm seeing a kernel, and eventually, all of the device handling will de done with separate device driver files that will be loaded based on the configuration. A very sophisticated SYSCALL feature to call into the kernel for various system calls and a virtual machine manager that will run for each DOS program that a user will want to run that will emulate as much as possible any environment from DOS 3.3x, 5.x, and 6.x depending on the needs of the program, which may require another Windows 3.1 carry over - some type of Program Information File (.PIF) to establish compatibility. 

This moves the project a long ways past dropping in a kernel and moving along. If you think all this can be done in one file, I'm all for it. 

It may not be one file, but what I see as a drop-in too is just using a setup.bat file which will setup all the files of Night with the FreeDOS base. And that doesn't include a new bootloader. Our kernel.sys can load ItsJustAMoon.sys (the ELF kernel) which can do all the other stuff.

[Damouze]

Op woensdag 22 augustus 2018 21:25:58 UTC+2 schreef Maarten Vermeulen:

Ultimately, whatever binary format you choose for your kernel does not really matter much, because the kernel and the virtual DOS instances are two separate aspects of the same OS. Even the "bootable" part of good-old MS-DOS / PC-DOS consisted of three different files, which were loaded one after the other. IO.SYS/IBMBIO.COM was loaded by the boot sector, and in turn loaded MSDOS.SYS/IBMDOS.COM, while the latter ultimately loaded the command shell, which was COMMAND.COM by default, but could be configured to be just about anything else later on.

IO.SYS, IBMBIO.COM, MSDOS.SYS and IBMDOS.COM were not even proper .COM files, and COMMAND.COM evolved into an .EXE file in the guise of a .COM file.

This only becomes an important matter once your scope reaches beyond the bare metal kernel and you start wandering into the world of (external) device drivers, loadable kernel modules, and userland services. To reiterate (and expand upon) Maarten's idea:

1. kernel.sys is a 16-bit flat binary whose only purpose is to load the kernel, ItsJustAMoon.sys.
2. ItsJustAMoon.sys is a binary in whatever format you choose, configures the system, loads and initializes devices drivers and loads a third program, the so-called Night Service Provider.
3. The Night Service Provider (NIGHTSP.SYS) which prepares the system for general use (let's say the Night equivalent of /sbin/init).
   
4. The Virtual DOS machine - let's call it NIGHTVDM.SYS - contains the userland part of the NightOS and is the basis for the Virtual DOS machine, which in turn can run a complete DOS.
5. DOS itself.

Although this seems a little more complex than a simple drop-in replacement of the FreeDOS kernel, it separates each purpose out into more or less its own executable, which may prove useful later on, especially if you want to add things like administration consoles, remote logins, etc. later on (DOS in RDP ;-)). The names are just examples of course, but I felt like staying in the "Night" theme for them,

 
This is my first post directly on this forum, so if I come off as a little forward, please forgive me.

[Maarten Vermeulen]

Shipping it with a setup.bat, installing Night into the FreeDOS environment is possible too. In that case we can use as much files as we need, while still doing a semi-drop-in kind of setup. Maybe more like a hybrid of drop-in and re-installing an entire operating system.

On Thursday, August 23, 2018 at 1:15:07 PM UTC+2, Damouze wrote:

 
This is my first post directly on this forum, so if I come off as a little forward, please forgive me.

Welcome!

[Antony]

Well...

On Wednesday, August 22, 2018 at 3:25:58 PM UTC-4, Maarten Vermeulen wrote:

On Wednesday, August 22, 2018 at 8:34:54 AM UTC+2, Antony wrote:

For what? 

 

On Monday, August 20, 2018 at 9:03:34 PM UTC+2, Antony wrote:

It's not so much of a debate as it is looking at the project and re-assessing what the goals and outcomes are.

Yeah...

That is/was an initial goal however, I think we should reassess the goal and scope of the project.

For pointing that out in a not so nice way.

 

What wasn't nice? 

I don't follow. Elaborate.

You actually answered the question already.

 

I think by the time this project is ready to ship, there will be way more than what could be considered "drop in" just by looking at the hardware platform changes. Intel plans to drop the legacy BIOS by 2020. What does that look like for NightKernel? How will FreeDOS work?

Well, FreeDOS will probably stick with the legacy BIOS, that means our (potential) userbase is too.

 

So develop with no plan for the future outside of FreeDOS. Got it.

It doesn't have to be an entire thread, but it does have to be a development consideration. There is a lot to consider that wasn't probably thought about 2-3 years ago when this project was conceived and started. 

I think to keep it organized we should start a thread for it. Our official development thread already has like over 380 posts. Simply searching for past info isn't really fun. Yes, you can use the search function, but that shouldn't be our mentality. This thread is for feedback and issues, this is not feedback or an issue but an entire reconsideration of our goals and scope. That may become a large topic very soon. I think it's best to start a new thread for this. Then this topic can stay about Feedback and Issues from e.g. the community. And asking for feedback about a linker script does fit the meaning of this topic, and asking for a reassessment does too. But in my opinion, not the reassessment itself.

 

This project is still evolving. FreeDOS's goal was to function 100% like MS-DOS. In that, it succeeded. The reason this project started is because the project owner for FreeDOS felt first and foremost that FreeDOS had to maintain it's 16-bit heritage to run classic DOS applications. There were some that were on the thread discussing improvements that would bring FreeDOS to 32-bit, but maintaining legacy prevented those changes.

Yes, the general reply of everyone at that day in 2015 was "Why? There already is a FreeDOS-32".

FreeDOS went an entirely different direction and if I remember correctly, development pretty has pretty much halted.

[Maarten Vermeulen]

On Thursday, August 23, 2018 at 6:46:34 PM UTC+2, Antony wrote:

So develop with no plan for the future outside of FreeDOS. Got it.

Calm down, I'm not trying to argue with you.

First of all: The truth is, I actually don't care what decision we take, as long as everyone agrees. I'm not against being a drop-in, I'm not against being something else. But we market ourselves as a drop-in right now. And if we make a decision I want us to be sure about it, because everyone already knows us as a drop-in. So if we are something else we need to say that to the 'community', and because of that I want us to be completely sure about it. Because after this change, we probably can not change it again because then people would likely lose faith in the project. And yes, maybe we started too early on marketing.

Because it's a project that's getting bigger every day (not only in code but also when it comes to people supporting the project) I'm just bringing up arguments so that we can make a decision. With arguments against which everyone can 'fight' with other arguments. When I said "It's what FreeDOS does and thus our current userbase" my goal was to start a discussion where everyone could bring up arguments to go with EFI. That's what I'm trying to do, bring up arguments against it so that we can get better arguments for it. I'm not against changes, I just want to get the good for's that can outrule the no's. If we don't have better for's there's no point in changing. I like change.

Next, I want everyone who's currently contributing to the project in any shape or form to make the decision with us. This so that we can truly make the right decision. And I'll get Mercury on this decision too.

[Antony]

Just so we're clear, I was never upset or angry. I work with a bunch of software engineers and software architects so I have gotten accustomed to hitting "talking" points directly because everything is fast paced and since we're on 2 week sprints for projects (from tech design to deliverable) there's not a lot of time for back and forth on things.

I'm still trying to figure out what I said that wasn't nice.

[Maarten Vermeulen]

On Friday, August 24, 2018 at 6:29:50 AM UTC+2, Antony wrote:

Just so we're clear, I was never upset or angry. I work with a bunch of software engineers and software architects so I have gotten accustomed to hitting "talking" points directly because everything is fast paced and since we're on 2 week sprints for projects (from tech design to deliverable) there's not a lot of time for back and forth on things.

Okay.

 

I'm still trying to figure out what I said that wasn't nice.

If you're refering to this:

For pointing that out in a not so nice way.

I meant that I was saying something that wasn't so nice, at least not in the way I did it.

[Mercury Thirteen]

My current thoughts:

1. We remain a single-file drop-in replacement, because:

     A. It makes it easy for folks to take Night for a spin, and being easy to install will maximize the potential testing base. We very well may need a secondary loader or additional files later on, but that bridge can be crossed upon reaching it.

     B. Fewer "moving parts" minimizes potential points of failure, which matters more at this early stage of the game than it will later. Additional complexities may be added at a later date.

     C. This is how the kernel has already been pitched and lends itself to aiding in Night's adoption. At this point in time, we don't necessarily want to appear as a completely different operating system... why not "ride" a bit on the reputation of FreeDOS, so to speak? Think about it: if a company, for example, is using FreeDOS and has already put together an application stack which will adequately handle all the work they have at hand, would they be as likely to venture into what they may perceive as a completely different OS? On the other hand, however, if it's a matter of merely swapping out one single file to gain full 32-bit goodness, well... that's a much lower perceived risk because the barrier to adoption is insanely low.

2. We continue using the FreeDOS bootloader, because:

     A. It already exists!

     B. It works great for all we really need at this point in time.

     C. The 1.5 smidgens of 16-bit code added to the kernel to handle entering Protected Mode isn't that big a deal.

3. We continue authoring the kernel in NASM alone, because:

     A. The primary goals for Night are full DOS compatibility and maximum efficiency. Making the lowest level code (e.g. the kernel itself) run as efficiently as possible is best achieved by the precision control given by hand tuned Assembly. Steve Jobs is said to have once encouraged the designers of the original Macintosh to make the boot procedure faster with the following reasoning: if an eventual user base of five million people will be booting up their computers at least once a day, shaving ten seconds off the boot time will save fifty million seconds per day. Over the course of a year, that likely works out to saving dozens of cumulative lifetimes worth of users' time. Granted, his math was slightly off (at fifty million users, a drop of ten seconds per user actually only saves 578.7 years worth of time in one year, or 1.59 years of time per day) but the concept remains valid: when certain operations are repeated more than a few times, their total combined time waste quickly adds up. Will we have five million users? Of course not. But there will be millions of instructions executed per second, and how we handle their efficiency is one opportunity we have to shine.

     B. Although most major operating systems are built with Assembly and C, I'm not convinced that's a good fit for our project. Especially when the majority of features which one could reason would be best implemented in C are all to be added later anyway.

     C. It's not like Linux is the only game in town and if we don't go to C then we have no other resources from which to pull. There have been other open source kernels (and even full operating systems!) authored in Assembly, giving us a considerable code base available for our perusal.

     D. Assembly is often thought of as a big, intimidating scramble of spaghetti code because, well... it's often written that way. But it doesn't have to be! I write some darn clean Assembly! lol

Now, that all being said... I have nothing against coding in C, per se; my reluctance simply lies in my hatred of inefficiency. If we are to add C, it should be at no significant penalty. With this in mind, I did a test.

Since Maarten has used both Assembly and C for Bird OS, I decided to use his source to see just how compact a mixed binary could be. I managed to get the linker to generate a somewhat decent map file, but this revealed a bigger problem: GCC (or maybe the linker itself - I'm not certain of the exact culprit at this point) appears to be littering the output code with x86.get_pc_thunk calls at the start of every function, apparently to support position independent code. With the sheer number of tasks which the processor will need to complete every second, adding additional function calls and register loads is simply not acceptable. If we can somehow overcome this and make the resulting binary similar in efficiency to its pure Assembly counterpart, I would consider incorporating C.

And by the way, to our new poster Damouze, welcome! And thanks for your input! :)

[Damouze]

Why, thank you! Always happy to provide some.

Btw, just for the fun of it and to see if I still 'got' it, I tried writing a NightDOS kernel loader (one that would be called kernel.sys and that would load the actual kernel from another file), unaware I was getting ahead of things. It's not finished yet, but I am happy to donate it to the project once it is, just in case you change your mind ;-). It contains assembly routines to search for and read a file from a FAT filesystem and it also moves itself out of the way for the real kernel to be able to load. I can't claim I wrote all of it myself, because it is cobbled together from ancient assembly programs I wrote, of which some of the code is really ancient, from my teenage years. Jeez, I sound old...

That said, I can see the benefit of keeping it all in one file as much as I can see the potential benefit of not doing so. One thing I did realize while writing the ill-fated bootloader is that it might be handy to divide your logical address space into kernel areas and user areas. I have yet to dive into the MM part of the NightDOS kernel, but if you're not already doing so, it might be something to consider. Borrow a page from the x86-64 approach and use the top half of the logical address space for kernel areas and the bottom half for user areas. Your VDM will not notice it (because it is in V86 mode), but the code you ultimately execute to start the VDM will benefit from it greatly, because it does not need to move any stuff out of the way by itself. Of course, with paging enabled, your code can be anywhere in physical memory, so it is just a matter of setting up the page tables appropiately.

[Antony]

Hi.

<snip>

Since Maarten has used both Assembly and C for Bird OS, I decided to use his source to see just how compact a mixed binary could be. I managed to get the linker to generate a somewhat decent map file, but this revealed a bigger problem: GCC (or maybe the linker itself - I'm not certain of the exact culprit at this point) appears to be littering the output code with x86.get_pc_thunk calls at the start of every function, apparently to support position independent code. With the sheer number of tasks which the processor will need to complete every second, adding additional function calls and register loads is simply not acceptable. If we can somehow overcome this and make the resulting binary similar in efficiency to its pure Assembly counterpart, I would consider incorporating C.

The answer to x86.get_pc_thunk calls is here - https://www.technovelty.org/linux/plt-and-got-the-key-to-code-sharing-and-dynamic-libraries.html

[Mercury Thirteen]

On Thursday, August 30, 2018 at 5:01:38 AM UTC-4, Damouze wrote:

Why, thank you! Always happy to provide some.

Btw, just for the fun of it and to see if I still 'got' it, I tried writing a NightDOS kernel loader (one that would be called kernel.sys and that would load the actual kernel from another file), unaware I was getting ahead of things. It's not finished yet, but I am happy to donate it to the project once it is, just in case you change your mind ;-). It contains assembly routines to search for and read a file from a FAT filesystem and it also moves itself out of the way for the real kernel to be able to load. I can't claim I wrote all of it myself, because it is cobbled together from ancient assembly programs I wrote, of which some of the code is really ancient, from my teenage years. Jeez, I sound old...

Interesting! Yes, I think we very well may need such a thing later on.

 

That said, I can see the benefit of keeping it all in one file as much as I can see the potential benefit of not doing so. One thing I did realize while writing the ill-fated bootloader is that it might be handy to divide your logical address space into kernel areas and user areas.

I've kinda done that in my head already... but I'm not sure I've communicated that to the rest of the group here lol  My thoughts for physical RAM are to use the first 2 MiB for the kernel binary itself, and make everything else allocated through the memory manager then remapped through paging. Then again, this may make little sense altogether... I haven't gotten close enough to the MM code to see one way or the other.

 

I have yet to dive into the MM part of the NightDOS kernel, but if you're not already doing so, it might be something to consider.

Me either! lol  Right now there essentially isn't a memory manager, we just simply parse the BIOS memory map, find the largest suitable block, and allocate chunks from there with no paging and no possibility for releasing or remapping. The dummy code in place performs the job required in that it does accept memory requests and it does return usable memory areas... but that's about it. I'd say getting the memory manager finished is the single biggest need in the kernel right now. I'm working on the FAT subsystem in my free time, but even when it's done and we can finally load files from disk, we won't get very far into multitasking without the MM in place.

 

Borrow a page from the x86-64 approach and use the top half of the logical address space for kernel areas and the bottom half for user areas. Your VDM will not notice it (because it is in V86 mode), but the code you ultimately execute to start the VDM will benefit from it greatly, because it does not need to move any stuff out of the way by itself. Of course, with paging enabled, your code can be anywhere in physical memory, so it is just a matter of setting up the page tables appropiately.

Sure, we could do that. Might be better than my plan above lol

[Mercury Thirteen]

Thank you! I'll check it out later today.

[Damouze]

Hi again,

The (proof-of-concept) kernel loader works! You will find the source code, as well as a modified build.sh script for it and the entire kernel attached to this message. In and of itself it is about 955 bytes long, but it is fully capable of loading the actual kernel file from disk. I hereby donate it to the Night Kernel project (and am of course willing to work on it more, should the need arise).

Op donderdag 30 augustus 2018 17:23:20 UTC+2 schreef Mercury Thirteen:

[Maarten Vermeulen]

Thanks! I put it on Github, should we need it in the future.

https://github.com/mercury0x000d/NightKernel/commit/e5d15f94dfad1b9bc8578fe1c42f1f48a1f122d1

[Mercury Thirteen]

Cool, thanks!

[Damouze]

My pleasure!

I am tinkering with the code a bit more and split out some code that I felt should be in different functions. There is now not only a "findfile" function, but also a "stat" function. This latter function basically returns a pointer to the directory entrry of the specified file in the root directory. No support for subdirectories (yet) I'm afraid. The code is now also able to detect the different FAT filesystem according to the BPB, the idea being that the loader should be able to handle basic file I/O requests (findfile, stat, load_from_file, etc) for FAT12, FAT16 and FAT32 filesystems. It currently only handles FAT12 filesystems, which is useful for floppies or small partitions, but not very useful for production system volumes.

I am also implementing routines for preloading modules, so that it is able to load external device drivers required for booting, should the need ever arise. Don't worry, it will still be perfectly capable of loading just the kernel! ;-).

Op vrijdag 21 september 2018 17:22:43 UTC+2 schreef Maarten Vermeulen:

[Maarten Vermeulen]

Someone is requesting a merger of his project and ours... I think it's a bit of a weird request. Anyway, the reason behind his request is simple:

His project is an open source C++ based 64-bit drop-in replacement for FreeDOS called fdpp (FreeDOS plus plus), and he'd like to 'extend the scope past the dosemu2 host platform' which fdpp uses. His project is now stable and he's planning a release. To extend the scope he needs a DOS-oriented ring-0 kernel, which is where we come in. Now for the requirements of the merger:

- As we don't have any FreeDOS related code: we get 'FreeDOS replacement code' immediately after the projects merge

- He gets a new host platform, Night Kernel (where dosemu2 stays a host platform too)

He offered a 32-bit command.com which you can find in his request which is linked below. Maybe I misunderstood the meaning of 'merge our projects'. Both projects are open source, if he wanted he could use our code and we could use his...

I think we shouldn't.

Request: https://github.com/mercury0x000d/NightKernel/issues/3

[Mercury Thirteen]

I think it's a bit out of the scope of our projects to merge, especially at this point. However, as our project is open source (as you point out), anyone is welcome to adapt our code as they see fit. We may be able to use some components of fdpp eventually, but we're not to that point yet.

[Antony]

HI,

I think a lot of things are worded very wrong in this "request", possibly because he doesn't understand the nature of the project. Night exists as it's own kernel separate from FreeDOS (as such it will probably have it's own command shell and utilities) there will be a subsystem that will allow Night to run DOS real and PM applications in a virtualized manner; duplicating DOS command in 32/64 bit environment isn't really a need per se, every OS has commands like copy, move, format, fdisk, fsck (or chkdsk)...'

I think it would be a one sided transaction, especially since Mercury is adamant about assembly language; it's safe to assume the utilities are written in C so they will definitely be of no use (pun intended)

That being said, I think he should just fork the kernel and build around it.

-

-T

On Sunday, October 7, 2018 at 4:53:33 PM UTC-4, Maarten Vermeulen wrote:

[Mercury Thirteen]

Agreed. I think a typical fork would be better than merging projects, as it seems both have different goals.

[Maarten Vermeulen]

He posted new stuff (including replies to Antony's post), he confirmed that he meant us to become one project. Although, we would keep our own source code separate. So... What's the plan? Keep denying and point him to the fork feature?

[Mercury Thirteen]

I'm working on a reply on GitHub as we speak.

[Antony]

I missed the part where there was discussion on using FreeDOS code. I thought Night was just a plain 32-bit kernel with the added bonus of a VM to run DOS apps.

[Mercury Thirteen]

I'm the one to blame for that. The problem is that I evolve the project in my brain without dumping that information to this group for discussion. Then I forget that I didn't update the group, and speak as if everyone knows what I'm talking about. It's confusing, and not a good way to conduct a project. My apologies for that.

To better summarize what the project is (and what it's not) I updated our greeting message. I think it pretty well sums up what we're trying to achieve, but it's quite possible I left something out.

But more directly to your point, you're right. If I recall correctly when we last discussed it, Night was to be its own kernel but have a facility for interchangable API layers to run atop it to allow apps from other platforms to run. And before it's all said and done, we very well may go back to that model and just let the FreeDOS kernel itself be the compatibility layer for traditional DOS apps. It's honestly hard to tell what the limits of the possible are since we aren't even to the point yet where we can run code. I had little free time lately to work on the FAT driver (thankfully I allowed for that kind of the thing in the Milestones timeline, eh? lol) but I am slowly spinning those wheels back up, so to speak. As soon as that's done, we can start on a .COM loader and flesh out the memory manager. Then, if all goes well, we can decide which path to take on the API. 

[Damouze]

Hi,

Great work on rewriting the intro!

Only the links no longer work.

Op woensdag 10 oktober 2018 05:44:53 UTC+2 schreef Mercury Thirteen:

[Antony]

I think it's better to look at the DOS API (which is primarily INT 21h and INT 2Fh, look at what's called and what is sent back and then map them to whatever the underlying OS will become.

You've already written code for accessing the screen, INT 21h  AH=02h which writes to standard output puts 02h in AL, the character to write in DL which can then map to NK_WriteCharOut (byte c) as an example. So given that the entire API is available via RBIL and FreeDOS (but moreso FreeDOS and now GitHub for DOS 2.0) there's no real need to include the FreeDOS source as all that is needed really is just to map the calls into the kernel calls.

-T

[Mercury Thirteen]

Gahh, good catch! Fixed.

[Mercury Thirteen]

True, that would definitely take less memory space than loading a copy of the FreeDOS kernel for every app.

[Antony]

Somehow, I thought that was the ORIGINAL intent, maybe I was sleepy again when you made the decision to include the whole FreeDOS kernel. 

_T

[Mercury Thirteen]

Maybe I was too! lol

[Damouze]

Hi again,

After tinkering with the code some more, the loader has taken on a more modular shape. There are no real new features (yet), but I did make some improvements along the way:

* More modular approach, with assembly time configurations options in a file called 'config.inc'

* Split out the source code into various files, both for legibility and for modularization

* The loader now probes the filesystem type (it can distinguish between DOS 4.0 and DOS 7.0 bootsectors, as well as FAT12, FAT16 and FAT32. It cannot however deal with any filesystem than FAT12 at the moment (but I'm working on it)

The configuration.inc file contains the following defines:

* __DEBUG__: enables the debugging information code (this makes the binary somewhat larger)

* __HAVE_CFG_PARSER: does not do much yet, but this will allow the loader to include code to parse a file called NKLDR.INI and deal with statements in there and parse that information on to the loaded kernel.

* __HAVE_MODULE_INFO__: this will add module information to the loader code. This module information is compatible with DOS device headers, but has been extended to include 32-bit support for kernels that require it.

Once again, I freely donate this code to the project and will continue to work on it from time to time, providing updates and improvements.

Keep up the good work!

Op vrijdag 20 juli 2018 23:26:46 UTC+2 schreef Maarten Vermeulen:

We get more and more feedback from others. Well, not loads, but I think enough to make a thread for it. This thread is 'designed' for talking about GitHub Issues, emails and any other feedback we get. 

Feedback of the week:

https://github.com/mercury0x000d/NightKernel/issues/2

This is an issue from someone asking for documentation, so... Do we agree? And if so, who is going to make it. And... Where do we put it? (Documentation Hub?)

It would be good to at least document which compiler we use and what build options.

[Maarten Vermeulen]

Thanks for the loader once again. I'm looking for other ways that we can more easily share the loader at the moment, this may take a while though. The loader is on github.

On another note, should we ever need the old design documents, I still have them... All of them

[Mercury Thirteen]

Good deal!

[Antony]

In my reading of Windows NT, it appears that the DOS VM consists of ntio.sys and ntdos.sys which are modified IO.SYS and MSDOS.SYS from DOS 5.0

-T

[Tom Ehlert]

On Wednesday, October 10, 2018 at 1:35:38 PM UTC+2, Antony wrote:

I think it's better to look at the DOS API (which is primarily INT 21h and INT 2Fh, look at what's called and what is sent back and then map them to whatever the underlying OS will become.

unfortunately it's a bit more complicated then this - night want's to be a multitasking kernel.

then you have also to handle INT 10 (video) and also directly writing to screen
   *MK_FP(0xb800,0) = 0x4077;

and decide how to handle multiple programs writing to the same (virtual) screen.

accessing LPTx by DOS or BIOS should be managed.
accessing disk by INT 13 should be managed (or forbidden).
accessing COMx, most likely by outport(0x3f8) should be managed.

the MULTItasking part of the OS  makes everything a bit more complicated.

[Antony]

Valid point.

This is a starting point. The alternative in the original docs was to include a complete copy of the FreeDOS kernel to run in the VDM (virtual DOS machine).

I'm also trying to spare the time that would be needed to convert the FreeDOS kernel to assembly.

-T

[Maarten Vermeulen]

What are we going to do with the one open issue over on GitHub?

[Antony]

Has anyone heard from him?

I think that issue only opened a discussion on the scope of the project which I think you and Mercury handled well. 

It's probably a decision best made between you and him.

-T

[Mercury Thirteen]

I just posted a suggestion for closure over there. Let me know what you guys think.