Booting from FAT16

[James Harris]

I've just started looking at booting from FAT16 and it's proving to be a
bit of a challenge. That's because of what I want it to do and because
space is soooo limited! I've set out my idea below. How does it look to
you?

For nomenclature, the sequence to boot from a hard disk can be seen as

MBR --> VBR --> OSloader

i.e. Master Boot Record (first sector of a disk) loads the Volume Boot
Record (first sector of a partition). It, in turn, loads the (larger) OS
loader which is big enough to include code to do a lot more.

As a reminder, the layout of a FAT16 volume is

<reserved><FATs><rootdir><filespace>

AIUI there are two 'traditional' approaches. One is to make the reserved
area large enough for plenty of code. The other (taken by LILO, IIRC) is
to have the os loader in unmovable sectors of the filespace and to
'hardcode' their location into the VBR.

It is /possible/ for there to be plenty of space in the reserved section
of a FAT volume but that cannot be counted on. AIUI it depends on how it
has been formatted. In an already formatted volume the reserved area
might be only one sector, the VBR itself and nothing else. So I don't
believe I can rely on there being any more than 512 bytes in that space.

I'd also like to avoid the LILO approach. It's a pain when a new OS
loader is installed, requiring something to update the hardcoded block
addresses in the VBR.

So, what can be done?

Going back to the boot steps the MBR will be 'standard', i.e. although
the BIOS will have loaded it to 0x7c00 it will move itself out of the
way (often to 0x600) and load the VBR to 0x7c00. It could be any MBR. It
would not need to be specific to a particular OS.

Effectively, the first point I will get control is when my VBR is loaded
to 0x7c00 and started.

I will then need to load the OS loader but than can be difficult in just
512 bytes as I would like it to be a normal file. No special sectors. No
unmovable sectors. My plan, then, is to split the work as follows.

The VBR will look through the root directory until it finds the OS
loader by name. When it finds it it will load just its first sector and
will transfer control to that sector.

The first sector of the OS loader won't have to include code to read the
root directory again meaning it will /hopefully/! have enough program
space to follow the FAT chain to load more of itself.

So loading would be in at least these parts:

1. VBR finds OS loader file and loads 1st sector of it.

2. OS loader's first sector follows the FAT chain to load more of (maybe
all of) OS loader.

3. The OS loader is then large enough to do whatever's necessary.


That's the plan. What do you think of it? Any better ideas? Suggestions
welcome.



--
James Harris

[muta...@gmail.com]

On Monday, December 20, 2021 at 9:43:46 AM UTC+11, James Harris wrote:

> That's the plan. What do you think of it? Any better ideas? Suggestions
> welcome.

I don't know if there is enough space in a boot sector
to do what you want. Also I think it is better to support
CHS which uses up some space.

I have a FAT-12/16 boot sector you can take code from
if you want it. It's public domain so there is no restriction
on what you use it for, no matter what license you choose.

However, there appears to be a bug in it when used on a
disk with the root sector starting on a non-multiple of
track length.

It's available here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/src/pbootsec.asm

BFN. Paul.

[s_dub...@yahoo.com]

Q. Is this fat-LFN?
Q. What is the size of the OS?
Q. Are you sure the VBR cannot be made to load the entire OS? - eliminating the separate OS Loader?

The MBR I use is a vanilla ibm ~32 mb with one volume bootable, with the VBR expected at LBA = 20h.
The VBR is at LB 20h, The hobby OS begins at LB 21h and runs for 63.5k, it is outside the file system. The file system directory follows the OS, the file data follows the directory area. The file system is under development, it isn't needed to boot the OS, it will be needed to save data to files and to load utilities. - non fat.

I don't support fat, nor LFN, LFN is covered by ms patents, so I dropped my work on it. But back when I was working with fat, that VBR code was enough to load a hobby os off the fat filesystem --non-lfn.

The VBR that I'm using is of course loaded at 07C00h, it then relocates itself to below the EBDA and continues on to load the OS to 80:0000h, then far jumps to the OS. The self relocation is simple enough, and gets the vbr boot code out of the way of loading the OS, avoiding the possibility of over-writing itself.

Steve

[muta...@gmail.com]

On Monday, December 20, 2021 at 3:54:44 PM UTC+11, s_dub...@yahoo.com wrote:

> I don't support fat, nor LFN, LFN is covered by ms patents,
> so I dropped my work on it.

Patents only last 20 years, so LFN should be free from
any patents, if you wanted to restart work on it.

Note that I have public domain FAT code, including LFN
support (provided by someone else) available here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/src/fat.c

BFN. Paul.

[Alexei A. Frounze]

On Sunday, December 19, 2021 at 2:43:46 PM UTC-8, James Harris wrote:
...

> I will then need to load the OS loader but than can be difficult in just
> 512 bytes as I would like it to be a normal file. No special sectors. No
> unmovable sectors.

Not too difficult (beware of the overflow bug, though).
https://github.com/alexfru/BootProg

Alex

[James Harris]

On 20/12/2021 00:49, muta...@gmail.com wrote:
> On Monday, December 20, 2021 at 9:43:46 AM UTC+11, James Harris wrote:
>
>> That's the plan. What do you think of it? Any better ideas? Suggestions
>> welcome.
>
> I don't know if there is enough space in a boot sector
> to do what you want. Also I think it is better to support
> CHS which uses up some space.

Yes, space is very limited. Hence the idea to split it over two (or
possibly more) sectors.

So far I have a VBR which reads through the root dir, finds the target
file, and loads its first sector. I have been working on getting that
sector to walk the FAT and load the rest of the file but I can already
see that the space will be tight - and this is just for LBA addressing.

I very much agree with you about CHS. I'd rather use CHS if the target
partition is within range of CHS addressing because it's compatible with
more BIOSes. But then I'd want to use LBA addressing if the bootable
partition is out of range of LBA.

I should mention that after formatting the partition with mkfs.fat (or
mkfs.vfat) I found that the geometry settings in the BPB didn't seem to
match the number of heads and sectors presumed for my image file so a
further load failed.

The problem may be because it was an image file. It may work for a real
hard drive. But it really needs to work for both. I am using just LBA
reads for now.

Interestingly, the man pages say: "mkfs.fat can not create boot-able
filesystems. This isn't as easy as you might think at first glance for
various reasons and has been discussed a lot already. mkfs.fat simply
will not support it ;)"

I don't know why mkfs.fat cannot make a partition bootable.

>
> I have a FAT-12/16 boot sector you can take code from
> if you want it. It's public domain so there is no restriction
> on what you use it for, no matter what license you choose.
>
> However, there appears to be a bug in it when used on a
> disk with the root sector starting on a non-multiple of
> track length.
>
> It's available here:
>
> https://sourceforge.net/p/pdos/gitcode/ci/master/tree/src/pbootsec.asm

Thanks. I may get some hints from it.

A number of years ago I wrote a FAT12 VBR, currently visible at one of

http://codewiki.pldev.org/aen/fatbootsect.nasm.php
http://codewiki.pldev.org/aen/fatbootsect.nasm

The approach for FAT16 has to be a bit different. For example, I need to
manipulate a number of 32-bit values where before I could work with
16-bit values. Also, a FAT12 FAT is small enough to read in in its
entirety but that for a FAT16 volume could be much larger so blocks from
it would have to be read in as required.

The main thing is to free up space for other purposes such as:

1. Better diagnostics.
2. A secured boot where each program checks some kind of hash of the
next and makes sure it matches what was expected.

So the more space I can find the better.


--
James Harris

[James Harris]

On 20/12/2021 04:54, s_dub...@yahoo.com wrote:
> On Sunday, December 19, 2021 at 4:43:46 PM UTC-6, James Harris wrote:

...

>> So loading would be in at least these parts:
>>
>> 1. VBR finds OS loader file and loads 1st sector of it.
>>
>> 2. OS loader's first sector follows the FAT chain to load more of (maybe
>> all of) OS loader.
>>
>> 3. The OS loader is then large enough to do whatever's necessary.

...

> Q. Is this fat-LFN?

It doesn't matter. I search for a file by an uppercase 8.3 name so it
will work with either. And see below about patents.

> Q. What is the size of the OS?

That doesn't come in to it, and is unknown as yet. The boot process has
to load something I call the "os loader". Its job will be to load the OS
or to offer a boot menu or to produce good diagnostics or whatever.

> Q. Are you sure the VBR cannot be made to load the entire OS? - eliminating the separate OS Loader?

Yes, there are lots of checks I want to do before loading the OS proper.

>
> The MBR I use is a vanilla ibm ~32 mb with one volume bootable, with the VBR expected at LBA = 20h.

I'm using my own MBR for now as it made testing easier but the VBR
should work with any normal MBR.

> The VBR is at LB 20h, The hobby OS begins at LB 21h and runs for 63.5k, it is outside the file system. The file system directory follows the OS, the file data follows the directory area. The file system is under development, it isn't needed to boot the OS, it will be needed to save data to files and to load utilities. - non fat.

It's certainly an option to have a custom arrangement and in some ways
that's more secure but for now I'm going for the convenience of using
standard structures as they are easier to work with.

Ultimately, I'd like the boot to go via a small dedicated 'boot
partition' but that's for the future.

>
> I don't support fat, nor LFN, LFN is covered by ms patents, so I dropped my work on it. But back when I was working with fat, that VBR code was enough to load a hobby os off the fat filesystem --non-lfn.

Aren't patents about production of media? And possibly compiled code?
ISTR that source code was not affected.

>
> The VBR that I'm using is of course loaded at 07C00h, it then relocates itself to below the EBDA and continues on to load the OS to 80:0000h, then far jumps to the OS. The self relocation is simple enough, and gets the vbr boot code out of the way of loading the OS, avoiding the possibility of over-writing itself.

Sounds good.


--
James Harris

[James Harris]

I'm amazed at how much you managed to fit in one sector. There are some
learning points - for example, I hadn't thought of using pusha and popa
- but more importantly it looks as though you load entire clusters at
once. I hadn't thought of that and had been doing it sector by sector.

What did you mean about the overflow bug?


--
James Harris

[Alexei A. Frounze]

But you have to remember that with large clusters and large files
you may be using more RAM than needed.

> What did you mean about the overflow bug?

I missed the case of FAT16 occupying more than 64KB:
https://github.com/alexfru/BootProg/issues/1
It's obviously fixable.

Alex

[Rod Pemberton]

On Sun, 19 Dec 2021 22:43:43 +0000
James Harris <james.h...@gmail.com> wrote:

> I've just started looking at booting from FAT16 and it's proving to
> be a bit of a challenge. That's because of what I want it to do and
> because space is soooo limited! I've set out my idea below. How does
> it look to you?
>
> For nomenclature, the sequence to boot from a hard disk can be seen as
>
> MBR --> VBR --> OSloader
>
> i.e. Master Boot Record (first sector of a disk) loads the Volume
> Boot Record (first sector of a partition). It, in turn, loads the
> (larger) OS loader which is big enough to include code to do a lot
> more.
>
> As a reminder, the layout of a FAT16 volume is
>
> <reserved><FATs><rootdir><filespace>
>

Ok.

Here you say this is for FAT16, but later on in the thread, you state
you're using your own MBR and VBR:

> I'm using my own MBR for now as it made testing easier but the VBR
> should work with any normal MBR.

and,

> So far I have a VBR which reads through the root dir, finds the
> target file, and loads its first sector.

If your stuff is actually FAT16, you can study how other DOS and
Windows MBRs and VBRs conserve space here:
https://thestarman.pcministry.com/asm/mbr/index.html

--

[Rod Pemberton]

On Mon, 20 Dec 2021 16:17:40 +0000
James Harris <james.h...@gmail.com> wrote:

> So far I have a VBR which reads through the root dir, finds the
> target file, and loads its first sector.

Why wouldn't you require that the file always be the first entry in the
root dir? ... That conserves space for more code, right? ...

Most OSes have special utilities which you're probably aware of that
place system files and structures into their proper locations on the
hard disk. Yes?

So, I'd think that a simple utility should be able to switch two
directory entries, rewrite the first directory entry with the target
file, rewrite the other entry also.

> I have been working on getting that sector to walk the FAT

Avoiding this "extra" code is obviously the reason other solutions hard
code specific sectors. Well, it could be extra code, if you avoided
doing this.

> with mkfs.fat (or mkfs.vfat)

From Linux for FAT16, I had to do something like:
(I'm unsure if this is complete ...)

ms-sys /dev/sdc
mkfs.vfat -v -F 16 /dev/sdc1
ms-sys -6 /dev/sdc1


--

[Rod Pemberton]

Oops, that's not complete ...

dd if=/dev/zero of=/dev/sdc count=16470 bs=512
fdisk /dev/sdc

/* now the stuff above */

The options used for Linux fdisk were: c n p 1 a t e w

That was for 16470 sectors for 8MB FAT16.

For a floppy, count=1440.

Just an FYI, I'm not sure why I was using fdisk with a floppy
image/disk, but the Linux fdisk options I used were: n p 2 t 11 w.

If you're not familiar with Linux fdisk, you'll have to work through
those Linux fdisk options on your own, as I think some of them required
inputting information, like partition size or starting track number,
etc.

FAT32 was more convoluted.


--

[James Harris]

On 31/12/2021 09:37, Rod Pemberton wrote:
> On Fri, 31 Dec 2021 04:22:24 -0500
> Rod Pemberton <noe...@basdxcqvbe.com> wrote:
>
>> On Mon, 20 Dec 2021 16:17:40 +0000
>> James Harris <james.h...@gmail.com> wrote:
>
>>> So far I have a VBR which reads through the root dir, finds the
>>> target file, and loads its first sector.
>>
>> Why wouldn't you require that the file always be the first entry in
>> the root dir? ... That conserves space for more code, right? ...

Well, what's best, to make it easy to update the boot files or to make
it difficult? AFAICS it's an open question. I don't know if there is one
right answer.

If I wanted to make it difficult then I could store boot code in special
locations. But the whole point of looking through the root directory is
to treat the OS loader as a normal file so that updates are easier and
the process is more transparent.

>>
>> Most OSes have special utilities which you're probably aware of that
>> place system files and structures into their proper locations on the
>> hard disk. Yes?

I agree that boot steps often use non-standard mechanisms. Whether they
should or not is another matter.

...

>>> with mkfs.fat (or mkfs.vfat)
>>
>> From Linux for FAT16, I had to do something like:
>> (I'm unsure if this is complete ...)
>>
>> ms-sys /dev/sdc
>> mkfs.vfat -v -F 16 /dev/sdc1
>> ms-sys -6 /dev/sdc1
>>

I'd never heard of ms-sys and it's not part of my distribution but it
looks useful.

>>
>
> Oops, that's not complete ...
>
> dd if=/dev/zero of=/dev/sdc count=16470 bs=512
> fdisk /dev/sdc
>
> /* now the stuff above */

I do similar:

dd bs=1024 count=20480 if=/dev/zero of=1-flat.vmdk
sfdisk .... [see below]
sudo losetup --show -Pf 1-flat.vmdk
sudo mkfs.fat -h 2048 /dev/loop???p1 {with ??? replaced}

Then

dd bs=1 count=446 ${DDPARMS} if=mbr of=1-flat.vmdk
dd bs=1 count=2 ${DDPARMS} if=mbr of=1-flat.vmdk skip=510
dd bs=1 count=3 ${DDPARMS} if=vbr of=${PBEG}
dd bs=1 count=450 ${DDPARMS} if=vbr of=${PSTA} skip=62

where
DDPARAMS is conv=notrunc status=none
PBEG is 1-flat.vmdk seek=$(( 2048 * 512 ))
PSTA is 1-flat.vmdk seek=$(( 2048 * 512 + 62 ))

Block 2048 is where the partition begins.

>
> The options used for Linux fdisk were: c n p 1 a t e w
>
> That was for 16470 sectors for 8MB FAT16.
>
> For a floppy, count=1440.
>
> Just an FYI, I'm not sure why I was using fdisk with a floppy
> image/disk, but the Linux fdisk options I used were: n p 2 t 11 w.
>
> If you're not familiar with Linux fdisk, you'll have to work through
> those Linux fdisk options on your own, as I think some of them required
> inputting information, like partition size or starting track number,
> etc.

Rather than controlling fdisk manually one can recreate a partition
table from a text file. For example, I use

sfdisk 1-flat.vmdk <<END
label: dos
label-id: 0xe70bb971
device: 1-flat.vmdk
unit: sectors
1-flat.vmdk1 : start= 2048, size= 79872, type=6, bootable
END

The text file can be created by hand or by

sfdisk --dump

or option O once in fdisk.


--
James Harris