Which filesystem is best for a boot partition

[James Harris]

The thread "Hard disk boot - space for extra code" led to the idea of
a boot partition. Any opinions as to what filesystem would be best to
use for that partition? In the fullest setup it would have to store
code and, likely, data representing a menu.

As a reminder, the idea of the boot partition is to provide extra
space so that a boot manager or similar can run before starting code
in the partition which contains an OS. As such the boot sequence would
be, all in real mode,

BIOS --> MBR --> [boot partition VBR] --> [boot partition code]

The boot partition code would be a boot manager or suchlike. It would
then facilitate selection of something to boot and, preferably, start
the VBR of that partition or do whatever else it had to do.

So, what filesystem would be best for the boot partition or should it
have some format that is not compatible with any filesystem?

James

[BGB]

the simplest "reasonably common" filesystem is probably FAT.
it is not terribly difficult to write a boot-loader for it, ...

for non-filesystem options, there is the possibility of a "bootable
image", which could consist of:
bootsector, followed by some-odd kB for the setup, maybe followed by a
kernel, and maybe followed by an "initrd" analogue or similar.

I would not really advise the above for an HDD partition though, as MBR
entries are scarce as-is, so there is little to gain from using up an
entire partition where a few files in the root-directory or similar of a
file-system would do.


if designing a custom filesystem for a HDD, a person "could" also take
the route of having a dedicated "boot region", and then reserving, say,
32kB or 512kB or similar, specifically for holding the setup code and/or
kernel. granted, for a "typical" FS this is probably overkill (it is not
usually "that" difficult to make a boot-loader capable of loading a few
files from the root directory or similar).


failing that, a hybrid strategy could be to have unfragmented files for
the setup and kernel, and maybe special entries in the boot-sector or
similar for where to load them from (allowing the boot loader to not
need to be able to understand the filesystem, but still allowing
file-based setup and kernel code).

for example, the boot sector could contain a small "loader address
table", with entries like, say:
DWORD firstSector; //first sector of file
WORD sectorCount; //length of file in sectors
WORD segmentBase; //base address

maybe a firstSector value of 0 means "done loading, jump here", with
segmentBase giving the jump-address or similar.


or such...

[Rod Pemberton]

"James Harris" <james.h...@gmail.com> wrote in message
news:5931be65-7622-4af9...@l3g2000vbv.googlegroups.com...
...

> Any opinions as to what filesystem would be best to
> use for that partition?

a) FAT16. For one, it has a standard: ECMA-107 or ISO/IEC 9293.
For two, lots of DOS utilities are available. There are even a number of
DOS utilities for setting up and using Linux from DOS.

b) Roll your own. Hey, it's a.o.d!

c) Personally, I would consider ISO 9660. It has a standard: ECMA-119 or
ISO 9660. POSIX support can be added via the Rock Ridge standard. The
advantage here is that a simple command, like 'dd' for Linux or
rawrite/rawread or partcopy etc., can be used to read or write an entire
.iso image as the initial boot partition. The image has a proper filesystem
within. Your bootloader could then boot an .iso from harddisk, cd-rom,
solid state disk, USB disk or stick, etc. DOS is the minimum needed to
write an .iso image to harddisk. I'm not sure what, if any, DOS utils exist
for creating .iso's.

d) A .zip file. Mostly, it's for similar reasons as I listed in c). An
advantage and disadvantage is the compression. On the one hand the
compression reduces the "partition" size, while on the other, it requires
decompression code in your bootloader ... DOS is the minimum needed to
create a .zip file and write the image to harddisk. Optionally, you could
use FAT16 or something else as your filesystem and just use .zip to compress
the filesystem image. Alternately, using .tgz or .rpm would be similar.
Package managers or file compressors are a good way to store an entire
filesystem.

e) Personally, I would also consider UDF. It has a standard:ECMA-167 or
ISO/IEC13356. It is newer, better, and more flexible than ISO 9660.
Apparently, it's primarily used for DVDs currently. However, I would think
it should be useable on all media types as a general purpose filesystem.
However, I don't recall seeing any image files of UDF disks ... They
probably do exist though. If you can create UDF image files, then, in
general, it's better than ISO-9660 since it could do everything I listed
in c) and more. However, you won't be able to access it or recover files
from it using DOS. DOS has no UDF support. In that regard, IS0-9660 is
better, since DOS can access ISO-9660.

f) Personally, I'd avoid ext2 or ext3. In addition to them being somewhat
"flaky" IMO, support to access them via other OSes is non-existant.
Actually, the lack of support probably applies for all Linux filesystems.
I'm not sure whether ext4 or ReiserFS are "flaky". ReiserFS worked fine for
the very short time I used it.

g) Personally, I'd avoid NTFS too. Support to access it from other OSes
was lacking and perhaps still is. When access from another OS is available,
it seems that read access is available, but not write access. If you're
trying to repair NTFS files or filesystem from another OS, that's a problem.
You might want to check if NTFS has patents on it.

h) Personally, while LFN support is nice, I'd have to add FAT32 to the list
to avoid because of patent issues.


I have no real opinion on most other filesystems. You should probably
decide whether a DOS filesystem or DOS recovery is important or not. I do
like some of the features of the CBM-DOS filesystem, but that may just be
nostalgia talking.

(CBM-DOS is Commodore's DOS for their line of floppies for their [non-x86]
PCs: VIC-20, C64, C128, PET/CBM, SX-64, Plus/4. FYI, you've probably never
used the terms PC and DOS outside of an IBM clone context ... However, they
once were used that way with all early home microcomputers. Each was
referred to as a PC and each had it's own DOS coded specifically for that
machine and it's microprocessor, i.e., Apple, Amiga, Commodore, etc. or had
multiple DOSes in the case of IBM PCs.)


Here is a good list of filesystems on Wikipedia:
http://en.wikipedia.org/wiki/Comparison_of_file_systems

From that list, I might consider a modern, open source, filesystem by a
major corporation. There are a few modern filesystems by major corporations
on the list. The issue of what type of license, if any.

If you implement your own filesystem, you'll need a value for your partition
type. The list of standard partition types is quite full, but has some free
values and low-use or obsoleted values. See the link you posted in the
other thread. Also, 0xe2 is in use.

As a side note, you could also mark your partition as 0x00 "empty" with the
risk that someone might overwrite it when using DOS or Linux partition
utilities. Under certain circumstances, "convincing" someone or their
utilities to either overlook it or overwrite it may be desirable ...
Alternately, you could use a heavily overloaded partition value, such as
0x07, to obscure it's true format, i.e., one of many, but actually
different. Or, you could use the partition type assigned for
alt.os.development: 0x7f. You'd have to track down the AODPS "standard"
circa 2002 to see what they were doing with it.


Rod Pemberton

[BGB]

On 4/30/2012 10:15 PM, Rod Pemberton wrote:
> "James Harris"<james.h...@gmail.com> wrote in message
> news:5931be65-7622-4af9...@l3g2000vbv.googlegroups.com...
> ...
>
>> Any opinions as to what filesystem would be best to
>> use for that partition?
>
> a) FAT16. For one, it has a standard: ECMA-107 or ISO/IEC 9293.
> For two, lots of DOS utilities are available. There are even a number of
> DOS utilities for setting up and using Linux from DOS.
>

yes, and a person can probably get by using FAT-32 as well.

> b) Roll your own. Hey, it's a.o.d!
>
> c) Personally, I would consider ISO 9660. It has a standard: ECMA-119 or
> ISO 9660. POSIX support can be added via the Rock Ridge standard. The
> advantage here is that a simple command, like 'dd' for Linux or
> rawrite/rawread or partcopy etc., can be used to read or write an entire
> .iso image as the initial boot partition. The image has a proper filesystem
> within. Your bootloader could then boot an .iso from harddisk, cd-rom,
> solid state disk, USB disk or stick, etc. DOS is the minimum needed to
> write an .iso image to harddisk. I'm not sure what, if any, DOS utils exist
> for creating .iso's.
>

ISO isn't really designed with read/write in mind, so would be
ill-suited as a general purpose FS.

as an advantage though, it gives 32kB or so in which to put a boot loader.

> d) A .zip file. Mostly, it's for similar reasons as I listed in c). An
> advantage and disadvantage is the compression. On the one hand the
> compression reduces the "partition" size, while on the other, it requires
> decompression code in your bootloader ... DOS is the minimum needed to
> create a .zip file and write the image to harddisk. Optionally, you could
> use FAT16 or something else as your filesystem and just use .zip to compress
> the filesystem image. Alternately, using .tgz or .rpm would be similar.
> Package managers or file compressors are a good way to store an entire
> filesystem.
>

ZIP can be used moderately well as a filesystem, including for random
access and read/write (sort of, hackery involved), but has a critical flaw:
it doesn't really scale all that well (probably roughly 10k to 100k
files is a practical upper-limit).

the main issue would mostly be with its lack of true directories,
requiring essentially dealing with the entire central directory to be
able to know its contents.


using Deflate+FAT16 or TGZ or similar would have a major disadvantage:
it would not allow for random access, and would essentially require
decompressing the whole image (say, to a ramdisk or similar) to be able
to access any of its contents.


possible could be to use a compressed file-system based on
"micro-sectors" or similar, where there is a virtual sector-size of,
say, 16 or 64 bytes, and probably the FS would be spans-based rather
than using a FAT or similar. at 64-byte sectors, there would be an
approx 0.2% overhead due to an allocation bitmap.

compression could use a modified block-based Deflate variant or similar.
say, files are compressed in 64kB virtual-blocks or similar (as deflate
will not compress much better if using blocks > 2x the window size).


this could make a lot more sense for smaller volumes (up to several GB
or so), and (unlike FAT16) would not waste lots of space due to large
clusters.

a small sector size would largely eliminate the need for things like
end-packing as well, but would require larger sector numbers, and likely
reduce its effectiveness for large volumes (although an implementation
could allocate data using larger clusters as well, say a "cluster size"
of 64 sectors, roughly working out to 4kB, or 256 for 16kB).

in this case, "clusters" would be used more for "files which aren't
tiny" (whereas larger clusters are less well suited to files which will
often only use a small part of a cluster anyways).

> e) Personally, I would also consider UDF. It has a standard:ECMA-167 or
> ISO/IEC13356. It is newer, better, and more flexible than ISO 9660.
> Apparently, it's primarily used for DVDs currently. However, I would think
> it should be useable on all media types as a general purpose filesystem.
> However, I don't recall seeing any image files of UDF disks ... They
> probably do exist though. If you can create UDF image files, then, in
> general, it's better than ISO-9660 since it could do everything I listed
> in c) and more. However, you won't be able to access it or recover files
> from it using DOS. DOS has no UDF support. In that regard, IS0-9660 is
> better, since DOS can access ISO-9660.
>

not really looked into UDF to determine if they have any "good" ways of
dealing with read/write filesystems.

> f) Personally, I'd avoid ext2 or ext3. In addition to them being somewhat
> "flaky" IMO, support to access them via other OSes is non-existant.
> Actually, the lack of support probably applies for all Linux filesystems.
> I'm not sure whether ext4 or ReiserFS are "flaky". ReiserFS worked fine for
> the very short time I used it.
>

ReiserFS was essentially a giant B*Tree.

what largely killed it off though was that the guy who created it went
and killed his wife, which didn't really go over all that well in the
Linux community.

> g) Personally, I'd avoid NTFS too. Support to access it from other OSes
> was lacking and perhaps still is. When access from another OS is available,
> it seems that read access is available, but not write access. If you're
> trying to repair NTFS files or filesystem from another OS, that's a problem.
> You might want to check if NTFS has patents on it.
>

AFAIK, NTFS does have patents, but this is nothing unusual here.

> h) Personally, while LFN support is nice, I'd have to add FAT32 to the list
> to avoid because of patent issues.
>

FAT32 isn't really as much of an issue AFAIK.

the main "issue" is mostly regarding LFN, and mostly when using "mixed"
8.3 and LFNs on the same file at the same time.

if one uses only a single name for the file (either the 8.3 name or the
LFN, ignoring the 8.3 name entirely if an LFN exists), then arguably the
patent doesn't apply.


AFAIK, the patents also apply to FAT12 and FAT16 with LFNs as well, so
using FAT16 doesn't escape it (since FAT16 LFNs are implemented pretty
much exactly the same as the ones in FAT32).

unless there is some other patent here specific to FAT32 (rather than to
LFNs in general).

>
> I have no real opinion on most other filesystems. You should probably
> decide whether a DOS filesystem or DOS recovery is important or not. I do
> like some of the features of the CBM-DOS filesystem, but that may just be
> nostalgia talking.
>
> (CBM-DOS is Commodore's DOS for their line of floppies for their [non-x86]
> PCs: VIC-20, C64, C128, PET/CBM, SX-64, Plus/4. FYI, you've probably never
> used the terms PC and DOS outside of an IBM clone context ... However, they
> once were used that way with all early home microcomputers. Each was
> referred to as a PC and each had it's own DOS coded specifically for that
> machine and it's microprocessor, i.e., Apple, Amiga, Commodore, etc. or had
> multiple DOSes in the case of IBM PCs.)
>
>
> Here is a good list of filesystems on Wikipedia:
> http://en.wikipedia.org/wiki/Comparison_of_file_systems
>
> From that list, I might consider a modern, open source, filesystem by a
> major corporation. There are a few modern filesystems by major corporations
> on the list. The issue of what type of license, if any.
>
> If you implement your own filesystem, you'll need a value for your partition
> type. The list of standard partition types is quite full, but has some free
> values and low-use or obsoleted values. See the link you posted in the
> other thread. Also, 0xe2 is in use.
>

pretty much all are in use already.

many OS's use magic-values embedded in the boot-sector or similar
instead as their primary means of determining one filesystem from another.

> As a side note, you could also mark your partition as 0x00 "empty" with the
> risk that someone might overwrite it when using DOS or Linux partition
> utilities. Under certain circumstances, "convincing" someone or their
> utilities to either overlook it or overwrite it may be desirable ...
> Alternately, you could use a heavily overloaded partition value, such as
> 0x07, to obscure it's true format, i.e., one of many, but actually
> different. Or, you could use the partition type assigned for
> alt.os.development: 0x7f. You'd have to track down the AODPS "standard"
> circa 2002 to see what they were doing with it.
>

dunno if it is still around anywhere online.

I have a copy of it though.

[Harry Vaderchi]

A slight drawback is that you've used up a partition and a (conventional)
MBR only has 4 primary partitions; you want use 1 for the menu, 1 will be
the primary OS, another the alternative (which is a minimum for n-way
booting), so the min is 2 and max is 3!

--
[dash dash space newline 4line sig]

Albi CNU

[Harry Vaderchi]

On Tue, 01 May 2012 06:15:32 +0100, Rod Pemberton
<do_no...@notemailnot.cmm> wrote:

[good stuff snipped]

> f) Personally, I'd avoid ext2 or ext3. In addition to them being
> somewhat
> "flaky" IMO, support to access them via other OSes is non-existant.

well there is ext2fs under XP

.
>
> g) Personally, I'd avoid NTFS too. Support to access it from other OSes
> was lacking and perhaps still is. When access from another OS is
> available,
> it seems that read access is available, but not write access. If you're
> trying to repair NTFS files or filesystem from another OS, that's a
> problem.

I have a "dual" boot small linux (xpud) which has RW to NTFS.


But I agree with not using these formats for a small? boot menu - FAT16
should be rough and ready enough, assuming you even want to have a FS!

[James Harris]

On Apr 30, 10:47 pm, BGB <cr88...@hotmail.com> wrote:

...

> >    BIOS -->  MBR -->  [boot partition VBR] -->  [boot partition code]

...

> > So, what filesystem would be best for the boot partition or should it
> > have some format that is not compatible with any filesystem?

...

> I would not really advise the above for an HDD partition though, as MBR
> entries are scarce as-is, so there is little to gain from using up an
> entire partition where a few files in the root-directory or similar of a
> file-system would do.

Some existing boot managers put their code in an OS partition but that
scheme fails to separate the boot menu from what gets booted. It
integrates the boot menu into the partition of one particular
operating system. ISTM that if the boot menu is going to provide
options to boot various OSes it should not be incorporated into any
one of them for two reasons:

1. the user should be able to rebuild or remove an OS partition
without losing the boot menu for any other partitions

2. the user should be able to modify the boot menu from any OS. Often
such a menu can only be manipulated from the hosting OS, for example.
Grub's menu is modified while running Linux. Windows' boot menu is
modified while running Windows.

You are right, though, that partition entries are limited so is it
practical? This came up in another thread.

http://groups.google.com/group/alt.os.development/msg/83fb753fc85ad120

That post was about a couple of options so if I extract the salient
bits (and edit slightly) below maybe you can tell me if you think it
practical after all. Edited extract follows.

For a separate boot partition there are two basic requirements:

* some free disk space
* at least one unused partition table entry

Can we expect those to be available? I think so. Both are reasonable
to expect because if someone wants to install another OS on an
existing disk these are requirements whether there is to be a pre-OS
boot system or not. A new OS would require free space. It would also
require a partition for it to be installed in.

In terms of space, since the pre-OS partition will be small it also
won't take away anything significant from the space the user might
plan to install the OS on.

However, the requirement for an unused partition table entry is more
interesting. Say the disk has just one partition table entry which is
not already in use. It could be used for the OS or it could be used
for the pre-OS partition but not both.

So the question is; "if we use such a scheme how could we install both
our OSes and a pre-OS partition on a disk which has just one free
partition table entry?"

One option: use a couple of EBRs. Remember, these normally form a
chain where each one is pointed at by the preceding boot record. So we
would have

MBR --> EBR --> EBR

Each EBR normally contains the definition of one partition and a
pointer to the next EBR (or no pointer if it is the last).

Given the structure, say we take a remaining MBR partition
entry and create an extended partition. We can then split that
extended partition into a small piece for the pre-OS partition and a
larger piece for the OS.

If, on the other hand, there is already an extended partition (and
space) we can add the boot partition and the OS partition to the
chain.

Does that cover it?

James

[Rod Pemberton]

"BGB" <cr8...@hotmail.com> wrote in message
news:jno4gg$lii$1...@news.albasani.net...

> On 4/30/2012 10:15 PM, Rod Pemberton wrote:

...

> > e) Personally, I would also consider UDF. It has a standard:ECMA-167 or
> > ISO/IEC13356. It is newer, better, and more flexible than ISO 9660.
> > Apparently, it's primarily used for DVDs currently. However, I would
> > think it should be useable on all media types as a general purpose
> > filesystem. However, I don't recall seeing any image files of UDF disks
> > ... They probably do exist though. If you can create UDF image files,
> > then, in general, it's better than ISO-9660 since it could do everything
> > I listed in c) and more. However, you won't be able to access it or
> > recover files from it using DOS. DOS has no UDF support. In that
> > regard, IS0-9660 is better, since DOS can access ISO-9660.
> >
>
> not really looked into UDF to determine if they have any "good" ways of
> dealing with read/write filesystems.
>

Unlike ISO 9660, UDF apparently is read/write for all media, including
write-once.
http://en.wikipedia.org/wiki/Universal_Disk_Format

From the stats in tables on the Wikipedia filesystem comparison page, UDF
looks comparable to some of the better filesystems.
http://en.wikipedia.org/wiki/Comparison_of_file_systems

UDF seems to be similar to NTFS and ext4 except for a few things like
"sparse files" and "Security MAC labels". It's green where the columns in
the tables are predominantly green, and vice-versa for red.

From the "Supporting operating systems" table, it would seem UDF is the most
widely supported after FAT12/16/32. However, ISO 9660 is missing from that
table. Actually, ISO 9660 is missing from "Features" and "Allocation and
layout" too. I wonder what happened there ... ? The ISO 9660 page lists
the left-most 8 OSes in the table.

Obviously, UDF has not been adopted as a generic filesystem. The question
is "Why?".


Rod Pemberton

[Rod Pemberton]

"Harry Vaderchi" <ad...@127.0.0.1> wrote in message
news:op.wdmjhlwl1r0rdn@dell3100...

> On Mon, 30 Apr 2012 21:38:55 +0100, James Harris
> <james.h...@gmail.com> wrote:
>
> > The thread "Hard disk boot - space for extra code" led to the idea of
> > a boot partition.
>

> A slight drawback is that you've used up a partition and a (conventional)
> MBR only has 4 primary partitions;
>

Well, he could always have his disk startup code hardcoded to use a specific
region on the disk. Then, he doesn't waste a partitition entry. Of course,
standard partitioning tools wouldn't be aware that space had been allocated
without a partition entry. So, he'd need customized utilities, or he'd need
tell people to not allocate partitions there. That would interfere somewhat
with his idea of it working well with multiple OSes.


Rod Pemberton

[wolfgang kern]

James Harris polled:

A boot partition doesn't need to be compatible to any filesystem at all.
But during creating/evalution of an OS it helps to have a known environment
and this just depends on the choosen tool.

somehow later in the game... :
I'd say it depends on the type of OS you may have in mind...
A safe system would have all its internals (core/modules/debugger/..)
stored outside the of user-accessible range, while a workstation should
allow and show all available features to a programmer quite transparent.

the choice is yours and noone elses! (evita..)

__
wolfgang

[James Harris]

On May 1, 6:15 am, "Rod Pemberton" <do_not_h...@notemailnot.cmm>
wrote:

...

> > Any opinions as to what filesystem would be best to
> > use for that partition?
>
> a) FAT16.  For one, it has a standard: ECMA-107 or ISO/IEC 9293.
> For two, lots of DOS utilities are available.  There are even a number of
> DOS utilities for setting up and using Linux from DOS.

Yes, based on the feedback it seems FAT16 would be best - at least for
now. If it turns out later that some extra security is needed the file
system could be changed but FAT16 would be easy to develop with.

Any opinions on which variant of FAT16 should be used - type 4 or type
6? Size is not an issue but remember that the executable code is to be
16-bit so type 4 might be slightly easier. Would it be compatible with
subsequent drivers?

http://en.wikipedia.org/wiki/File_Allocation_Table#Initial_FAT16
http://en.wikipedia.org/wiki/File_Allocation_Table#Final_FAT16

Also, where to store the code? I was thinking of reserved sectors. The
fatgen doc, however,

http://www.google.co.uk/search?q=fatgen103

says for max compatibility not to use reserved sectors. Maybe PC
operating systems are OK with them and the compatibility comment
relates to others such as embedded devices.

If reserved sectors are not to be used the code could probably (just)
be read from the root dir as in FAT12 in

http://codewiki.wikispaces.com/fatbootsect.nasm

Such a course would make it easy to update - just copy the new code to
the partition and reboot.

James

[James Harris]

On May 2, 6:48 pm, "wolfgang kern" <nowh...@never.at> wrote:

...

> > So, what filesystem would be best for the boot partition or should it
> > have some format that is not compatible with any filesystem?
>
> A boot partition doesn't need to be compatible to any filesystem at all.
> But during creating/evalution of an OS it helps to have a known environment
> and this just depends on the choosen tool.

Yes. For the file system type I've decided (when I get back to it ...
I've started a short university course that's been taking up most of
my non-work time and I've also started sea kayaking lessons - though
with the way that's been going in my case it's more like swimming -
more submariner than mariner!) erm, anyway as I was saying, when I get
back to the boot partition idea I'll start with FAT because it is most
compatible. I can always change it later.

On the idea of a FAT partition, based on Microsoft's fatgen103
document I came across the following points. They show FAT as more
complex than is desirable but may be of interest to people here.

1. A very small FAT partition of, a meg or two, cannot be FAT16. Such
a small partition has to be FAT12 even if it is on a hard disk!

2. I can force the partition to be FAT16 if the number of clusters
(not the number of sectors) is high enough.

3. There is more than one version of FAT16

http://en.wikipedia.org/wiki/File_Allocation_Table#Initial_FAT16

4. There are many versions of the BPB (BIOS parameter block). Apart
from the total number of logical sectors which has 2-byte and 4-byte
variants I'm not too sure yet how software is supposed to detect that
a disk has an earlier format BPB than the one than it (the software)
is written for.

http://en.wikipedia.org/wiki/File_Allocation_Table#BIOS_Parameter_Block

5. MS says for maximum compatibility not to use the BPB Reserved
Sectors. If I'm going to follow that then the code which gets loaded
by the VBR would have to be in the normal part of the file space. Such
compatibility issues may only exist in obscure devices, though. All PC
operating systems may be more capable.

> the choice is yours and noone elses! (evita..)

"evita"? Does she get a choice as well?

James

[Rod Pemberton]

"James Harris" <james.h...@gmail.com> wrote in message

news:122d037d-05db-4af6...@w10g2000vbc.googlegroups.com...
...

> ... Microsoft's fatgen103 ...


Does anyone know if fatgen103 is actually compliant with
ECMA-107 or ISO/IEC 9293?


Unfortunately, MS is notorious for "tweaking" specifications.
I'm sure fatgen103 must be complied with for Windows and DOS.


Rod Pemberton