Proposal: *One* disk image format for *all* IIgs emus

[Henrik 'Ratte' Gudat]

Hi,

The arrival of IIgs emulators for both PCs and Macs is a great thing, but
there's one drawback. I think we should all agree upon ONE common disk format
for disk images (block-by-block, not raw data) and 3.5" disk dumps (nibbelized
raw data).

I think this would make life for everyone a whole lot easier. And right now
while all IIgs emus are still under development, it'd be a good opportunity to
avoid a biiiiiiig mess in forseeable future. AFAIK Disk Copy images (dImg) seem
to be pretty much standard on the Mac side, but IMO there's no reason to
restrict a file format to a particular platform.

Any comments?

henrik

Fast Eddie Labs

[Joshua M. Thompson]

Henrik 'Ratte' Gudat wrote:

> The arrival of IIgs emulators for both PCs and Macs is a great thing,
> but there's one drawback. I think we should all agree upon ONE common
> disk format for disk images (block-by-block, not raw data) and 3.5"
> disk dumps (nibbelized raw data).

I agree completely. I brought up the idea a few days ago on xgs-list,
and we all tossed around some ideas. To summarize (and add a bit of my
own opinion):

1. The format should be the same for a 5.25", 3.5", or hard disk image
2. It should be easy to create and manipulate these images from Unix,
MacOS, Windows, and ideally GS/OS.
3. The name of the image and some comments should be stored in the
header somewhere for easy viewing. Comments would be things like
"You must load this image in S5,D1 in order to boot from it."
4. For 5.25" and 3.5" floppies we should be able to store the data in
raw block dump format or as raw nibbles.
5. There should be some sort of compression. Gzip would be an ideal
format since the source code for doing this is GPL'd (although this
may be a problem for FastEddie since GPL requires source code
availability).

I'm willing to change my disk image format, since there aren't yet a
great deal of .XGS format images out there that would need to be
converted. All that needs to be done is to come up with the final
format.

Now, without trying to plug my product here, the .XGS format isn't all
that bad a start.:-) It _almost_ fulfills numbers 1-3 of the above
suggestions (I say almost because I don't think anyone's ported the
image utilities to MacOS yet). My header format is a bit convulted
though and could use some cleaning up as well as some extensions to
support the nibble format ond compression options.

--
email: in...@mich.com Senior Systems Engineer
web: http://www.optera.com/~invid mich.com, Inc. Farmington, MI
PGP fingerprint: 90 05 50 DA 0E 18 CD 47 C5 25 83 1F A5 41 17 73

[Thye Chean]

Henrik 'Ratte' Gudat wrote:
> 
> Hi,

> 
> The arrival of IIgs emulators for both PCs and Macs is a great thing, but
> there's one drawback. I think we should all agree upon ONE common disk format
> for disk images (block-by-block, not raw data) and 3.5" disk dumps (nibbelized
> raw data).

Since there are basically only 2 emulators for GS (I don't believe Apple will ever

release GUS based on its past record), I think it is really a great suggestion, and

should not be difficult to do. XGS team has already expressed the interest of

redesigning the disk image format, so this is really a good time for it.

 

I was wondering whether there is a better way for PC user to create disk image...

right now I have to insert GS floppy to Mac, create a DC image on Mac, copy

it to PC floppy and convert it to xgs format. Any faster way for PC or Mac will

be welcome. Maybe just use DC image should be fine too...

 

Thye Chean

[Zaphod1342]

This is a wonderful idea, and I suppose everyone wants it. But, it can't
happen because everybody wants different things for it. Some say it
should be compressed. That's good for the downloader. Others want it
easy to implement and straightfoward. That's good for the developer.
Being a developer, I suppose you'd like a straightfoward format. No, on
second thought, you're the developer of Fast Eddie. Compression shouldn't
be a problem for a mastermind like you.

Keep up the good work! Zaphod

[Jeff Blakeney]

On 16 Jan 1997 01:49:04 GMT, Joshua M. Thompson <in...@mich.com>
wrote:

>1. The format should be the same for a 5.25", 3.5", or hard disk image
>2. It should be easy to create and manipulate these images from Unix,
> MacOS, Windows, and ideally GS/OS.
>3. The name of the image and some comments should be stored in the
> header somewhere for easy viewing. Comments would be things like
> "You must load this image in S5,D1 in order to boot from it."
>4. For 5.25" and 3.5" floppies we should be able to store the data in
> raw block dump format or as raw nibbles.

These all sound quite good but what do you plan to do about the 5.25"
formats that already exist? There are a lot of .DSK and .NIB disks
already available that a lot of people would have to convert if you
came up with a different way of handling them.

>5. There should be some sort of compression. Gzip would be an ideal
> format since the source code for doing this is GPL'd (although this
> may be a problem for FastEddie since GPL requires source code
> availability).

This I don't really see a need for. Large capacity drives don't cost
a whole lot and having an uncompressed image format makes it much
easier for people like me to write utilities and such to deal with. I
wouldn't want to try to incorporate someone's compression code into
any utility I might write nor would I want to have to learn the
compression technique to write my own code. For quick and dirty hacks
a compressed image just doesn't help.

Mind you, if the compression was an option and you could use either
compressed on non-compressed images then there wouldn't be a problem.

[Joshua M Thompson]

Zaphod1342 <zapho...@aol.com> wrote:

: This is a wonderful idea, and I suppose everyone wants it. But, it can't

: happen because everybody wants different things for it. Some say it
: should be compressed. That's good for the downloader. Others want it
: easy to implement and straightfoward. That's good for the developer.

There is no reason these features have to be mutually exclusive. I can
think of several ways to implement all these features in a straighforward
manner. The only real obstacle is going to be opinion, but from what I
have seen so far I think everybody's opinions on this are close enough
together that nearly everyone can be satisfied.

[Henrik 'Ratte' Gudat]

In <5bk1eg$p...@server1.mich.com> Joshua writes:

> I agree completely. I brought up the idea a few days ago on xgs-list,
> and we all tossed around some ideas. To summarize (and add a bit of my
> own opinion):

thats' great you've been playing with that idea, too

> 1. The format should be the same for a 5.25", 3.5", or hard disk image

yep

> 2. It should be easy to create and manipulate these images from Unix,
> MacOS, Windows, and ideally GS/OS.

yep

> 3. The name of the image and some comments should be stored in the

[..]

yep

> 4. For 5.25" and 3.5" floppies we should be able to store the data in
> raw block dump format or as raw nibbles.

yep

> 5. There should be some sort of compression. Gzip would be an ideal
> format since the source code for doing this is GPL'd (although this
> may be a problem for FastEddie since GPL requires source code
> availability).

You name it :-)
I agree that compression would be kinda cool, but this would only work for me
if I have the source code and nobody claims a copyright on that. But I guess
these restrictions are obvious. I'm not sure how you will be handling
compression at runtime. If there's some sort of compression algorithm we can
recycle, disk images woulkd have to be decompressed before actually using them.

In a common disk format, I would expect:
- a signature right at the beginning
- disk image version byte
- name of disk (space for looong file names)
- disk type (3.5/generic or 5.25)
- 8 bit data or nibbelized data
- interleave (for non-nibbelized 3.5" disk images), 0 for unknown
- size of disk in blocks
- compression flag
- write protected flag
- comment
- enough space for future additions

Again, if anyone has a good compression algorithm that would do the trick, i'd
appreciate it.

> Now, without trying to plug my product here, the .XGS format isn't all
> that bad a start.:-) It _almost_ fulfills numbers 1-3 of the above
> suggestions (I say almost because I don't think anyone's ported the
> image utilities to MacOS yet). My header format is a bit convulted
> though and could use some cleaning up as well as some extensions to
> support the nibble format ond compression options.

Could you post your header (after cleaning :-))) )? We do not need to reinvent
the wheel completely, and honestly I don't care if the disk name is at +10 or
+100. :-)

cu,
henrik

[David Andrew Ross]

Jeff Blakeney (jef...@bconnex.net) wrote:

: >5. There should be some sort of compression. Gzip would be an ideal

: > format since the source code for doing this is GPL'd (although this
: > may be a problem for FastEddie since GPL requires source code
: > availability).

: This I don't really see a need for. Large capacity drives don't cost

: a whole lot and having an uncompressed image format makes it much
: easier for people like me to write utilities and such to deal with. I
: wouldn't want to try to incorporate someone's compression code into
: any utility I might write nor would I want to have to learn the
: compression technique to write my own code. For quick and dirty hacks
: a compressed image just doesn't help.

: Mind you, if the compression was an option and you could use either
: compressed on non-compressed images then there wouldn't be a problem.

I was one of the big supporters of compression, mainly because
of my large archive of XGS images. Size is a problem, especially for
people who want to download images over slow links. I've got 5.5 gigs
of disk space, but having a few large HD images as well as lots of normal
800K disk images around on my drive really did have an effect. Having
compression be optional would be nice I guess, but I think I would write
some code to run through the incoming directory of my archive and remove
any uncompressed images. =) BTW: The archive will be back up at the
start of Feb. Once again, ROM images and things of that nature will
not be permitted.

--
David Ross
dr...@pobox.com

[Thye Chean]

David Andrew Ross wrote:

>      I was one of the big supporters of compression, mainly because
> of my large archive of XGS images.  Size is a problem, especially for
> people who want to download images over slow links.  I've got 5.5 gigs
> of disk space, but having a few large HD images as well as lots of normal
> 800K disk images around on my drive really did have an effect.  Having
> compression be optional would be nice I guess, but I think I would write
> some code to run through the incoming directory of my archive and remove
> any uncompressed images.  =)  BTW: The archive will be back up at the
> start of Feb.  Once again, ROM images and things of that nature will
> not be permitted.

This can be solved quite easily. Use DriveSpace or whatever utilities. For

network transfer, why don't we just zip it up just like other utilities? Or

StuffIt, or gzip it... take a common one for all machines.

 

I strongly oppose the compression format if it affects performance, especially

on a large capacity disk image (eg 32MB). XGS is slow enough already...

 

Thye Chean

[Nathan Mates]

In article <5bjl2j$m...@elna.ethz.ch>,

Henrik 'Ratte' Gudat <GUD...@EZINFO.VMSMAIL.ETHZ.CH> wrote:
>The arrival of IIgs emulators for both PCs and Macs is a great thing, but
>there's one drawback. I think we should all agree upon ONE common disk format
>for disk images (block-by-block, not raw data) and 3.5" disk dumps (nibbelized
>raw data).

I think you're still thinking in a //e format where disk images are
the norm simply because of the numbers of hacked OSs out there. On
the other hand, the GS has most to all of its stuff in nicely readable
*files*, with the occasional (mostly demos) block-by-block disk. There
*ALREADY* is a file format that does all this.

SHRINKIT.

Let's look at all its features: Supports disk images (5.25" *and*
3.5" *and* any other size you want). Supports saving off files and/or
directory trees (disk images suck once you've gotten a
HD). Compression: yes. Optional comments in files: yes. Long file
names internally: yes. *REAL* Apple II friendly: yes. All legal Apple
II and GS ftp sites _already_ have their stuff in shrinkit form, no
need to convert or make alternate sites. Source code freely available:
yes. [And probably a few more I forgot]

If you were to make a .SHK'd volume that's pretty big, you may want
to decompress to a temporary file on disk/ram and run off there, or
just make a readonly disk decompressing stuff.

Unless you're malicious, there is no need to split the Apple II
community between the legitimate Apple II users and the emulated
crowd. If you want to give people access to the huge existing ftp
archives with almost no trouble, support shrinkit.

Nathan Mates

--
<*> Nathan Mates http://www.visi.com/~nathan/ <*>
# What are the facts? Again and again and again-- what are the _facts_?
# Shun wishful thinking, avoid opinion, care not what the neighbors
# think-- what are the facts, and to how many decimal places? -R.A. Heinlein

[Henrik 'Ratte' Gudat]

Ok, in order to move along, i'm making some concrete proposals now. :)

Compression: since compression should be optional, may I suggest we drop this
just for the moment and add a compression feature later. As some other said, it
makes a difficult matter only more complicated. If someone really wants
compression *now*, he or she can use a compression tool locally.

Raw data: i would like to have a disk image format with plain raw data, no
header info whatsoever. It would contain blocks of 512 bytes each, file length
must be exactly a multiple of 512 in order to be recognized as such.

Format w/ header: I have written what flags etc. I would expect in a more
sophisticated image file format. As I said above, compression would be disabled
for the time being. We can use XGS' header and add whatever is missing. This
format would be a container for both nibblized data and raw data.

- henrik

[Jeff Blakeney]

On 17 Jan 1997 21:12:53 GMT, sna...@Glue.umd.edu (David Andrew Ross)
wrote:

> I was one of the big supporters of compression, mainly because
>of my large archive of XGS images. Size is a problem, especially for
>people who want to download images over slow links. I've got 5.5 gigs
>of disk space, but having a few large HD images as well as lots of normal
>800K disk images around on my drive really did have an effect. Having
>compression be optional would be nice I guess, but I think I would write
>some code to run through the incoming directory of my archive and remove
>any uncompressed images. =) BTW: The archive will be back up at the
>start of Feb. Once again, ROM images and things of that nature will
>not be permitted.

The easiest way to deal with this issue is to keep the disk images
uncompressed. If anyone wants to have all their disk images
compressed while using an emulator, they can use one of the disk
compression utilities that are available for their particular
computer.

I believe most people will only keep a small number of images on their
hard drive at a time. Any that they want to keep handy to use they
would keep archived either on their hard drive or somewhere else.
Archiving includes programs like gzip, pkzip and StuffIt.

Archiving is also the way to save space on ftp servers and speed up
transfers over slow links.

Another thing to consider, what happens if a IIgs emulator user has
AutoArk or Hard Pressed installed? The GS/OS will be compressing the
data it is saving to the virtual hard drive and then the emulator will
also try to compress that data. Now either the emulator's compression
routine is going to have to be smart enough not to try to compress
already compressed data (because this usually ends up being larger
than the original compressed size) or it will have to use a much
better compression algorithm to ensure that the resulting disk image
does not get bigger. Just another little thing to think about.

[Joshua M Thompson]

Henrik 'Ratte' Gudat <GUD...@EZINFO.VMSMAIL.ETHZ.CH> wrote:

: Ok, in order to move along, i'm making some concrete proposals now. :)

Sorry I'm late, we're having major RAID problems with the mich.com news
server, so my news access comes and goes.

: Compression: since compression should be optional, may I suggest we drop this

: just for the moment and add a compression feature later. As some other said, it
: makes a difficult matter only more complicated. If someone really wants
: compression *now*, he or she can use a compression tool locally.

Fine by me. As long as we have some left over flags I see no problem with
worrying about this later.

: Raw data: i would like to have a disk image format with plain raw data, no

: header info whatsoever. It would contain blocks of 512 bytes each, file length
: must be exactly a multiple of 512 in order to be recognized as such.

That's basically the existing .DSK and .PO image files. XGS will definately
be reading these "natively" very soon to simply things.

: Format w/ header: I have written what flags etc. I would expect in a more

: sophisticated image file format. As I said above, compression would be disabled
: for the time being. We can use XGS' header and add whatever is missing. This
: format would be a container for both nibblized data and raw data.

And with that, here's what little header currently exists:

/* Description of an XGS image file header. */
/* Note that multi-byte values are in LSB order! */

typedef struct {
char magic[4]; /* Should be "XGS!" */
byte version; /* Format version (currently 0x00) */
byte pad1; /* padding (x86 compatibility) */
word16 num_blocks; /* Number of blocks in image */
byte protected; /* Nonzero if image is protected */
char name[32]; /* Name of this image */
char desc[1024]; /* Description of this image */
byte pad2[3]; /* padding (x86 compatibility) */
byte spare[8]; /* Spare bytes */
} xgs_image_header;

The darn thing really needs redesigned; I had to add the pad and spare
bytes back when I first compiled XGS on Linux/Alpha, because of the
different alignment requirements and because I stupidly had a pointer
field stored in the header that was filled in when the header was loaded,
(on the Alpha pointers are 8 bytes instead of four).

[Joshua M Thompson]

Nathan Mates <nat...@visi.com> wrote:

: I think you're still thinking in a //e format where disk images are

: the norm simply because of the numbers of hacked OSs out there. On
: the other hand, the GS has most to all of its stuff in nicely readable
: *files*, with the occasional (mostly demos) block-by-block disk. There
: *ALREADY* is a file format that does all this.

: SHRINKIT.

We've been talking about that actually. There are a few problems with it
however:

1. It takes a fair amount of code to handle NuFX archives, especially
if one wants to use the file-by-file images (you have to build a
whole fake filesystem to be seen by the emulated drives). Right now
we're dealing with emulation-specific formats that are pretty much
a header slapped in front of some raw disk data.

2. The compression format doesnt allow for writing to images unless you
decompress the whole thing first, make your changes, and rewrite the
archive. This is also the big reason I'd rather deal with a compressed
format some other time. :)

3. No support for images stored as raw track nibble dumps. Without this,
there will be some programs that you're going to have to crack to
run on the emulator (mostly IIe stuff, but we want to support that
too).

[Henrik 'Ratte' Gudat]

In <5btg57$8...@server1.mich.com> Joshua writes:

[compression]

> Fine by me. As long as we have some left over flags I see no problem with
> worrying about this later.

ok

[512b-raw-images]

> That's basically the existing .DSK and .PO image files. XGS will definately
> be reading these "natively" very soon to simply things.

ok. These do also exist on the Mac (hdrv and vdsk). No big deal.

> typedef struct {
> char magic[4]; /* Should be "XGS!" */
> byte version; /* Format version (currently 0x00) */
> byte pad1; /* padding (x86 compatibility) */
> word16 num_blocks; /* Number of blocks in image */
> byte protected; /* Nonzero if image is protected */
> char name[32]; /* Name of this image */
> char desc[1024]; /* Description of this image */
> byte pad2[3]; /* padding (x86 compatibility) */
> byte spare[8]; /* Spare bytes */
> } xgs_image_header;

That is looking pretty good. The only flags I'm missing are obviously the
compression and raw/nibble flags. Hmm, maybe we can change the signature "XGS!"
to something more generic? :-)

I have another suggestion that is a bit against the cross-platform spirit, but
I think I would be a good thing for the user. If we added a creator signature
right after the "magic[4]" plus some spare bytes, each emulator could store
some private data in the image, such as the default device mapping for this
image. Or any other information that would only apply for that particular
emulator. Alternately that information could be added to the end of the file,
right after the disk content, so it would not interfere wityh the header.

BTW, can we make the blocks 32bit? The extended smartport interface allows for
>65535 blocks, and so does GS/OS.

char magic[4]; /* "wxyz"
char creator[4]; /* "XGS!" or whatever emu
^^^^^^^^^^^^^^^^^^
word16 header; /* length of header
^^^^^^^^^^^^^^

byte version; /* Format version (currently 0x00) */
byte pad1; /* padding (x86 compatibility)

word32 num_blocks; /* Number of blocks in image
^^^^^^

byte protected; /* Nonzero if image is protected */

byte imageType; /* 0=raw data, 1=nibble (whatever)
^^^^^^^^^^^^^^^^^

char name[32]; /* Name of this image

char desc[1024]; /* Description of this image

byte pad2[3]; /* padding (x86 compatibility)

byte spare[8]; /* Spare bytes

cu,
henrik

[Joshua M Thompson]

Ok, here's my proposal. This is has all the same things as your proposal,
plus a few enhancements:

typedef struct {
char magic[4]; /* "2IMG" */
char creator[4]; /* Creator signature */
word16 header_len; /* Length of header in bytes k */
word16 version; /* Image version */
word32 image_format; /* Image data format (see below) */
word32 flags; /* Format flags (see below) */
word32 num_blocks; /* Number of 512-byte blocks in this image */
word32 data_offset; /* File offset to start of data */
word32 data_len; /* Length of data in bytes */
word32 cmnt_offset; /* File offset to start of comment */
word32 cmnt_len; /* Length of comment in bytes */
word32 creator_offset; /* File offset to start of creator data */
word32 creator_len; /* Length of creator data in bytes */
word32 spare[4]; /* Spare words (pads header to 64 bytes) */
} image_header;

Format numbers:

0 = DOS 3.3-order dump of disk data. This is really only valid for
a floppy disk image.
1 = ProDOS-order dump of the disk data. This is the only valid format
for a hard disk image.
2 = Raw dump of the disk nibbles. This is only valid for a floppy disk
image.

Flags:

0x80000000 = Image is write protected (1 = yes, 0 = no)

I've done several things with this layout:

1. Made sure all the fields are naturally aligned. x86 doesn't care, but
some architectures do. I did this mainly by sticking to word32s, which
are quicker to deal with on many architectures anyway.

2. Extended the version to 16 bits and the image format and flags to 32 bits.
Might as well leave plenty of spares. :)

3. Added offset/length pairs for the file data, file comment, and creator-
specific data. An offset of zero means that information is not present
in the image. Thus comments need not consume 1k of header unless they
are present, and if they need more than 1k they can use that too. Same
goes for the creator-specific data.

I should be able to switch XGS over to the new format in version 0.48
(since 0.47 is almost out the door).

[Ethan Fischer]

In article <5bto2k$7...@elna.ethz.ch>,

Henrik 'Ratte' Gudat <GUD...@EZINFO.VMSMAIL.ETHZ.CH> wrote:
>

>char magic[4]; /* "wxyz"
>char creator[4]; /* "XGS!" or whatever emu
>^^^^^^^^^^^^^^^^^^
>word16 header; /* length of header
>^^^^^^^^^^^^^^
>byte version; /* Format version (currently 0x00) */
>byte pad1; /* padding (x86 compatibility)
>word32 num_blocks; /* Number of blocks in image
>^^^^^^
>byte protected; /* Nonzero if image is protected */
>byte imageType; /* 0=raw data, 1=nibble (whatever)
>^^^^^^^^^^^^^^^^^
>char name[32]; /* Name of this image
>char desc[1024]; /* Description of this image
>byte pad2[3]; /* padding (x86 compatibility)
>byte spare[8]; /* Spare bytes
>

How about variable length fields? Then the name could be longer than 32
characters and the description could contain (for instance) the Apple
Software Map (.asm) info. Also, the "spare" field could contain as much
(or as little) info as necessary. Then if specific emulators wanted to
put in customization info, they could put it in the spare field.
Something like this:

char magic[20]; /* "Apple II Disk Image" + 0x00 byte */

byte version; /* Format version (currently 0x00) */

word32 header; /* length of header */
word32 num_blocks; /* Number of blocks in image */

byte protected; /* Nonzero if image is protected */

byte image_type; /* 0=raw data, 1=nibble (whatever) */
word16 name_length; /* length of name */
word32 desc_length; /* length of description */
word32 spare_length; /* length of "spare" field */
char name[]; /* 0x00 delimited string */
char desc[]; /* 0x00 delimited string */
byte spare[]; /* allow for format extensions */

--
Ethan Fischer
all...@u.washington.edu
http://weber.u.washington.edu/~allanon

[Henrik 'Ratte' Gudat]

In <5bu7ba$b...@server1.mich.com> Joshua writes:

> Ok, here's my proposal. This is has all the same things as your proposal,
> plus a few enhancements:

[..]

Hi Joshua,

I think we have a deal here. I will add the format today to Fast Eddie - I hope
it'll make it into the upcoming beta update.

Thanks for your cooperation - I think we have just agreed upon an important
thing.

cu,
henrik

[Joshua M Thompson]

Henrik 'Ratte' Gudat <GUD...@EZINFO.VMSMAIL.ETHZ.CH> wrote:

: I think we have a deal here. I will add the format today to Fast Eddie - I hope

: it'll make it into the upcoming beta update.

: Thanks for your cooperation - I think we have just agreed upon an important
: thing.

It's my pleasure. I'll add this into XGS ASAP, plus a converter util to
convert the existing XGS images over.

[Kent Dickey]

In article <5bu7ba$b...@server1.mich.com>,

Joshua M Thompson <in...@mich.com> wrote:
>Ok, here's my proposal. This is has all the same things as your proposal,
>plus a few enhancements:
>

>typedef struct {
> char magic[4]; /* "2IMG" */
> char creator[4]; /* Creator signature */
> word16 header_len; /* Length of header in bytes k */
> word16 version; /* Image version */
> word32 image_format; /* Image data format (see below) */
> word32 flags; /* Format flags (see below) */
> word32 num_blocks; /* Number of 512-byte blocks in this image */
> word32 data_offset; /* File offset to start of data */
> word32 data_len; /* Length of data in bytes */
> word32 cmnt_offset; /* File offset to start of comment */
> word32 cmnt_len; /* Length of comment in bytes */
> word32 creator_offset; /* File offset to start of creator data */
> word32 creator_len; /* Length of creator data in bytes */
> word32 spare[4]; /* Spare words (pads header to 64 bytes) */
>} image_header;

Sorry, I'm joining this party late.

First, let's define some terms. "raw image" = series of bytes, no header.
A "raw 5.25" disk = 35*16*256 = 143360 bytes. A "Raw 3.5" disk =
1600*512 = 819200 bytes. A "Nibblized image" is a file containing the
low-level disk bytes (d5,aa,96 type of stuff) in an as-yet-undetermined
structure.

I really like simplicity. Rather than assuming a header is required,
I want to hear some arguments saying why it is needed. Raw 5.25" formats
have been very successful. I must be missing the uproar that is demanding
to add comments to files.

My suggestion is to make the 800K disk default format be a raw
image with no extra info at all. Users will have the easiest
time creating and dealing with these images.

Who am I: I've contributed the 5.25" read/write code to XGS, written my
own GS simulator (HP workstations only, not available for PCs), and have
been using Apples for years. I wrote the compression algorithm in
Shrinkit 10 years ago because I wanted Wishbringer to compress to fit on
one disk, and do it faster than Dalton's Disk Disintegrator. But, I
hate user-interface stuff, so I gave my code to Andy Nicholas, who
was writing Shrinkit and looking for compression code. He cleaned it
up, and added 100X more code to create Shrinkit.

Andy Nicholas thought all this whizzy file format stuff would be useful
in Shrinkit. But who ever used comments? Who ever cared what was in an
archive other than the list of files? Who's ever used comments in GSOS
files? And now, with NuFX being so complex, it's discarded out of hand
as a file format. But then we go and invent another one! I recommend
NuFX with uncompressed disk images if you feel such a strong desire for
headers, variability, etc. I don't know NuFX very well, but just a
one-file NuFX archive can't be that complex. And without compression,
as soon as you've located the data, you can just deal with it like it
was a raw file. Reading from a compressed disk image would be OK as
long as you re-wrote the archive as an uncompressed image. Checksums
might be a problem (Andy liked CRC's, which is a real pain to
recalculate partials). Or just treat the compressed image as
read-only. Most people wouldn't care.

To be fair to Andy, I think he did NuFX as a university project, so at
least he got college credit for it...I wish I'd thought of that angle.

So, my suggestion is:

5.25": the .dsk or .po raw formats available today.
3.5": raw format, as a series of ProDOS blocks.

For nibblized images, 5.25" has a lame format already: .nib.
I recommend creating a new format, .nb2, with some enhancements,
but still kept very simple.

I also suggest we look into a naming scheme to encode useful
info about a disk in the name. For instance, 5.25" Time Zone exists as
.nib disks just so that each disk could have a unique volume number.
Ignoring Windows (and I always do), a convention where emulators
strip off extensions from filenames would let us encode useful info
in the filename that the emulator could re-create.

For instance, timezone_a.nib could become tzone_a.vol=254.dsk,
and disk B could be tzone_a.vol=253.dsk.

DOS 3.2 could be supported by having a raw image format of
35*13*256 = 116480 bytes, and a ".d32" extenstion. Why mess
about with .nib format when it is not required?

"Write-protecting" an image could be done in a platform-specific
way. On Unix, chmod -w. On Windows-like platforms: ATTRIB -whatever.
On the Mac, setting the file to be locked. If someone really
hates this, put it in the filename. ".lck" in the extenstion
area could mean read-only. Useful for the Wizardry boot disk.

For compatibility, all emulators should detect that a file that is
exactly 128 bytes too long probably has a macbinary header, and just
skip the first 128 bytes. If the file has the extension ".bin",
then definitely skip the first 128 bytes.

I recommend emulators at least be able to read from ShrinkIt compressed
disk archives, especially since many 3.5" disks are already in this
format. Read-only access would be OK.

Extensions would be "applied" to the file in right-to-left order.
So, if you can handle gzipped files, a foo.dsk.shk.gz.bin would
mean you would un-macbin it, un-gzip it, then un-shrinkit it
to get at the data. If you're emulator did not support one of
these steps, just print a message saying so.

I propose that 800K images end in ".a35" as a quickie indicator of
the size. ".dsk" might also be OK, but then a plain list of files
would make it hard to tell 3.5" from 5.25" disks. ".800" is also
good. Heck, support them all, but generate just one.

I recommend all emulators recognize ".po" 5.25" images as being in
ProDOS order, and automatically re-arrange the sectors.

So, foo.po.lck.bin would be a 140K 5.25" image with a Mac binary
header in ProDOS order that was read-only.

Putting "hints" in the filenames gives us lots of room for expansion
in the future without locking us into anything today.
If an emulator chooses not to support a hint, that is fine.
It lets emulators store any extra info (if they find it useful) in the
filename and still be compatible with everyone else.

As for Nibblized images, I'd like to see that format as just a stream of
track data. Each track would have a length indication at the start, then
qtr_track number, followed by the encoded track data. I'm a stickler
for details, so the format must be able to differentiate between 9 & 10
bit sync bytes, and be able to recreate any possible 5.25"/3.5" disk.
An image can skip tracks, or use tracks just 0.5 tracks apart on
5.25" drives. An emulator must be able to handle different track
lengths (within a certain tolerance, say 195ms - 205ms Apple II 5.25"
equivalent--I'll calculate the bytes later, and a similar range
for 3.5").

Emulators would tell nibblized images from raw data images by
the .nb2 extension (.nib is taken by a lame format already..anything
that works with .nib could have been cracked pretty simply, except
for Broderbund's damn 18-sector format, but there are cracks
of all those games, boy can I digress). Other meaningful extensions
would continue to apply, like .bin, .lck, etc.

So, remove the header, and put it in the filename.

Now, let me attack headers directly. I wanted to transfer some raw
images to my Mac to run on Fast Eddie. I knew Disk Copy format would
work, so I got some documentation (sparse, very sparse) and wrote
a converter. Problem: A field in the header called "tag_checksum"
was not described. I searched the net for over an hour. It turns
out that everybody just seems to stick a 0 in it. Sigh.
BUT, people do use the silly data checksum, which makes re-writing
a portion of the image a pain (since you have to calculate the
difference in the checksum).

So how many fields will really be used in this header? And can
you handle all the cases? What if I put the comment after the
data in the file (or vice-versa)? Will you support that?
What about if someone creates a format that has the data not
word aligned?

We should create a list of all the possible extensions, and
make recommendations to emulators as to what formats they should
support. Like .hqx is probably not required (but nice!).
But .bnsc is less clear whether it should be required.

Kent Dickey
ke...@cup.hp.com

[Henrik 'Ratte' Gudat]

Kent, I'm glad you joined the party :-). We have been talking about this in
e-mail a little bit, but I'd like to take the opportuniy and outline with a few
words why I believe there should be a header-ed format.

First, you know already that IMO the existing disk images should not simply
ignored. The user should be to use those files he/she downloads from the
archives _today_.

The problem I'm having with packing all the access attributes into the filename
or other places where they do not belong (at least on the Mac) is that this is
not what a
Mac user is expecting. Whatever disk image we are discussing, it should respect
the conventions of the platforms it will used on.
For example, having the disk type in the filename is not a very good idea on
the Mac. I agree that this is used a lot (for example with compacted files),
but the Mac has a Type field in the catalog for that. Similarly, when the user
is locking the file, he/she is expecting the file to be locked for the next
session. Locking the file via the MacOS is not a good approach because the lock
flag should not apply on the plain file in the Finder. Therefore, we need a
flag for locked images.

The same reasoning applies for the other stuff. It's the only way I can think
of that makes sense on any platform. It would allow anyone to move a disk image
from, say, a HP workstation :-) to XGS or Eddie and continue where he/she left
off - without having to reconfigure a disk.

So, basically the question is header or no header. if the header does also have
a comment field (I don't think it's crucial, but since we;'re right at it, why
not) is not the real problem here.

I will definitely add functions for converting between headerless images and
headfull ones.

cu,
henrik

Fast Eddie Labs

[Joshua M Thompson]

Headers do have some uses, even really silly ones like being able to run
the Unix "file" command on a mysterious file and find out it's a disk
image. I also happen to like having the image name in the header along
with some optional comments, especially if you deal with MS-DOS short
filenames (like it or not, there's a LOT of people out there that have
Windows machines).

The naming convention is a nice idea, but unless you have very liberal
filename conventions, it won't work. Again I stress Windows, which may
be an abomination but it's also a large chunk of the potential emulator
users. In fact I happen to like using XGS on Windows 95 now more than
on Linux; the Windows version is a LOT faster because VC++ actually
knows how to optimize for a Pentium.

Now, this doesn't mean an emulator can't have a support for the raw image
format as well. I do plan on having XGS natively read .DSK, .PO, .DO, and
.NIB files by examining the filename extension and making an intelligent
decision on the image format (unless of course it's a 2IMG file, which
can be determined from the header..tada :) ).

[Joshua M Thompson]

Also Henrik, before I forget, it was suggested to me in email to fix the
order of the file to simply things:

header
disk data
creator data
comments

Since the first three are generally fixed in size and the last one can
grow and shrink, this layout makes it easy to modify the file in-place.
It sort of makes the offset+length pairs redundant (since you know the
order, you could get by with just lenghts), but we'll keep them just in
case we want to add stuff in the future.

[Nathan Mates]

In article <5c2sel$p...@elna.ethz.ch>,

Henrik 'Ratte' Gudat <GUD...@EZINFO.VMSMAIL.ETHZ.CH> wrote:

>First, you know already that IMO the existing disk images should not simply
>ignored. The user should be to use those files he/she downloads from the
>archives _today_.

And all the GS software sites in existance use .shk or .bsq. So
you should be supporting it *first* not some afterthought.

Yes, it's a little more work to be backwards compatible. If you're
not going to deal with it, what's next? Linearizing the hires screen
just because that every 9 lines deal is too complicated? Unless you're
dealing with a proper superset of older things, backwards compatible
*always* requires work.

[Joshua M Thompson]

Nathan Mates <nat...@visi.com> wrote:

: Yes, it's a little more work to be backwards compatible. If you're

: not going to deal with it, what's next? Linearizing the hires screen
: just because that every 9 lines deal is too complicated? Unless you're
: dealing with a proper superset of older things, backwards compatible
: *always* requires work.

Hey, while working on XGS I had _lots_ of cool ideas for ways to improve
the "hardware." Maybe it's time for an emulated "GSX". :)

[matthew p. conte]

| Yes, it's a little more work to be backwards compatible. If you're
| not going to deal with it, what's next? Linearizing the hires screen
| just because that every 9 lines deal is too complicated? Unless you're
| dealing with a proper superset of older things, backwards compatible
| *always* requires work.

nate, i don't see how the apple II community will be hurt by this... most
of the people who use the emulators (in my estimation, i could be way off)
are either prior apple2 users, or current apple2 users... that means that
they either have the software for their apple2's, or have no use for it
anymore because they don't have the machines... so if they have the
software in a format that is exclusive to the emulators, what's bad about
that?

and i really highly doubt programmers are going to start making GS
utilities/games on the emulators, just to distribute them to the public in
XGS format, etc... so the apple2 community isn't suffering...

besides, to get a .SHK file to an actual prodos disk, you go through a
bunch of steps, and the same with the .XGS files... its probably easier
with .XGS files because there's no compression... i can take an .xgs
image, strip the header off it, and use nulib with the "cd" switch to make
a disk image...

hey, that's what they make conversion utilities for anyway...

late,
matt

[Henrik 'Ratte' Gudat]

In <5c31k0$1...@server1.mich.com> Joshua writes:

> header
> disk data
> creator data
> comments

Hi Joshua & everyone else,

yeah, that makes sense. let's keep it that way.

- henrik

Fast Eddie Labs