Hard-sectored disk image format.

[Mark Garlanger]

Sorry to start up the disk image format discussion again, but was talking with John, the creator of the Applesauce tool/software and he mentioned the need to have something better than H8D to help preserve the volume number and other things like disk interleave.

  We discussed the alternate format I had come up with a few years back - h17disk, but there is a bunch of complexity which is not needed for many use cases. We came up with some changes that would hopefully make it much easier for programs currently using H8D to switch over to the new format, with minimal changes. The file will have a block that is identical to the raw data in the H8D file. In the new H17Disk file, the H8D data will start at offset 256 and the size of H8D data can be read at offset 252. Only programs/apps that want to know about any of the additional information like header/checksum data, interleave, etc. need to look at other parts of the file.

  I'm trying to flesh out the rest of the details for the format, if anyone wants to help with the design, let me know. 

Mark

[Glenn Roberts]

This would really help Mark. Thanks

Sent from my iPad

On Aug 20, 2022, at 6:09 PM, Mark Garlanger <garl...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/CAAjkm79A2M9nwg06AuC9GFcjeK3%3D8%2BXkQdE6FNxLczQAA__gTg%40mail.gmail.com.

[dwight]

When I first started, all the disk were interleaved 1 to 1. This was the worst timing for HDOS. HDOS always skips the next sector so it required an additional revolution of the disk to read sequential sectors ( typical reads ).

Basic needed more time between sectors than raw reads of data, for processing of the data. I forget what was optimal.

Being able to keep or adjust the interleaving. I believe having the ability to keep the original interleaving or adjust the interleaving is desired. 

Faster processors like the Z80 can take advantage of more optimal interleaving.

As I noted above, 1 to 1 interleaving will always be the worst, regardless of processor speed because of the required skipping of the next sequential sector, built into HDOS.

One might even want to adjust interleaving of tracks based on the data used. Having the interleaving of a track set for the aggressive read rate of skipping only one sector for program data would actually be worse for reading basic data files than the original 1:1 sectors, depending on speed of the processor.

Then, one has the possibility of the processor adjusting interleaving on the fly to self optimize data availability, based on write or access time.

Keeping the original interleaving is desired as part of the record. I'm not sure of the need to keep such information as checksum unless someone was using a broken checksum for security purposes. Checksums are useful for recovering corrupted data though. I recall doing such on damaged tapes that were used for my Poly88. In that case, having the raw data bits is more important than the meaning of what the reading program thinks they mean. Things like bit sequence can be missing but one can still recover the original intent of things like programs. Things like data, with no redundant information ( like programs or text ) may be impossible to recover. For this purpose, being able to read the data bits from a track may be the most important thing ( thinking of a drive with poor speed control ).

Dwight 

 

---

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of Mark Garlanger <garl...@gmail.com>
Sent: Saturday, August 20, 2022 3:09 PM
To: se...@googlegroups.com <se...@googlegroups.com>
Subject: [sebhc] Hard-sectored disk image format.

 

[Mark Garlanger]

Attached is an updated spec for h17disk format ver. 2. Support in Applesauce is currently being tested with support for this new format.

If anyone has any comments on it, let me know.

Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/BYAPR01MB531911A273E084663A756094A36E9%40BYAPR01MB5319.prod.exchangelabs.com.

[glenn.f...@gmail.com]

A lot of work Mark. Thanks for taking this on.

 

A few suggestions:

 

The Overview section would be more balanced if it explained exactly when the .h17disk format is needed, vs the H8D.  H8D has many good practical uses and we already have hundreds of disks in this format. It meets the needs of many uses including the H89LDR program, Les Bird’s H89 emulator, and the JUKEBOX.  Just having the H8D information often serves the purpose.  But it *does* throw out information that would be useful or important from an archival perspective or if we ever wanted to exactly reproduce a disk.  I don’t know if any manufacturers used disk copy protection but that might also be something that could be captured by .h17disk.  I don’t foresee us converting all the existing H8Ds to h17disks so the two will probably coexist for some time. It makes sense, where possible, to capture any new disks we find in the more complete h17disk format.

 

I am concerned that the current format is not robust enough in the face of bit errors.  It’s true we’ve come to think of disk storage as highly reliable, but if a bit or byte is lost I’m concerned about how to detect (or even correct).  If this is truly an archival format then we should account for that.  I suggest adding a CRC to each block.  The H8/H89 even already have a very good CRC routine in the ROM (routine at 0x02E7; checksum maintained at 0x2017).

 

A few other nits I highlighted in my copy

 

• Line 2: “this document documents” should be “this document specifies”
• Many words unnecessarily capitalized (e.g. Hard-sectored Disk Images).
• Line 24: ascii à ASCII
• Line 26: “should set this field to the version of the specification to which it complies”
• Line 32: “The next 4 bytes of the block specify the length…”
• Line 31: “4-byte identifier”.  In this case “4-byte” is an adjective so must be hyphenated.  This also occurs on lines 4, 15, and probably elsewhere.
• Line 45: states that the parameters block is an “optional block”  but the next line states that “the block is mandatory”
• Line 78: so the H8D data block must *always* start at file offset 256?  The idea is you can just skip the first sector and read the H8D data?  Does that allow enough space for the preceding blocks?

 

Good work!

 

Glenn

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/CAAjkm79_TTOVDTAYWXVL9frdQRXVqc33f0AOkeurXU%2Bzc%2BO4SQ%40mail.gmail.com.

[dwight]

CRC is always a good idea. A 32 bit CRC can usually handle an error burst of 12 bits in length. If one keeps the one sector for each CRC, the correctable life of the files will be more robust. A CRC of the entire disk is not as useful.

I should note I repaired as many as 3 bit errors on one block of cassette tape with only a check sum. These were often more than 12 bits apart. In some ways it was relatively easy as the bits missed a clock and the rest of the bits were not properly aligned. Shifting the bits left gaps for the unknown bit, making it easier to fix.

Do realize that if the errors are greater than the 12 bits apart, even a CRC won't be able to repair the data ( assuming a typical 32bit CRC). Having multiple repositories is essential.

Dwight

---

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of glenn.f...@gmail.com <glenn.f...@gmail.com>
Sent: Saturday, August 27, 2022 6:28 PM
To: se...@googlegroups.com <se...@googlegroups.com>
Subject: RE: [sebhc] Hard-sectored disk image format.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/02a601d8ba7d%2478f5e990%246ae1bcb0%24%40gmail.com.

[Mark Garlanger]

Thanks for the review, I'll try to address the comments and get another version posted.

As for the CRC, it doesn't seem like it is really needed here. The check that is in the H89 ROM, is not technically a CRC calculation, it is just a basic rolling 8-bit XOR checksum which is quite weak. But for any imaged disk in this format, the checksum from the disk will be saved and can be checked by a program if they want. If an H8D file is converted, then the only protection is after the file was converted and the checksums added. As for any changes/damages to the file while it is stored, those will be detected/reported by any modern system. There would be read errors reported, because the harddrive/ssd drive checksums would not match. The key to having a good copy, is to have redundancy in storage, (online/multiple local copies/ multiple people with copies, etc).

For the start of the H8D data, yes it will always start at 256. The blocks do not have to be in order, if a blocks can fit before the H8D block, they can be there, if not, they will be stored after the H8D block. There is also the padding block to ensure the H8D will start at 256. It was chosen to have it at this fixed location, so that it would be trivial for any program currently using H8Ds, to be converted to the new format without having to be modified to understand the whole file or even the tagged blocks. The disk param block is mandatory, and must be the first block after the File Header. That way a current program can look at byte 17 to know how many sides the image is, and byte 18 to know the number of tracks, it would then extract the H8D starting at byte 256, and read (256 * 10) * <sides> * <track> bytes.  If they want to use any of the additional information in the file, they can be enhanced to do that, but this allows the current program to act like they are still using H8D images.

The params block is not going to be mandatory, and a bunch of that first paragraph is left over from the v1 h17disk document. I'll clean that up.

Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/SA1PR11MB6941D4CBCF18A33A56E6D08DA3779%40SA1PR11MB6941.namprd11.prod.outlook.com.

[Glenn Roberts]

Thanks for the clarification Mark. Great work 

Sent from my iPad

On Aug 28, 2022, at 2:21 AM, Mark Garlanger <garl...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/CAAjkm78jsxKypk0cFz8JODHGwbJSX096hroW5E%3DcjoZD%3Dgx%2BDQ%40mail.gmail.com.

[dwight]

Well thought out Mark.

Dwight

---

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of Glenn Roberts <glenn.f...@gmail.com>
Sent: Sunday, August 28, 2022 2:35 AM
To: se...@googlegroups.com <se...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/8FD417F8-A9FA-483B-97A9-FEC1AAF209D0%40gmail.com.

[Mark Garlanger]

Thanks, and to clarify, h17disk version 2 is not meant to be an archival format. The first version of h17disk tried to do everything, including archival by saving all the data returned by the FC5025. Due to the complexities, there was no support for it other than the disk imaging software for the FC5025 that I wrote to handle Heath hard-sectored disks. After talking with John from Applesauce, we saw the need for a much simpler format that could be used in place of the H8D, but which saves all the data from the disk. The key goal for this revision was to make it as simple as possible for existing programs to move to the new format. Having the H8D data, number of sides, and number of tracks all at a fixed offset, should make it trivial for any program that currently supports H8D to add support for the new format.

  Applesauce already has an archival format A2R  - https://applesaucefdc.com/a2r/ for disks imaged with that hardware. For archival when imaging with the FC5025, maybe h17disk version 1, can be changed to something like h17archive and I can write some code to convert it into the new version 2 format, but keep the disk imaging software storing to the original format. 

Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/SA1PR11MB6941D59E342EBB27871D976CA3779%40SA1PR11MB6941.namprd11.prod.outlook.com.