H8D Utility 3: Reading CP/M files from disk image

[Les Bird]

CP/M experts (Darrell?),

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

[Douglas Miller]

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

[glenn.f...@gmail.com]

My favorite reference for CP/M internals is Andy Johnson-Laird’s “The Programmer’s CP/M Handbook.”  Includes ASM and C code examples.  Well worth studying…   I have the paper copy but found this on line:

 

http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

 

 

 

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Douglas Miller
Sent: Friday, September 10, 2021 12:43 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

 

On 9/10/21 11:25 AM, Les Bird wrote:

CP/M experts (Darrell?),

 

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

 

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/1df4846f-2419-a3dc-a69b-691498faefb1%40gmail.com.

[Steven Hirsch]

On 9/10/21 12:49 PM, glenn.f...@gmail.com wrote:
> My favorite reference for CP/M internals is Andy Johnson-Laird’s “The
> Programmer’s CP/M Handbook.” Includes ASM and C code examples. Well worth
> studying… I have the paper copy but found this on line:
>
>
> http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

I'll second and third this. He explains the CP/M filesystem in great depth.

[Douglas Miller]

It is also worth noting that reading files is "relatively" simple, but writing (adding) files to a disk is a whole other effort, and requires great care as not to corrupt the disk/filesystem.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/016b01d7a663%24dfc6a670%249f53f350%24%40gmail.com.

[Les Bird]

Thanks guys, very helpful.

Glenn, that book is excellent. Great detail on how the file system works.

Les

[glenn.f...@gmail.com]

Les: if it’s of any use, my “VPIP” utility builds a copy of the CP/M directory in RAM.  I used Johnson-Laird’s book as a reference.  VPIP source attached plus some test jig code…  see routine bldcdir() .  Only accessing the directory, not the file contents themselves,  but may be some useful skeleton code in here for you.

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Les Bird
Sent: Friday, September 10, 2021 4:09 PM
To: SEBHC <se...@googlegroups.com>
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

Thanks guys, very helpful.

 

Glenn, that book is excellent. Great detail on how the file system works.

 

Les

 

On Friday, September 10, 2021 at 4:03:25 PM UTC-4 Douglas Miller wrote:

It is also worth noting that reading files is "relatively" simple, but writing (adding) files to a disk is a whole other effort, and requires great care as not to corrupt the disk/filesystem.

On 9/10/21 11:49 AM, glenn.f...@gmail.com wrote:

My favorite reference for CP/M internals is Andy Johnson-Laird’s “The Programmer’s CP/M Handbook.”  Includes ASM and C code examples.  Well worth studying…   I have the paper copy but found this on line:

 

http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

 

 

 

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Douglas Miller
Sent: Friday, September 10, 2021 12:43 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

 

On 9/10/21 11:25 AM, Les Bird wrote:

CP/M experts (Darrell?),

 

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

 

--

You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/1df4846f-2419-a3dc-a69b-691498faefb1%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/016b01d7a663%24dfc6a670%249f53f350%24%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/a2cba384-951e-4a78-a67f-5f640c24fd9dn%40googlegroups.com.

[s shumaker]

I'm getting errors on that link but Archive.Org also has a copy.

Steve

[glenn.f...@gmail.com]

Click on the link in my original email, not the reply (the URL got truncated/"folded").  Here's the direct link:

The Programmer's CP/M Handbook

 

-----Original Message-----
From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of s shumaker
Sent: Friday, September 10, 2021 5:06 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

I'm getting errors on that link but Archive.Org also has a copy.

--

You received this message because you are subscribed to the Google Groups "SEBHC" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/df97b1f4-b4f6-7820-b1d5-17770870d6af%40att.net.

[norberto.collado koyado.com]

Is this for CP/M on the H17 floppy controller? Just curious...

---

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of glenn.f...@gmail.com <glenn.f...@gmail.com>
Sent: Friday, September 10, 2021 2:31 PM
To: se...@googlegroups.com <se...@googlegroups.com>
Subject: RE: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/01b101d7a68b%243fa08b70%24bee1a250%24%40gmail.com.

[glenn.f...@gmail.com]

More general. It’s about understanding how CP/M directory structure works in general, regardless of the media format.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/SN6PR01MB3855728FF5D5D771CAC0D0D7F7D69%40SN6PR01MB3855.prod.exchangelabs.com.

[norberto.collado koyado.com]

I'm a "RAW" guy, so I only think about sectors and have the HW put it in the right place; read sector x, write sector x to duplicate the media.

---

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of glenn.f...@gmail.com <glenn.f...@gmail.com>

Sent: Friday, September 10, 2021 2:59 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/025b01d7a68f%242d51d1f0%2487f575d0%24%40gmail.com.

[Darrell Pelan]

Hi Les,

As other's mentioned the Programmer's CP/M  Handbook is an absolute must. It took a lot of reading, thinking, and rereading to understand the details. 

My code for reading and writing CP/M files is in the H8DUtility updates I made in version 2.2. An improved version is in the Disk Image Utility I wrote that supports CP/M IMG files for Flash Floppy and MS-DOS disks for Z-100's. The code is at Disk Image Utility on Github. The utility was recently posted on the webpage SEBHC Software

Let me know how I can help.

  Darrell

[Darrell Pelan]

The definitions I used in the code are below. Depending on the disk size, the directory starts in a different place. The first comment below refers to BIOS parameters to define the disk type. The code is my table of parameters used by the code to determine where to look. The disk images from Dave Dunfield's IMD are interleaved. The IMG files used by Flash Floppy are not interleaved. The H-17 images (100k) do not have a disk identifier ( the value could be E5 or 00).

 /*

        Disk type: byte 5 in sector 0 on H-37 disks (starting from 0) to define disk parameters

        Allocation Block size: number of bytes in an the smallest block used by CP/M on the disk. must be a multiple of 128 (0x80)

                AB numbers start with 0. The directory starts in AB 0.

        Directory Stat: start of directory entries in bytes

        Allocation Block Number Size: number of bytes used in directory entry to reference an allocation block

        Dir Size: number of bytes used for the directory

         0040 =         DPEH17 EQU 01000000B

         0060 =         DPEH37 EQU 01100000B

         0008 =         DPE96T EQU 00001000B

         0004 =         DPEED EQU 00000100B

         0002 =         DPEDD EQU 00000010B

         0001 =         DPE2S EQU 00000001B

        */

        // 0.Disk type, 1.Allocation block size, 2.Directory start, 3.Allocation block byte size, 4.dir size, 5.interleave, 6.Sectors per Track, 7.Sector Size,

        // 8.# Tracks, 9. # heads, 10 Skew Start Sector (0 based)

        public int[,] DiskType =

        {

            // 0        1          2         3       4        5    6       7       8   9  10     position in table 0..10

            {0xff, 0x800, 0x4800, 2, 0x2000, 6, 18, 0x200, 80, 2, 10}, // 0 1.44 MB Small Z-80 

            {0x6f, 0x800, 0x2800, 2, 0x2000, 3, 5, 0x400, 80, 2, 4}, //   1 800k H37 96tpi ED DS

            {0x6b, 0x800, 0x2000, 2, 0x2000, 3, 16, 0x100, 80, 2, 12}, // 2 640k H37 96tpi DD DS

            {0x67, 0x800, 0x2800, 1, 0x2000, 3, 5, 0x400, 40, 2, 4}, //   3 400k H37 48tpi ED DS

            {0x23, 0x800, 0x2000, 1, 0x2000, 1, 8, 0x200, 40, 2, 0}, //   4 320k Z100 48tpi DD DS

            {0x62, 0x400, 0x1000, 1, 0x2000, 3, 16, 0x100, 40, 1, 12}, // 5 160k H37 48tpi DD SS

            {0x63, 0x800, 0x2000, 1, 0x2000, 3, 16, 0x100, 40, 2, 4}, //   6 320k H37 48tpi DD DS

            {0x60, 0x400, 0x1e00, 1, 0x800, 3, 10, 0x100, 40, 1, 3}, //   7 100k H37 48tpi DD SS

            {0xE5, 0x400, 0x1e00, 1, 0x800, 4, 10, 0x100, 40, 1, 0}, //   8 100k Default H17 48tpi SD SS

            {0x00, 0x400, 0x1e00, 1, 0x800, 4, 10, 0x100, 40, 1, 0}, //   9 100k Default H17 48tpi SD SS

        };

  Darrell

On Friday, September 10, 2021 at 12:25:01 PM UTC-4 Les Bird wrote:

[Mark Garlanger]

Hey Les,

     I also have some code here - https://github.com/mgarlanger/heath-imager/blob/master/src/libs/cpm.cpp for reading CP/M, that seems to be working well. It handles 100k, 200k, and 400k images. Although the 200k hasn't been tested well.

Mark

--

You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/c79657c2-d2cf-4a84-be2d-79b82ebf8250n%40googlegroups.com.

[Les Bird]

Darrell, Mark, thanks a lot guys. I should have enough info now to do what I need.

Les

[Darrell Pelan]

Les,

Glad to help.

  Darrell

[Les Bird]

Darrell,

Not sure if you've been keeping up with H8DUtility3 (https://github.com/lesbird/H8DUtility3/blob/main/README.md) but I'm making lots of progress with it. I solved the CP/M file extraction issue I was having. Your code helped a lot. The problem was I was not taking into account the sector skewing. Once I got that figured out I was able to read and extract CP/M files.

I got the H37 disk imager working for reading disk images but writing is not working yet. No support yet on the Heathkit side for this but I have started the process. So far the disk imager will read:

1. H37 HDOS/CPM format (single/double density, 40/80 track, 1/2 sides)

2. PC MSDOS format (single/double, 40/80 track, 1/2 sides)

3. Zenith Z100 all types

It will not read PC high density disks - not supported by WD1797.

So now the problem is - do I add MSDOS directory cataloging, file extraction, etc for the PC/Z100 formats? I think I need to. Do you know anything about the MSDOS file format and is it the same format for the Z100 series computers (ZDOS)?

Also, we will need to support IMD convert for the .H37 format (not in yet) and the ability to insert files into a disk image (not in yet).

Les

[Darrell Pelan]

Hi Les,

Does the .h37 format keep the original sector skewing? If so, you will need to account for that using a function like buildSkew() in my code. I'm currently in Bonaire and can provide more detail on Thursday, if  needed. I decided to use .img format in my app since I was focused on the Flash Floppy Gotek. The .img format is not skewed which makes coding easier. The HxC Gotek doesn't care about skew, so the conversion function just stores it in logical order when writing an IMD file.

My Disk Image Utility program supports MS-DOS FAT 12 disks, which is enough to support floppy disks. I don't think FAT 16 was used on floppies. The code supports reading, extracting, and writing files to the image. I didn't spend the time to write delete code or View for MS-DOS yet. Z-DOS uses FAT 12, so the code supports it. 

There is also code to support writing .imd files, though it doesn't account for disk skew. If the .h37 format maintains the physical disk skew, you will need to modify the code to list the sectors in the correct order when writing the IMD track header information. The current code just writes the sectors to the .img file in the same order as the source file, assuming logical order.

   Darrell

You received this message because you are subscribed to a topic in the Google Groups "SEBHC" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sebhc/eefVyCPeSbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/2eb38df4-7555-46cd-8dff-7f2b9e7f18fdn%40googlegroups.com.

[Les Bird]

Darrell,

H37 does not retain skewing. It's just a sector by sector dump in logical order, track by track. The data appended to the end contains sectors per track, sector size, number of tracks, number of sides and single/double density so we know how to reconstruct the disk.

As Douglas mentioned in another post trying to extract skewing data would be a bit complex.

Les

[Darrell Pelan]

So if you want to extract data from the disk you will need to depend on the data in byte 5 for CP/M disks. The ZDOS disks are not skewed.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/bbf1bc60-a26b-4333-9f22-0a34c9900d26n%40googlegroups.com.

[Douglas Miller]

Note, the H37 skew was *physical* skew, the sectors were formatted on the track in an order that optimized sequential access. For physical skew, you don't need any special handling to access the data, assuming the data is organized in numerical sector order (1,2,3...). The issue for the images is that you don't have the skew information to reformat a new diskette that is identical to the original. The data will still be the same, but it may not perform as well.

The H17 disks had a *logical* skew, at least for some OSes, and that does require knowledge of in order to access the data. It is also possible to format H17 diskettes using a physical skew (on top of the logical skew), which complicates things a lot when trying to predict performance.

To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/CAKhgrKs2anDXMc%3DUXFnFC6h8PtLK4aKSwHip0ZLtDzUddxJD-Q%40mail.gmail.com.