H8D Utility 3: Reading CP/M files from disk image

71 views
Skip to first unread message

Les Bird

unread,
Sep 10, 2021, 12:25:01 PM9/10/21
to SEBHC
CP/M experts (Darrell?),

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

H8DUtility3.png

Douglas Miller

unread,
Sep 10, 2021, 12:42:49 PM9/10/21
to se...@googlegroups.com

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

glenn.f...@gmail.com

unread,
Sep 10, 2021, 12:49:55 PM9/10/21
to se...@googlegroups.com

My favorite reference for CP/M internals is Andy Johnson-Laird’s “The Programmer’s CP/M Handbook.”  Includes ASM and C code examples.  Well worth studying…   I have the paper copy but found this on line:

 

http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

 

 

 

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Douglas Miller
Sent: Friday, September 10, 2021 12:43 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

 

On 9/10/21 11:25 AM, Les Bird wrote:

CP/M experts (Darrell?),

 

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

 

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

Steven Hirsch

unread,
Sep 10, 2021, 3:51:43 PM9/10/21
to se...@googlegroups.com
On 9/10/21 12:49 PM, glenn.f...@gmail.com wrote:
> My favorite reference for CP/M internals is Andy Johnson-Laird’s “The
> Programmer’s CP/M Handbook.” Includes ASM and C code examples. Well worth
> studying… I have the paper copy but found this on line:
>
>
> http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

I'll second and third this. He explains the CP/M filesystem in great depth.

Douglas Miller

unread,
Sep 10, 2021, 4:03:25 PM9/10/21
to se...@googlegroups.com

It is also worth noting that reading files is "relatively" simple, but writing (adding) files to a disk is a whole other effort, and requires great care as not to corrupt the disk/filesystem.

Les Bird

unread,
Sep 10, 2021, 4:08:39 PM9/10/21
to SEBHC
Thanks guys, very helpful.

Glenn, that book is excellent. Great detail on how the file system works.

Les

glenn.f...@gmail.com

unread,
Sep 10, 2021, 4:54:37 PM9/10/21
to se...@googlegroups.com

Les: if it’s of any use, my “VPIP” utility builds a copy of the CP/M directory in RAM.  I used Johnson-Laird’s book as a reference.  VPIP source attached plus some test jig code…  see routine bldcdir() .  Only accessing the directory, not the file contents themselves,  but may be some useful skeleton code in here for you.

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Les Bird
Sent: Friday, September 10, 2021 4:09 PM
To: SEBHC <se...@googlegroups.com>
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

Thanks guys, very helpful.

 

Glenn, that book is excellent. Great detail on how the file system works.

 

Les

 

On Friday, September 10, 2021 at 4:03:25 PM UTC-4 Douglas Miller wrote:

It is also worth noting that reading files is "relatively" simple, but writing (adding) files to a disk is a whole other effort, and requires great care as not to corrupt the disk/filesystem.

On 9/10/21 11:49 AM, glenn.f...@gmail.com wrote:

My favorite reference for CP/M internals is Andy Johnson-Laird’s “The Programmer’s CP/M Handbook.”  Includes ASM and C code examples.  Well worth studying…   I have the paper copy but found this on line:

 

http://oldcomputers.dyndns.org/public/pub/manuals/a_programmers_cpm_handbook_(1983).pdf

 

 

 

 

From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of Douglas Miller
Sent: Friday, September 10, 2021 12:43 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

It's not trivial, but the allocation map enumerates the logical blocks. Also, each file may have multiple directory entries, where the extent tells you what order the data (allocation blocks) are in. This all is predicated on the Disk Parameter Block (DPB) for the image - which is not always stored in the image itself. I think all Heath formats have a DPB in sector 0, but other formats (e.g. MMS) do not. The RC byte in the directory entry tells you how many "records" (128 byte) are contained in the *last* extent. Complicating things, each directory entry may contain more than one extent - depending on the DPB.

I'm not sure the best place to get the information you need. CP/M "system alteration guide" usually has a lot of information, as does the "programmers guide". There is also example C code in the cpmtools project, although that is not always easy to read.

One source for cpmtools code is https://github.com/lipro-cpm4l/cpmtools

This all is just a starting point. Maybe look into it and then more-specific questions will arise.

 

On 9/10/21 11:25 AM, Les Bird wrote:

CP/M experts (Darrell?),

 

I'm having a hard time understanding the CP/M disk structure. I'm working on H8DUtility 3 again and added viewing of HDOS files from the app (see screen shot) but for the life of me I cannot seem to figure out how CP/M works. I've searched and searched but can't find examples of how to get to the file data from a CP/M directory entry (the CP/M directory structure is discussed a lot). The directory entry gives me "extent", "sectors" and an allocation map (16 bytes) but I have no idea how to use that data to get me to where the data is on the disk image. Hoping someone here can describe how this works.

 

--

You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/fc05b025-4cfa-477b-9be8-725704010a50n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/1df4846f-2419-a3dc-a69b-691498faefb1%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

vpip3.c
dirtest.c

s shumaker

unread,
Sep 10, 2021, 5:06:34 PM9/10/21
to se...@googlegroups.com
I'm getting errors on that link but Archive.Org also has a copy.

Steve

glenn.f...@gmail.com

unread,
Sep 10, 2021, 5:31:44 PM9/10/21
to se...@googlegroups.com

Click on the link in my original email, not the reply (the URL got truncated/"folded").  Here's the direct link:

The Programmer's CP/M Handbook

 

-----Original Message-----
From: se...@googlegroups.com <se...@googlegroups.com> On Behalf Of s shumaker
Sent: Friday, September 10, 2021 5:06 PM
To: se...@googlegroups.com
Subject: Re: [sebhc] H8D Utility 3: Reading CP/M files from disk image

 

I'm getting errors on that link but Archive.Org also has a copy.

--

You received this message because you are subscribed to the Google Groups "SEBHC" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

norberto.collado koyado.com

unread,
Sep 10, 2021, 5:40:57 PM9/10/21
to se...@googlegroups.com
Is this for CP/M on the H17 floppy controller? Just curious...

From: se...@googlegroups.com <se...@googlegroups.com> on behalf of glenn.f...@gmail.com <glenn.f...@gmail.com>
Sent: Friday, September 10, 2021 2:31 PM
To: se...@googlegroups.com <se...@googlegroups.com>
Subject: RE: [sebhc] H8D Utility 3: Reading CP/M files from disk image
 

glenn.f...@gmail.com

unread,
Sep 10, 2021, 5:59:52 PM9/10/21
to se...@googlegroups.com

More general. It’s about understanding how CP/M directory structure works in general, regardless of the media format.

norberto.collado koyado.com

unread,
Sep 10, 2021, 6:25:11 PM9/10/21
to se...@googlegroups.com
I'm a "RAW" guy, so I only think about sectors and have the HW put it in the right place; read sector x, write sector x to duplicate the media.


Sent: Friday, September 10, 2021 2:59 PM

Darrell Pelan

unread,
Sep 10, 2021, 7:27:23 PM9/10/21
to SEBHC
Hi Les,

As other's mentioned the Programmer's CP/M  Handbook is an absolute must. It took a lot of reading, thinking, and rereading to understand the details. 

My code for reading and writing CP/M files is in the H8DUtility updates I made in version 2.2. An improved version is in the Disk Image Utility I wrote that supports CP/M IMG files for Flash Floppy and MS-DOS disks for Z-100's. The code is at Disk Image Utility on Github. The utility was recently posted on the webpage SEBHC Software

Let me know how I can help.

  Darrell

Darrell Pelan

unread,
Sep 10, 2021, 8:38:32 PM9/10/21
to SEBHC
The definitions I used in the code are below. Depending on the disk size, the directory starts in a different place. The first comment below refers to BIOS parameters to define the disk type. The code is my table of parameters used by the code to determine where to look. The disk images from Dave Dunfield's IMD are interleaved. The IMG files used by Flash Floppy are not interleaved. The H-17 images (100k) do not have a disk identifier ( the value could be E5 or 00).
 /*
        Disk type: byte 5 in sector 0 on H-37 disks (starting from 0) to define disk parameters
        Allocation Block size: number of bytes in an the smallest block used by CP/M on the disk. must be a multiple of 128 (0x80)
                AB numbers start with 0. The directory starts in AB 0.
        Directory Stat: start of directory entries in bytes
        Allocation Block Number Size: number of bytes used in directory entry to reference an allocation block
        Dir Size: number of bytes used for the directory
         0040 =         DPEH17 EQU 01000000B
         0060 =         DPEH37 EQU 01100000B
         0008 =         DPE96T EQU 00001000B
         0004 =         DPEED EQU 00000100B
         0002 =         DPEDD EQU 00000010B
         0001 =         DPE2S EQU 00000001B

        */
        // 0.Disk type, 1.Allocation block size, 2.Directory start, 3.Allocation block byte size, 4.dir size, 5.interleave, 6.Sectors per Track, 7.Sector Size,
        // 8.# Tracks, 9. # heads, 10 Skew Start Sector (0 based)
        public int[,] DiskType =
        {
            // 0        1          2         3       4        5    6       7       8   9  10     position in table 0..10
            {0xff, 0x800, 0x4800, 2, 0x2000, 6, 18, 0x200, 80, 2, 10}, // 0 1.44 MB Small Z-80 
            {0x6f, 0x800, 0x2800, 2, 0x2000, 3, 5, 0x400, 80, 2, 4}, //   1 800k H37 96tpi ED DS
            {0x6b, 0x800, 0x2000, 2, 0x2000, 3, 16, 0x100, 80, 2, 12}, // 2 640k H37 96tpi DD DS
            {0x67, 0x800, 0x2800, 1, 0x2000, 3, 5, 0x400, 40, 2, 4}, //   3 400k H37 48tpi ED DS
            {0x23, 0x800, 0x2000, 1, 0x2000, 1, 8, 0x200, 40, 2, 0}, //   4 320k Z100 48tpi DD DS
            {0x62, 0x400, 0x1000, 1, 0x2000, 3, 16, 0x100, 40, 1, 12}, // 5 160k H37 48tpi DD SS
            {0x63, 0x800, 0x2000, 1, 0x2000, 3, 16, 0x100, 40, 2, 4}, //   6 320k H37 48tpi DD DS
            {0x60, 0x400, 0x1e00, 1, 0x800, 3, 10, 0x100, 40, 1, 3}, //   7 100k H37 48tpi DD SS
            {0xE5, 0x400, 0x1e00, 1, 0x800, 4, 10, 0x100, 40, 1, 0}, //   8 100k Default H17 48tpi SD SS
            {0x00, 0x400, 0x1e00, 1, 0x800, 4, 10, 0x100, 40, 1, 0}, //   9 100k Default H17 48tpi SD SS

        };

  Darrell
On Friday, September 10, 2021 at 12:25:01 PM UTC-4 Les Bird wrote:

Mark Garlanger

unread,
Sep 10, 2021, 9:07:09 PM9/10/21
to se...@googlegroups.com
Hey Les,

     I also have some code here - https://github.com/mgarlanger/heath-imager/blob/master/src/libs/cpm.cpp for reading CP/M, that seems to be working well. It handles 100k, 200k, and 400k images. Although the 200k hasn't been tested well.

Mark



--
You received this message because you are subscribed to the Google Groups "SEBHC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sebhc+un...@googlegroups.com.

Les Bird

unread,
Sep 10, 2021, 10:27:41 PM9/10/21
to SEBHC
Darrell, Mark, thanks a lot guys. I should have enough info now to do what I need.

Les

Darrell Pelan

unread,
Sep 11, 2021, 9:43:38 AM9/11/21
to SEBHC
Les,

Glad to help.

  Darrell

Les Bird

unread,
Oct 11, 2021, 10:53:21 AM10/11/21
to SEBHC
Darrell,

Not sure if you've been keeping up with H8DUtility3 (https://github.com/lesbird/H8DUtility3/blob/main/README.md) but I'm making lots of progress with it. I solved the CP/M file extraction issue I was having. Your code helped a lot. The problem was I was not taking into account the sector skewing. Once I got that figured out I was able to read and extract CP/M files.

I got the H37 disk imager working for reading disk images but writing is not working yet. No support yet on the Heathkit side for this but I have started the process. So far the disk imager will read:
1. H37 HDOS/CPM format (single/double density, 40/80 track, 1/2 sides)
2. PC MSDOS format (single/double, 40/80 track, 1/2 sides)
3. Zenith Z100 all types

It will not read PC high density disks - not supported by WD1797.

So now the problem is - do I add MSDOS directory cataloging, file extraction, etc for the PC/Z100 formats? I think I need to. Do you know anything about the MSDOS file format and is it the same format for the Z100 series computers (ZDOS)?

Also, we will need to support IMD convert for the .H37 format (not in yet) and the ability to insert files into a disk image (not in yet).

Les

Darrell Pelan

unread,
Oct 11, 2021, 12:44:19 PM10/11/21
to se...@googlegroups.com
Hi Les,

Does the .h37 format keep the original sector skewing? If so, you will need to account for that using a function like buildSkew() in my code. I'm currently in Bonaire and can provide more detail on Thursday, if  needed. I decided to use .img format in my app since I was focused on the Flash Floppy Gotek. The .img format is not skewed which makes coding easier. The HxC Gotek doesn't care about skew, so the conversion function just stores it in logical order when writing an IMD file.

My Disk Image Utility program supports MS-DOS FAT 12 disks, which is enough to support floppy disks. I don't think FAT 16 was used on floppies. The code supports reading, extracting, and writing files to the image. I didn't spend the time to write delete code or View for MS-DOS yet. Z-DOS uses FAT 12, so the code supports it. 

There is also code to support writing .imd files, though it doesn't account for disk skew. If the .h37 format maintains the physical disk skew, you will need to modify the code to list the sectors in the correct order when writing the IMD track header information. The current code just writes the sectors to the .img file in the same order as the source file, assuming logical order.

   Darrell

You received this message because you are subscribed to a topic in the Google Groups "SEBHC" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sebhc/eefVyCPeSbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sebhc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sebhc/2eb38df4-7555-46cd-8dff-7f2b9e7f18fdn%40googlegroups.com.

Les Bird

unread,
Oct 11, 2021, 2:07:53 PM10/11/21
to SEBHC
Darrell,

H37 does not retain skewing. It's just a sector by sector dump in logical order, track by track. The data appended to the end contains sectors per track, sector size, number of tracks, number of sides and single/double density so we know how to reconstruct the disk.

As Douglas mentioned in another post trying to extract skewing data would be a bit complex.

Les

Darrell Pelan

unread,
Oct 11, 2021, 10:10:48 PM10/11/21
to se...@googlegroups.com
So if you want to extract data from the disk you will need to depend on the data in byte 5 for CP/M disks. The ZDOS disks are not skewed.

Douglas Miller

unread,
Oct 11, 2021, 10:53:17 PM10/11/21
to se...@googlegroups.com

Note, the H37 skew was *physical* skew, the sectors were formatted on the track in an order that optimized sequential access. For physical skew, you don't need any special handling to access the data, assuming the data is organized in numerical sector order (1,2,3...). The issue for the images is that you don't have the skew information to reformat a new diskette that is identical to the original. The data will still be the same, but it may not perform as well.

The H17 disks had a *logical* skew, at least for some OSes, and that does require knowledge of in order to access the data. It is also possible to format H17 diskettes using a physical skew (on top of the logical skew), which complicates things a lot when trying to predict performance.

Reply all
Reply to author
Forward
0 new messages