MBR and sector size

[muta...@gmail.com]

I noticed from here:

https://en.wikipedia.org/wiki/Master_boot_record

that the first 512 bytes of the hard disk doesn't store
the sector size. So when you need to get to the first
partition, you have the LBA number, but you don't know
how many bytes you need to seek to.

I guess this means I need to configure my hard disks
in my own BIOS and OS with the sector size.

BFN. Paul.

[Grant Taylor]

On 10/29/21 6:16 PM, muta...@gmail.com wrote:
> that the first 512 bytes of the hard disk doesn't store the sector
> size.

I thought sector size was a hard drive physical property and not
something that could easily be adjusted. Thus there is effectively no
need to look it up.

As I understand it, there are two vastly predominant sector sizes; the
venerable 512 bytes and the new 4096 bytes (8 x traditional 512 bytes).
All drives that have sectors larger than 512 are multiples of 512 and
have virtual 512 sectors for backwards compatibility.



--
Grant. . . .
unix || die

[muta...@gmail.com]

On Saturday, October 30, 2021 at 12:10:09 PM UTC+11, Grant Taylor wrote:

> > that the first 512 bytes of the hard disk doesn't store the sector
> > size.

> I thought sector size was a hard drive physical property and not
> something that could easily be adjusted. Thus there is effectively no
> need to look it up.

When the hard disk data is extracted into a flat file of
sectors, how do I know how many bytes to skip to go
to a particular LBA address?

I have an expectation that data can be copied to and
from devices and still be accessible.

I need to either distinguish devices based on sector size
or I need to introduce a dummy sector. If I have a dummy
sector I would have an expectation that the firmware
would maintain that sector as if it was real.

Or maybe I am looking at things incorrectly.

BFN. Paul.

[wolfgang kern]

On 30/10/2021 14:46, muta...@gmail.com wrote:

>>> that the first 512 bytes of the hard disk doesn't store the sector
>>> size.

sector size is found in the ATA/ATAPI IDENTIFY-(packet)-DEVICE HD (CD).

>> I thought sector size was a hard drive physical property and not
>> something that could easily be adjusted. Thus there is effectively no
>> need to look it up.

> When the hard disk data is extracted into a flat file of
> sectors, how do I know how many bytes to skip to go
> to a particular LBA address?

LBA28 and LBA48 in PIO mode always access only 512 bytes,
it needs other commands to rd/wr 4k blocks.

> I have an expectation that data can be copied to and
> from devices and still be accessible.

> I need to either distinguish devices based on sector size
> or I need to introduce a dummy sector. If I have a dummy
> sector I would have an expectation that the firmware
> would maintain that sector as if it was real.

how to add a dummy sector to an existing HD ?

> Or maybe I am looking at things incorrectly.

probably
__
wolfgang

[muta...@gmail.com]

On Sunday, October 31, 2021 at 1:21:31 AM UTC+11, wolfgang kern wrote:

> > I need to either distinguish devices based on sector size
> > or I need to introduce a dummy sector. If I have a dummy
> > sector I would have an expectation that the firmware
> > would maintain that sector as if it was real.

> how to add a dummy sector to an existing HD ?

Oh - some of the things I do is actually speculation on what
would be ideal rather than something I'm planning on doing
at 3pm tomorrow.

So to add a dummy sector to an existing HD, all you need
is firmware loaded that returns with 512 bytes of data that
the HD has no clue about, when LBA 0 or maybe LBA -1 is
requested

If LBA 0 is chosen, the data will need to be unloaded, all
the LBA references have 1 added to them, and then the
HD is reloaded.

If anyone complains, I'll ask Jens to bomb them.

BFN. Paul.

[wolfgang kern]

On 30/10/2021 16:54, muta...@gmail.com wrote:

>> how to add a dummy sector to an existing HD ?
>
> Oh - some of the things I do is actually speculation on what
> would be ideal rather than something I'm planning on doing
> at 3pm tomorrow.
>
> So to add a dummy sector to an existing HD, all you need
> is firmware loaded that returns with 512 bytes of data that
> the HD has no clue about, when LBA 0 or maybe LBA -1 is
> requested
>
> If LBA 0 is chosen, the data will need to be unloaded, all
> the LBA references have 1 added to them, and then the
> HD is reloaded.
>
> If anyone complains, I'll ask Jens to bomb them.

LBA 0 is a valid sector, known and used as MBR
LBA -1 may exist as 2^48-1 in far distant future,
today it will just lockup the drive with ERR.
512*(2^48-1)==144'115'188'075'855'360 bytes
__
wolfgang

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Saturday, October 30, 2021 at 12:10:09 PM UTC+11, Grant Taylor wrote:
>
>> > that the first 512 bytes of the hard disk doesn't store the sector
>> > size.
>
>> I thought sector size was a hard drive physical property and not
>> something that could easily be adjusted. Thus there is effectively no
>> need to look it up.
>
>When the hard disk data is extracted into a flat file of
>sectors, how do I know how many bytes to skip to go
>to a particular LBA address?

For a SCSI drive, the READ CAPACITY command will return the
maximum logical block address and the logical block length in
bytes.

For an ATA/SATA device use the ATA8 Identify Device command
to get the logical sector size (words 117-118, and word 106 bits <15>, <14>, and <12>).

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Sunday, October 31, 2021 at 1:21:31 AM UTC+11, wolfgang kern wrote:
>
>> > I need to either distinguish devices based on sector size
>> > or I need to introduce a dummy sector. If I have a dummy
>> > sector I would have an expectation that the firmware
>> > would maintain that sector as if it was real.
>
>> how to add a dummy sector to an existing HD ?
>
>Oh - some of the things I do is actually speculation on what
>would be ideal rather than something I'm planning on doing
>at 3pm tomorrow.
>
>So to add a dummy sector to an existing HD, all you need
>is firmware loaded that returns with 512 bytes of data that
>the HD has no clue about, when LBA 0 or maybe LBA -1 is
>requested

signed block numbers are a really bad idea, particularly on
a twos complement system.

[Grant Taylor]

On 10/30/21 6:46 AM, muta...@gmail.com wrote:
> When the hard disk data is extracted into a flat file of sectors,
> how do I know how many bytes to skip to go to a particular LBA address?

I don't know the technical answer to that.

But I do know that I've never had to worry about that in 20+ years of
doing a lot of drive / file system imaging.

I either image a file system (contents of a partition) or I image the
entire drive (including partition information). I access the former as
a raw file system. I access the latter via Linux kernel's ability to
see partitions on a loopback and then access the contents of a partition
therein.

The only time that I've needed to access something independently is when
I'm needing to wipe the first sector for some reason. And that's
inherently the first sector. So I don't need to find out where an LBA
address is in an image.

> I have an expectation that data can be copied to and from devices
> and still be accessible.

My experience is that as long as the source and destinations are the
same type (partition contents / file system -or- whole drive) I don't
have any problems doing this.

> I need to either distinguish devices based on sector size or I need
> to introduce a dummy sector. If I have a dummy sector I would have
> an expectation that the firmware would maintain that sector as if it
> was real.

You should be able to extract the information from any standard file
system / partition data in the image. I'd expect a quick analysis /
pattern match on the first 4kB of data would clearly give you
information to know what things are. Either it will have a standard
partition table (indicating it's a whole drive image) or it will have
standard file system information (indicating it's a partition image) or
it will not have either of those (indicating that it's likely an
atypical use) which can probably be accessed as a raw device.

I seriously doubt that you need to create a dummy sector or burn a
sector as a dummy sector. I would bet dollars to doughnuts that you can
look for a different pattern and find it exists in almost all existing
drives / images thereof.

> Or maybe I am looking at things incorrectly.

;-)

[muta...@gmail.com]

On Sunday, October 31, 2021 at 4:00:13 AM UTC+11, Grant Taylor wrote:

> I either image a file system (contents of a partition) or I image the
> entire drive (including partition information). I access the former as
> a raw file system. I access the latter via Linux kernel's ability to
> see partitions on a loopback and then access the contents of a partition
> therein.

Do you agree that in the latter case, Linux will need to
hardcode the number 512 for that to work?

> You should be able to extract the information from any standard file
> system / partition data in the image. I'd expect a quick analysis /
> pattern match on the first 4kB of data would clearly give you
> information to know what things are. Either it will have a standard
> partition table (indicating it's a whole drive image)

I know it is a whole drive image. I just don't know whether
it is using 512 or 4096 or some other number sectors. I was
expecting this information to be recorded some place in the
MBR, the same as it is in the FAT boot sector.

Once you get to the FAT, everything is cool - the info is there,
but how are you supposed to get there?! Maybe I should change
the MBR format the same as other people did.

BFN. Paul.

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Sunday, October 31, 2021 at 4:00:13 AM UTC+11, Grant Taylor wrote:
>
>> I either image a file system (contents of a partition) or I image the
>> entire drive (including partition information). I access the former as
>> a raw file system. I access the latter via Linux kernel's ability to
>> see partitions on a loopback and then access the contents of a partition
>> therein.
>
>Do you agree that in the latter case, Linux will need to
>hardcode the number 512 for that to work?

No, it will use the BLKBSZGET/BLKSZSGET ioctl's to retrieve
the sector size for the device.

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Sunday, October 31, 2021 at 4:00:13 AM UTC+11, Grant Taylor wrote:

>I know it is a whole drive image. I just don't know whether
>it is using 512 or 4096 or some other number sectors. I was
>expecting this information to be recorded some place in the
>MBR, the same as it is in the FAT boot sector.

That's a bit of a chicken and egg problem, in that you really
should know the sector size before reading the device.

Both SCSI and ATA provide a means for the device driver to
determine the sector size of the media (512, 2048, 4096,
527, 180, 100, 384 - I've seen them all in real world disks).

Most modern operating systems intentionally hide the details
of the underlying storage system from the application, for
good reason.

[muta...@gmail.com]

On Sunday, October 31, 2021 at 7:22:24 AM UTC+11, Scott Lurndal wrote:

> >I know it is a whole drive image. I just don't know whether
> >it is using 512 or 4096 or some other number sectors. I was
> >expecting this information to be recorded some place in the
> >MBR, the same as it is in the FAT boot sector.

> That's a bit of a chicken and egg problem, in that you really
> should know the sector size before reading the device.
>
> Both SCSI and ATA provide a means for the device driver to
> determine the sector size of the media (512, 2048, 4096,
> 527, 180, 100, 384 - I've seen them all in real world disks).

Shouldn't that have been encapsulated in this BIOS call
or some other BIOS call?

http://www.ctyme.com/intr/rb-0621.htm

I'm now thinking that my pseudo-BIOS which exports fread etc
to the OS, should be using the MVS flavor of the C library, which
is focused on an expectation that the underlying system will
give you a block, and you will be told the size of that block, and
the underlying system does not provide a facility to keep track of
a byte offset within that block for you.

As opposed to a C library that can make use of a seek() syscall
that will position on byte offset 3 or whatever in a file for you.

I think that is the concept I have been missing.

BFN. Paul.

[Grant Taylor]

On 10/30/21 11:30 AM, muta...@gmail.com wrote:
> Do you agree that in the latter case, Linux will need to hardcode
> the number 512 for that to work?

No, not as such.

It really uses the same partition support for physical drives as it does
loopback images. I say this because the loopback image can be treated
as pseudo physical device that the same partition support works on top of.

The partition support knows about the (possible) sector size(s) that the
partitioning scheme supported and thus tries the (possible)
combination(s) that are appropriate. There is no need to hard code what
any given device / image is. Simply try the list of possibilities in
order from most likely to least likely. If it's not one of the known
and supported types, then it's not supported.

[muta...@gmail.com]

On Sunday, October 31, 2021 at 10:05:12 AM UTC+11, muta...@gmail.com wrote:

> I'm now thinking that my pseudo-BIOS which exports fread etc
> to the OS, should be using the MVS flavor of the C library, which
> is focused on an expectation that the underlying system will
> give you a block, and you will be told the size of that block, and
> the underlying system does not provide a facility to keep track of
> a byte offset within that block for you.
>
> As opposed to a C library that can make use of a seek() syscall
> that will position on byte offset 3 or whatever in a file for you.
>
> I think that is the concept I have been missing.

Is there existing terminology for this? Block devices versus
character streams perhaps?

Thanks. Paul.

[Scott Lurndal]

Congratulations, you've re-invented Unix 'block' and 'character'
devices.

[muta...@gmail.com]

On Sunday, October 31, 2021 at 11:04:37 AM UTC+11, Scott Lurndal wrote:

> >> I'm now thinking that my pseudo-BIOS which exports fread etc
> >> to the OS, should be using the MVS flavor of the C library, which
> >> is focused on an expectation that the underlying system will
> >> give you a block, and you will be told the size of that block, and
> >> the underlying system does not provide a facility to keep track of
> >> a byte offset within that block for you.
> >>
> >> As opposed to a C library that can make use of a seek() syscall
> >> that will position on byte offset 3 or whatever in a file for you.
> >>
> >> I think that is the concept I have been missing.
> >
> >Is there existing terminology for this? Block devices versus
> >character streams perhaps?

> Congratulations, you've re-invented Unix 'block' and 'character'
> devices.

An ordinary file is considered to be a "device", is it?

Regardless, would you expect a C library to have different paths
for dealing with block devices vs character devices?

And does MVS deal exclusively with block devices?

When I went to implement PDPCLIB for MVS, I already had
a DOS/OS2 version with only minor differences, but for MVS
I knew I needed to deal with "records" and I wasn't sure whether
it could fit in with the code I already had, so I simply put some
#ifdefs and created brand new versions of fread/fgets/etc to
deal with MVS records.

It never occurred to me that the MVS flavor would be relevant
to Unix block devices.

BFN. Paul.

[Rod Pemberton]

On Fri, 29 Oct 2021 17:16:01 -0700 (PDT)
"muta...@gmail.com" <muta...@gmail.com> wrote:

> I noticed from here:
>
> https://en.wikipedia.org/wiki/Master_boot_record
>
> that the first 512 bytes of the hard disk doesn't store
> the sector size. So when you need to get to the first
> partition, you have the LBA number, but you don't know
> how many bytes you need to seek to.
>

According to an article by Seagate linked to by Wikipedia (link below),
LBAs are always 512 bytes. (Is this correct? IDK ...) Apparently,
LBAs are always 512 bytes even for the 4KB block devices. (I guess
this could be true for 4KB block devices if using LBA48.)

"Each 512-byte sector is assigned a unique LBA, from zero (0) to the
number required based on the size of the disk."

Also, according to the same article, OSes and partioning software are
either 4K aware or they aren't. I.e., for 4K aware OS and partitioning
software, LBA 0 should be aligned to 0th byte offset or the 1st 512
block of the 0th 4KB block ("aligned"), instead of the 2nd, 3rd, 4th,
5th, 6th, 7th, or 8th 512 block of the 0th 4KB block ("unaligned"). If
the OS and partition software is 4K aware, then the partioning software
also aligns the partition start to a 4KB boundary. Apparently,
alignment of partitions to non-4K boundaries is to be avoided. If a
partion is unaligned on a 4K device, then it must be re-aligned using
special tools, or deleted and recreated with proper 4K alignment.

There is a table for Windows OSes as to whether or not they're 4K
aware. The article also lists at which kernel version Linux is 4K
aware thereafter.

https://www.seagate.com/tech-insights/advanced-format-4k-sector-hard-drives-master-ti/

In other words, or AIUI, LBAs work as we expect, reading or writing
only 512 byte blocks for both 512 byte or 4KB block devices. And, so,
if the image is 512 byte block based, everything is fine. And, if the
image is 4KB block, the partitioning software should've aligned the
partition start to a 4KB boundary. If the image is 4KB block, and the
partition is unaligned, someone must fix the partition start on the 4KB
device to be 4K aligned, and remake the image.

--
Democrats are trying to tax billionaires but repeal the SALT cap which
benefits the rich, at the same time.

[wolfgang kern]

yes. when this 4K blocks joined in there were lots of misunderstanding.
it was/is recommended to have everything aligned to 4K for speed reason
only because the device internal cache and the tracks are on 4K bounds.

As my OS-modules and the apps I wrote for it were often less than 512
bytes it would have been a huge waste if I'd aligned all to 4K.
I decided to merge several smaller parts to reside within a block even
rare needed all of them at the very same time. So I kept my 512 byte
access but just avoided that any 512 byte crossed the 4K bound.
__
wolfgang

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Sunday, October 31, 2021 at 11:04:37 AM UTC+11, Scott Lurndal wrote:
>
>> >> I'm now thinking that my pseudo-BIOS which exports fread etc
>> >> to the OS, should be using the MVS flavor of the C library, which
>> >> is focused on an expectation that the underlying system will
>> >> give you a block, and you will be told the size of that block, and
>> >> the underlying system does not provide a facility to keep track of
>> >> a byte offset within that block for you.
>> >>
>> >> As opposed to a C library that can make use of a seek() syscall
>> >> that will position on byte offset 3 or whatever in a file for you.
>> >>
>> >> I think that is the concept I have been missing.
>> >
>> >Is there existing terminology for this? Block devices versus
>> >character streams perhaps?
>
>> Congratulations, you've re-invented Unix 'block' and 'character'
>> devices.
>
>An ordinary file is considered to be a "device", is it?
>
>Regardless, would you expect a C library to have different paths
>for dealing with block devices vs character devices?

The entire point of Unix was to treat all files as simple streams
of bytes. The C library didn't give a shit about the file type.
The Operating System ensured that all files (disk, tape, terminal,
card readers, line printers, et alia) appear to the application as
a simple stream of bytes.

>
>And does MVS deal exclusively with block devices?

MVS, and the old mainframe paradigm of logical record
sizes, record formats, blocking factors are very thankfully
obsolete.

[Joe Monk]

"MVS, and the old mainframe paradigm of logical record
sizes, record formats, blocking factors are very thankfully
obsolete."

You should probably take a good look at whatever is in that pipe that youre smoking ... Youre very clearly hallucinating.

https://www.reddit.com/r/mainframe/comments/mgjzs8/mainframes_in_credit_card_processing/

Joe

[muta...@gmail.com]

On Monday, November 1, 2021 at 1:07:33 AM UTC+11, Scott Lurndal wrote:

> >> Congratulations, you've re-invented Unix 'block' and 'character'
> >> devices.
> >
> >An ordinary file is considered to be a "device", is it?
> >
> >Regardless, would you expect a C library to have different paths
> >for dealing with block devices vs character devices?

> The entire point of Unix was to treat all files as simple streams
> of bytes. The C library didn't give a shit about the file type.
> The Operating System ensured that all files (disk, tape, terminal,
> card readers, line printers, et alia) appear to the application as
> a simple stream of bytes.

I thought the goal was for the application to get a simple
stream of bytes, not necessarily the C library.

For the C library, I think/guess that it should be aware of
whether it is dealing with a block device or a character
device. If it is dealing with a block device, then, for instance,
it shouldn't issue a write() syscall until it has a buffer of
suitable size for that device. Similarly, it should be conscious
when seeking that it is seeking to a block boundary, and
reading should be done of an entire block.

> >And does MVS deal exclusively with block devices?

> MVS, and the old mainframe paradigm of logical record
> sizes, record formats, blocking factors are very thankfully
> obsolete.

I'm with Joe on that, but regardless, I wish to support this
paradigm AND the Unix paradigm. It's just unclear to me
whether Unix should be compiled with both flavors of code,
one to deal with character devices and one to deal with
block devices. Currently Unix is exclusively character and
MVS is exclusively block.

BFN. Paul.

[muta...@gmail.com]

On Monday, November 1, 2021 at 5:16:45 AM UTC+11, muta...@gmail.com wrote:

> > >Regardless, would you expect a C library to have different paths
> > >for dealing with block devices vs character devices?
>
> > The entire point of Unix was to treat all files as simple streams
> > of bytes. The C library didn't give a shit about the file type.
> > The Operating System ensured that all files (disk, tape, terminal,
> > card readers, line printers, et alia) appear to the application as
> > a simple stream of bytes.

> I thought the goal was for the application to get a simple
> stream of bytes, not necessarily the C library.
>
> For the C library, I think/guess that it should be aware of
> whether it is dealing with a block device or a character
> device. If it is dealing with a block device, then, for instance,
> it shouldn't issue a write() syscall until it has a buffer of
> suitable size for that device. Similarly, it should be conscious
> when seeking that it is seeking to a block boundary, and
> reading should be done of an entire block.

I think I know what was bugging me.

The Unix implementation of the C library doesn't need to
care about blocks - Unix will take care of that.

My situation is that I will have a C library directly doing BIOS
calls (and maybe UEFI calls), not OS calls, so I'm the one that
needs to deal with the block devices, specifically sector reads
and writes, so I need to use the "MVS" flavor of PDPCLIB,
which is incorrectly named "MVS" and should instead have
been called "block".

I think.

BFN. Paul.

[Rod Pemberton]

On Sun, 31 Oct 2021 14:07:31 GMT
sc...@slp53.sl.home (Scott Lurndal) wrote:

> The C library didn't give a shit about the file type.

Could you clarify? ...

I.e., both fopen() and freopen() require the file type, either text or
binary, for the three modes: read 'r', write 'w', and append 'a'. ANSI
C defaults to opening files as text, unless the file is opened as
binary, either by specifying binary mode by adding a 'b' to the mode,
or by specifying update mode by adding a '+' to the mode.

--
Why does Mr Zuckerberg's metaverse avatar appear as if it's being
strangled by an invisible noose? chest out, shoulders back, ...

[Rod Pemberton]

On Sun, 31 Oct 2021 14:07:31 GMT
sc...@slp53.sl.home (Scott Lurndal) wrote:

> >Regardless, would you expect a C library to have different paths
> >for dealing with block devices vs character devices?
>
> The entire point of Unix was to treat all files as simple streams
> of bytes.

In general, I'd agree with you that Unix "treat[s] all files as simple
streams of bytes" as a matter of Unix design philosophy. I would
actually claim that the concept includes not just files, but also
includes all permanent storage media and memory mapped devices.

But, to say the (emphasis added) "The ENTIRE POINT of Unix" is only to
treat files as streams, is a bit much, at least for me anyway. I.e.,
you a multitude of Unix design philosophies and capabilities into just
one primacy concept.

[Rod Pemberton]

On Mon, 1 Nov 2021 02:14:17 -0500
Rod Pemberton <noe...@basdxcqvbe.com> wrote:

> On Sun, 31 Oct 2021 14:07:31 GMT
> sc...@slp53.sl.home (Scott Lurndal) wrote:
>
> > >Regardless, would you expect a C library to have different paths
> > >for dealing with block devices vs character devices?
> >
> > The entire point of Unix was to treat all files as simple streams
> > of bytes.
>
> In general, I'd agree with you that Unix "treat[s] all files as simple
> streams of bytes" as a matter of Unix design philosophy.

Oh, I said "in general" because I think the Unix concept was
"Everything is a file," whereas C's concept is that every C object maps
onto a sequence of contiguous C bytes. I.e., it seems as if you've
merged or conflated the two concepts into one.

[Rod Pemberton]

This is a Usenet newsgroup that you're posting to from Google Groups.
(The latter part is an assumption since you use Gmail.)

I.e., those on Usenet use newsreaders to read the thread. As such,
replies are not ordered or threaded the same way as on browser based
Google Groups. In fact, I can select numerous different ordering and
threading combinations in this newsreader. So, when you snip so much
context, as well as the header lines which show to whom you replied to,
we have to go back up through the thread to determine just whom you
replied to. In this case, you were replying to a post which is up six
messages in my newsreader, because other replies were threaded below,
before yours. You were replying to Scott Lurndal, not Paul Edwards.
So, this should've been kept at the top of your reply:

>On Sunday, October 31, 2021 at 11:04:37 AM UTC+11, Scott Lurndal
>wrote:

[Scott Lurndal]

I did not say that they were not present, just obsolete. Most of that
code was written thirty years ago.

[Scott Lurndal]

"muta...@gmail.com" <muta...@gmail.com> writes:
>On Monday, November 1, 2021 at 1:07:33 AM UTC+11, Scott Lurndal wrote:
>
>> >> Congratulations, you've re-invented Unix 'block' and 'character'
>> >> devices.
>> >
>> >An ordinary file is considered to be a "device", is it?
>> >
>> >Regardless, would you expect a C library to have different paths
>> >for dealing with block devices vs character devices?
>
>> The entire point of Unix was to treat all files as simple streams
>> of bytes. The C library didn't give a shit about the file type.
>> The Operating System ensured that all files (disk, tape, terminal,
>> card readers, line printers, et alia) appear to the application as
>> a simple stream of bytes.
>
>I thought the goal was for the application to get a simple
>stream of bytes, not necessarily the C library.

The "C library" as you call it simply calls the operating
system. The C library sees only a stream of bytes.

>> MVS, and the old mainframe paradigm of logical record
>> sizes, record formats, blocking factors are very thankfully
>> obsolete.
>
>I'm with Joe on that, but regardless, I wish to support this
>paradigm AND the Unix paradigm.

The advantage of the Unix paradigm is that the _application_
can build whatever level of records or blocks that it needs
on top of the byte stream. That's why unix has lseek/fseek.

There's no need to have that stuff in the operating system
(or in the case of VAX/VMS, in ring 1 (executive)).

[Scott Lurndal]

Rod Pemberton <noe...@basdxcqvbe.com> writes:
>On Sun, 31 Oct 2021 14:07:31 GMT
>sc...@slp53.sl.home (Scott Lurndal) wrote:
>
>> The C library didn't give a shit about the file type.
>
>Could you clarify? ...
>
>I.e., both fopen() and freopen() require the file type, either text or
>binary,

On Unix systems that fopen file type (text/binary) is completely ignored and
has no effect. That's not true on Windows, but from a C and C Library
perspective, they're johnny-come-lately and they break a lot of the clean
unix concepts.

"The character 'b' shall have no effect, but is allowed for
ISO C standard conformance. [Option End]"

https://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html

[Scott Lurndal]

Rod Pemberton <noe...@basdxcqvbe.com> writes:
>On Mon, 1 Nov 2021 02:14:17 -0500
>Rod Pemberton <noe...@basdxcqvbe.com> wrote:
>
>> On Sun, 31 Oct 2021 14:07:31 GMT
>> sc...@slp53.sl.home (Scott Lurndal) wrote:
>>
>> > >Regardless, would you expect a C library to have different paths
>> > >for dealing with block devices vs character devices?
>> >
>> > The entire point of Unix was to treat all files as simple streams
>> > of bytes.
>>
>> In general, I'd agree with you that Unix "treat[s] all files as simple
>> streams of bytes" as a matter of Unix design philosophy.
>
>Oh, I said "in general" because I think the Unix concept was
>"Everything is a file,"

I will agree that "the entire point of" was a bit hyperbolic, but
I stand by the statement with respect to data access via the file
system. As someone who was plugging away at the time with mainframe
record subsystems (IBM, Burroughs and RSX-11/VMS), I understand
completely why Dennis and Ken chose the simple bytestream and allowed
the application to apply structure to it.

It was Plan 9 that coined the "everything is a file" meme. SVR4
did add 'mmap' and pseudo filesystems like /proc, which were
somewhat Plan 9-ish, but mmap came to SVR4 via solaris.

[Rod Pemberton]

On Mon, 1 Nov 2021 02:14:17 -0500
Rod Pemberton <noe...@basdxcqvbe.com> wrote:

> On Sun, 31 Oct 2021 14:07:31 GMT
> sc...@slp53.sl.home (Scott Lurndal) wrote:

> > >Regardless, would you expect a C library to have different paths
> > >for dealing with block devices vs character devices?
> >
> > The entire point of Unix was to treat all files as simple streams
> > of bytes.
>
> In general, I'd agree with you that Unix "treat[s] all files as simple
> streams of bytes" as a matter of Unix design philosophy. I would
> actually claim that the concept includes not just files, but also
> includes all permanent storage media and memory mapped devices.
>
> But, to say the (emphasis added) "The ENTIRE POINT of Unix" is only to
> treat files as streams, is a bit much, at least for me anyway. I.e.,
> you a multitude of Unix design philosophies and capabilities into just
> one primacy concept.
>

you reduced a multitude

[Joe Monk]

On Monday, November 1, 2021 at 8:58:51 AM UTC-5, Scott Lurndal wrote:

> There's no need to have that stuff in the operating system
> (or in the case of VAX/VMS, in ring 1 (executive)).

Hmmm.... without that stuff in the OS, how does the OS build paging files? Or spool files?

Joe

[Scott Lurndal]

Paging files contain fixed (page-sized) blocks. There's nothing to do but write
them to the device. No record formats to support. Unix supports paging just fine
without any user-level file system support in the kernel.

Spool files are an artifact of the unit record world, thankfully also obsolete.

In Unix, simple redirection to a stream of bytes (with appropriate formatting
characters to replace the channel codes from the mainframe days) is sufficient
for printed output.

[Joe Monk]

On Tuesday, November 2, 2021 at 9:41:49 AM UTC-5, Scott Lurndal wrote:

>
> Spool files are an artifact of the unit record world, thankfully also obsolete.
>

All windows printing is done with spool files, hence the existence of the print spooler.

Joe

[Scott Lurndal]

And none of the handling for that is in the NTOS kernel. It's in usermode code like
it should be.

[Joe Monk]

All spool file handling in MVS is in JES2. Its not part of the nucleus either.

Access Methods (BSAM, QSAM, etc.) are not in the nucleus either. The nucleus issues CCWs just like everything else.

Joe

[Grant Taylor]

On 11/6/21 5:26 PM, Joe Monk wrote:
> All spool file handling in MVS is in JES2.

What about JES3?

What about other 3rd party alternatives?