USB Install Fails, Complains about CD-ROM

[Kent West]

I have a Dell Latitude E7250 laptop. I'm trying to install Debian to it using a USB stick.

I've tried both of these .ISOs:

debian-9.4.0-amd64-xfce-CD-1.iso

debian-buster-DI-alpha2-amd64-xfce-CD-1.iso

I used my desktop Debian box to download these via Firefox from https://www.debian.org/CD/http-ftp/

I inserted a USB stick, and ran:

# sudo cp debian-9.4.0-amd64-xfce-CD-1.iso /dev/sdc

as per the instructions at https://www.debian.org/CD/faq/#write-usb

I then ejected the USB stick from my desktop Debian box, and inserted it into the laptop, and then booted the laptop to the USB stick.

The graphical install does not seem to recognize the trackpad (which is recognized in the laptop's EFI firmware settings, so I know it works), but that's a minor issue, as I can tinker with that later, and just use the keyboard to install for now.

The real problem is that after going through the first three or four screens, the install halts, complaining about not being able to read the CD-ROM.

Googling the issue suggested a couple of possible fixes, but I've had no success yet.

Any help?

Thanks!

--

Kent West                    <")))><
Westing Peacefully - http://kentwest.blogspot.com

[Robert Menes]

Hi Kent,

It's much easier to write the image to the USB stick using the dd command instead:

# dd if=debian-9.4.0-amd64-xfce-CD-1.iso of=/dev/sdc bs=1M

This should give you a working install stick.

--Robert

[Thomas Schmitt]

Hi,

Kent West wrote:
> The real problem is that after going through the first three or four
> screens, the install halts, complaining about not being able to read the
> CD-ROM.

Report this to debi...@lists.debian.org and add the original messages
of the complaing software.

I'd say that if the software can show installation screens then the
part of copying to USB stick was done properly. But the USB stick and
its ISO 9660 filesystem seems not to be recognized as substitue for a CD.
That's not normal.


Have a nice day :)

Thomas

[Curt]

On 2018-05-11, Kent West <we...@acu.edu> wrote:
>
> The real problem is that after going through the first three or four
> screens, the install halts, complaining about not being able to read the
> CD-ROM.

I would guess that in the installer's parlance it's referring to your usb stick.

Have you tried another stick (flaky stick)? Or another usb port (flaky
port)?

> Googling the issue suggested a couple of possible fixes, but I've had no
> success yet.

If only we knew what they were--like going to the console and mounting
the usb stick manually?

> Any help?
>
> Thanks!
>
>
>


--

[songbird]

Kent West wrote:
> --000000000000b378b9056bf066d4
> Content-Type: text/plain; charset="UTF-8"

try the netinst images instead and when you do the copy
make sure it is sync'd before removing USB device. on my
system the cp returns very quickly but the sync may take
some time before everything is written to the USB stick.


songbird

[Kent West]

Yes, but mounting didn't work, no matter what I tried. But as reported in another post just now, the problem turned out to be that the stick had a "mac" partition table; I converted it to "GPT", and now all is well (well, except for the trackpad not working, which I'll worry about later).

[Kent West]

While chasing down a completely different issue unrelated to this install, I learned that EFI boot drives need to have a GPT partition table. On a lark, I ran "gparted --list", and discovered that the flash drive had a "mac" partition table.

Wha-a-ah-h-h??

Okay. So I ran gparted, selected the drive, and created a new "GPT" partition table, then repeated all my former steps, and bang! Success! It's currently pulling down gobs of stuff (I elected to install Cinnamon and KDE and Gnome - always fun to overload a drive unnecessarily ;-) ).

Well, the trackpad still doesn't work in the installer, but I have half a suspicion that once the system boots, the trackpad will work. We'll see in a few.

Thanks for the responses, all!

[Kent West]

Interesting....

Just for kicks, after getting a working system, I plugged the USB stick back in, and again ran "parted --list", and the stick again shows up as a "mac" partition table.

Weird. 

But it worked to install, so I'm not gonna bother thinking about it any more.

Well, the trackpad still doesn't work in the installer, but I have half a suspicion that once the system boots, the trackpad will work. We'll see in a few.

Yep, when I booted into the installed system, the trackpad works just fine.

Now to tackle that nasty broadcom wireless.... :-(

[Pascal Hambourg]

Le 11/05/2018 à 20:33, Kent West a écrit :
>
> I learned that EFI boot drives need to have a GPT partition table. On a

This is not correct. The UEFI specification supports boot from a drive
with an MSDOS partition table. Otherwise why would there be an "EFI
system partition" type identifier (0xef) for MSDOS partition tables ?
The Debian installer hybrid image has an MSDOS partition table. (It also
has Apple and GPT partition tables, but they are bogus)

> lark, I ran "gparted --list", and discovered that the flash drive had a
> "mac" partition table.
>
> Wha-a-ah-h-h??

It is part of the image.

> Okay. So I ran gparted, selected the drive, and created a new "GPT"
> partition table, then repeated all my former steps, and bang! Success!

Copying the image again overwrote the GPT partition table and anything
else you may have written to the stick. So it does not explain the
success. You may have done something wrong the first time.

[Kent West]

On Fri, May 11, 2018 at 2:59 PM, Pascal Hambourg <pas...@plouf.fr.eu.org> wrote:

Le 11/05/2018 à 20:33, Kent West a écrit :

I learned that EFI boot drives need to have a GPT partition table. On a

This is not correct. The UEFI specification supports boot from a drive with an MSDOS partition table. Otherwise why would there be an "EFI system partition" type identifier (0xef) for MSDOS partition tables ? The Debian installer hybrid image has an MSDOS partition table. (It also has Apple and GPT partition tables, but they are bogus)

That's good to know. I guess my source material (https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/) is wrong. Or I misunderstood it.

lark, I ran "gparted --list", and discovered that the flash drive had a
"mac" partition table.

Wha-a-ah-h-h??

It is part of the image.

Okay. So I ran gparted, selected the drive, and created a new "GPT"
partition table, then repeated all my former steps, and bang! Success!

Copying the image again overwrote the GPT partition table and anything else you may have written to the stick. So it does not explain the success. You may have done something wrong the first time.

Possibly. With my first attempt, the "cp" happened very quickly. I didn't do a "sync", but I did use Cinnamon's system-tray thingie to eject the USB drive before unplugging it. With the second attempt, the "cp" took minutes (long enough I was beginning to think it had hung).

At any rate, I have a working system (except for wirelessly connecting to a secured network (unsecured works fine)), so I'm happy.

Thanks!

[Thomas Schmitt]

Hi,

Kent West wrote:
> > I learned that EFI boot drives need to have a GPT partition table.

No. They do not. An MBR partition of type 0xEF is well ok, too.

> > discovered that the flash drive had a "mac" partition table.
> > Wha-a-ah-h-h??

It has an MBR partition table with partition 2 having type 0xEF,
an invalid GPT, and a useless Apple Partition Map.

Pascal Hambourg wrote:
> It also has Apple and GPT partition tables, but they are bogus

The GPT is not valid because there is a non-"protective" MBR partition
table. The APM is valid, but should be of no interest for any firmware
that does not expect a HFS or HFS+ filesystem. (And there is no such
filesystem in the ISO image.)

Kent West wrote:
> > So I ran gparted, selected the drive, and created a new "GPT"
> > partition table, then repeated all my former steps, and bang! Success!

Pascal Hambourg wrote:
> Copying the image again overwrote the GPT partition table and anything else
> you may have written to the stick. So it does not explain the success. You
> may have done something wrong the first time.

I agree that repeating
https://www.debian.org/CD/faq/#write-usb
overwrites the main GPT and the protective MBR partition table.
It does not overwrite the backup GPT at the end of the storage medium.

Does the usable USB stick report again the Apple Partition Map ?


Kent West wrote:
> With my first attempt, the "cp" happened very quickly. [...]

> With the second attempt, the "cp" took minutes

Well, copying a CD sized image at 10 MB/s should take a bit more than a
minute. "Very quickly" sounds suspicicious.

But if only a few MB were copied (for what strange reason ever) there
remains the riddle why it booted and probably loaded the initrd.
Why does it fail when it looks for the ISO filesystem ?

It would be helpful to know where exactly the installation startup
failed with what message exactly.

[Kent West]

Yes:

westk@westkent:~/Downloads$ sudo parted --list

Model: ATA CT960BX200SSD1 (scsi)

Disk /dev/sda: 960GB

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Disk Flags: 

Number  Start   End     Size    Type      File system     Flags

 1      1049kB  15.0GB  15.0GB  primary   ext4            boot

 2      15.0GB  960GB   945GB   extended

 5      15.0GB  35.0GB  20.0GB  logical   ext4

 6      35.0GB  45.0GB  9999MB  logical   ext4

 7      45.0GB  65.0GB  20.0GB  logical   ext4

 8      65.0GB  70.0GB  4999MB  logical   ext4

 9      70.0GB  928GB   858GB   logical   ext4

10      928GB   960GB   32.0GB  logical   linux-swap(v1)

Model: ATA ST1000DM003-1CH1 (scsi)

Disk /dev/sdb: 1000GB

Sector size (logical/physical): 512B/4096B

Partition Table: msdos

Disk Flags: 

Number  Start   End     Size    Type     File system  Flags

 1      1049kB  1000GB  1000GB  primary  ext4

Warning: The driver descriptor says the physical block size is 2048 bytes, but

Linux says it is 512 bytes.

Ignore/Cancel? i                                                          

Model:  USB Flash Drive (scsi)

Disk /dev/sdc: 62.7GB

Sector size (logical/physical): 2048B/512B

Partition Table: mac

Disk Flags: 

Number  Start   End     Size   File system  Name   Flags

 1      2048B   6143B   4096B               Apple

 2      3693kB  4119kB  426kB               EFI

Kent West wrote:
> With my first attempt, the "cp" happened very quickly. [...]
> With the second attempt, the "cp" took minutes

Well, copying a CD sized image at 10 MB/s should take a bit more than a
minute. "Very quickly" sounds suspicicious.

But if only a few MB were copied (for what strange reason ever) there
remains the riddle why it booted and probably loaded the initrd.
Why does it fail when it looks for the ISO filesystem ?

It would be helpful to know where exactly the installation startup
failed with what message exactly.

I no longer have a failing setup, but this Ubuntu user was seeing the exact same thing I was seeing, except in Debian words/colors:

 https://askubuntu.com/questions/127398/usb-drive-install-of-ubuntu-12-04-server-fails-cant-find-components-from-cd-r

In case of link-rot, here's what he writes:

1. The computer boots up the installation process ok.
2. It gets through the Ubuntu language, locale and keyboard selection.
3. Then starts loading additional components. At this point it gets about a quarter of the way through then throws big error message saying:

[!!] Load installer components from CD

There was a problem reading data from the CD-ROM. Please make sure it is in the drive. If retrying does not work., you should check the integrity of your CD-ROM.

Failed to copy file from CD-ROM. Retry?

Have a nice day :)

Thanks! You, too! 

Thomas

[Thomas Schmitt]

Hi,

Kent West wrote:
> Warning: The driver descriptor says the physical block size is 2048 bytes,
> but Linux says it is 512 bytes.

The Apple Partition Map block size is 2048 indeed. Else it could not
coexist with the GPT.

> Disk /dev/sdc: 62.7GB
> Sector size (logical/physical): 2048B/512B
> Partition Table: mac

> Number  Start   End     Size   File system  Name   Flags
>  1      2048B   6143B   4096B               Apple

This is the Apple Partition Map itself.

>  2      3693kB  4119kB  426kB               EFI

This is APM partition 1. It marks the same range as MBR partition 2
and GPT partition 2. In there is the FAT filesystem with the start programs
for EFI.

With

/sbin/fdisk -l /dev/sdc

you would see the MBR partitions. One of type "Empty" and one of type EFI".

> this Ubuntu user was seeing the exact same thing I was seeing, excepti

> in Debian words/colors:
> https://askubuntu.com/questions/127398/usb-drive-install-of-ubuntu-12-04-s
erver-fails-cant-find-components-from-cd-r

Did you get a complaint about a failed integrity test ?

Have a nice day :)

Thomas

[Rick Thomas]

Hi Kent,

After doing the cp or dd to write the .iso to the USB, do you do a “sync” before you eject it? Writing to a USB stick can seem to go quite fast, but that’s because of buffering. Often it takes quite a while (a minute or more for a very big write on a machine with plenty of RAM) to clear the buffers and write thru to permanent media.

Just a thought,
Rick

PS: Here’s what I use to write a .iso file to a USB stick:
dd if=/debian-live-9.4.0-amd64-mate.iso of=/dev/sdc bs=1M conv=fsync status=progress
sync
eject /dev/sdc
Hope it helps!

[Kenneth Parker]

I was burned once by this sort of Foolishness.  (Some USB Drives even have "Blinky Lights", to tell when Activity occurs).  I do the File Copy, it comes back to a prompt right away.  But then, when I say "sync", it shows how slow Access is, to writing to some of these USB Drives.  

Never pull out a USB, until you are *SURE* it is safe to do so.  That was a good lesson for me!

Kenneth Parker

[Brian]

On Fri 11 May 2018 at 15:49:59 -0500, Kent West wrote:

> I no longer have a failing setup, but this Ubuntu user was seeing the exact
> same thing I was seeing, except in Debian words/colors:
>
>
> https://askubuntu.com/questions/127398/usb-drive-install-of-ubuntu-12-04-server-fails-cant-find-components-from-cd-r
>
> In case of link-rot, here's what he writes:
>
>

> 1. The computer boots up the installation process ok.
> 2. It gets through the Ubuntu language, locale and keyboard selection.
> 3. Then starts loading additional components. At this point it gets

> about a quarter of the way through then throws big error message saying:
>

> *[!!] Load installer components from CD*

>
> There was a problem reading data from the CD-ROM. Please make sure it is in
> the drive. If retrying does not work., you should check the integrity of
> your CD-ROM.
>
> Failed to copy file from CD-ROM. Retry?

This user used Universal-USB-Installer-1.8.9.4 to write a Ubuntu 12.04
Server (32bit) image to a USB stick - a somewhat different technique from
yours. The tool used to create the USB image quite possibly modified and
damaged the image when writing it out.

AfAIK, there is no well-documented account of dd, cat or cp reproducibly
failing to write a Debian installation image properly to a USB device.

--
Brian.

[Pascal Hambourg]

Le 12/05/2018 à 01:04, Rick Thomas a écrit :
>
> After doing the cp or dd to write the .iso to the USB, do you do a “sync” before you eject it?

I don't, because I don't feel the need to.
According to its man page description, sync "flush file system buffers",
but the destination is a raw device, not a filesystem, so filesystem
buffers should not be involved.

> Writing to a USB stick can seem to go quite fast

Not in my experience when writing to the raw USB device.
The USB drives I use have an activity light. It stops blinking as soon
as the dd or cp command terminates. I also monitor block I/O with vmstat
and block writes drop as soon as dd or cp terminates. If I run sync, I
do not see any more activity from the light or vmstat.

[Pascal Hambourg]

Le 11/05/2018 à 22:38, Thomas Schmitt a écrit :
>
> Pascal Hambourg wrote:
>> It also has Apple and GPT partition tables, but they are bogus
>
> The GPT is not valid because there is a non-"protective" MBR partition
> table. The APM is valid, but should be of no interest for any firmware
> that does not expect a HFS or HFS+ filesystem. (And there is no such
> filesystem in the ISO image.)

Why then does parted complain about a block size discrepancy ?

> But if only a few MB were copied (for what strange reason ever) there
> remains the riddle why it booted and probably loaded the initrd.
> Why does it fail when it looks for the ISO filesystem ?

Probably because the MBR and the boot files are located at the beginning
of the image and were part of the data which were actually written.

[Thomas Schmitt]

Hi,

Pascal Hambourg wrote:
> Why then does parted complain about a block size discrepancy ?

Because the Apple Partition Map announces to count blocks with size 2048
whereas the Linux device file announces 512 (via ioctl(BLKSSZGET) ?)).

It is quite a poor choice of parted to hop on the Apple Partition Map
rather than onto the MBR partition map, which will be of interest for
much more firmwares and operating systems.

(I should really pester debian-cd more about that inappropriate option
-isohybrid-apm-hfsplus in the xorriso run.)

I wrote:
> > > But if only a few MB were copied (for what strange reason ever) there
> > remains the riddle why it booted and probably loaded the initrd.
> > Why does it fail when it looks for the ISO filesystem ?

> Probably because the MBR and the boot files are located at the beginning of
> the image and were part of the data which were actually written.

Well, Kent obviously experienced a running Linux kernel and the attempt to
execute software from the initrd.

The message "Load installer components from CD" can be found in initrd.gz
as package Description which belongs to "load-cdrom"
https://packages.debian.org/stretch/load-cdrom
This package is not to see as file in the ISO.

It could be that there is no problem with finding the "CD-ROM" but rather
with installing or activating that package. So maybe it could not even
begin its work.

About its role in debian-installer
https://d-i.alioth.debian.org/doc/talks/debconf6/paper/
says
"The first three stages are where the fundamental difference between
installation methods can be seen. All components (udebs) used there
need to be included in the initrd[2] with which the installer is booted."
and lists
"load-cdrom (anna) [...] Retrieve and unpack additional components"
as part of stage 2.

-----------------------------------------------------------------------

So the problem obviously occurs while the installer is still using the initrd. The filename in the ISO is /install.amd/gtk/initrd.gz or
/install.amd/initrd.gz.

In debian-9.4.0-amd64-xfce-CD-1.iso the initrds and the kernel are stored in
the lower blocks indeed. ("Startlba" is the logical block address with 2048
bytes per block. "Blocks" is the number of occupied blocks.):

Startlba , Blocks , Filesize , ISO image path
===============================================
...
5642 , 19182 , 39282964 , '/install.amd/gtk/initrd.gz
24824 , 1 , 66 , '/install.amd/gtk/install.b
24825 , 2063 , 4224800 , '/install.amd/gtk/vmlinuz'
24825 , 2063 , 4224800 , '/install.amd/vmlinuz'
26888 , 7667 , 15700220 , '/install.amd/initrd.gz'
34555 , 1 , 45 , '/install.amd/install.bat'
...

In debian-buster-DI-alpha2-amd64-xfce-CD-1.iso :

Startlba , Blocks , Filesize , ISO image path
===============================================
...
5694 , 19570 , 40077490 , '/install.amd/gtk/initrd.gz'
25264 , 1 , 66 , '/install.amd/gtk/install.bat'
25265 , 2181 , 4466448 , '/install.amd/gtk/vmlinuz'
25265 , 2181 , 4466448 , '/install.amd/vmlinuz'
27446 , 7811 , 15995172 , '/install.amd/initrd.gz'
35257 , 1 , 45 , '/install.amd/install.bat'
...

The messages

Load installer components from CD

There was a problem reading data from the CD-ROM.

are well known to the web. But occurences show no pattern about particularly
bad releases or spin-off distros. The reported remedies are not much
convincing as technical improvements, but they all involve re-copying the
ISO onto USB stick.
I.e. they all could simply have helped by trying again and having more
success with copying.


Kent could try to damage the USB stick content for experiments.

First:

# Damage ISO after the end of initrds and kernel
dd if=/dev/zero bs=2048 count=350000 seek=50000 of=/dev/sdc

And after checking out the effects of that damage:

# Damage the GTK initrd itself
dd if=/dev/zero bs=2048 count=40000 seek=10000 of=/dev/sdc

(This does not damage the USB stick. Only its content data.)

-----------------------------------------------------------------------

[Mike Kupfer]

Kent West wrote:

> I have a Dell Latitude E7250 laptop. I'm trying to install Debian to it using a USB stick.

[...]

> The real problem is that after going through the first three or four
> screens, the install halts, complaining about not being able to read
> the CD-ROM.

Yes, I ran into the same problem with my E7250, using a netinst image
and a live image.

When trying to boot the live image I noticed these messages on the
console:

usb 1-2: device descriptor read/64, error -110
...
usb 1-2: reset high-speed USB device number 2 using xhci_hcd

I tried booting from a different USB port; that worked once, but on a
subsequent attempt it failed with the same issue.

I did eventually get Debian installed, using a convoluted process that I
won't describe here because I suspect it's not relevant (see below).
I'm afraid that once I got Debian installed, I never went back to figure
out what was causing the earlier failures.

After reading your posting, I tried again to create a live USB stick. I
used cp to copy the image to /dev/sdb, and I ran sync afterwards. Same
failure.

I then tried using dd instead of cp, and I used a different USB port on
my desktop to write to the USB stick. This time I could boot the live
image. I checked kern.log, and it had

usb 2-2: device descriptor read/64, error -110

but did not have the "reset" message. (And I notice that this time
it's "usb 2-2" rather than "usb 1-2".)

I tried to boot off the USB stick a second time, and this time I got the
failure again. So I suspect the problem has to do with peculiarities of
the E7250, rather than issues writing the USB stick.

I have a copy of kern.log from the first (successful) boot, and I have
the dmesg output from the second (failed) boot. If anyone wants to look
at them, let me know.

mike

[Pascal Hambourg]

Le 12/05/2018 à 21:38, Thomas Schmitt a écrit :
>
> Pascal Hambourg wrote:
>> Why then does parted complain about a block size discrepancy ?
>
> Because the Apple Partition Map announces to count blocks with size 2048
> whereas the Linux device file announces 512 (via ioctl(BLKSSZGET) ?)).

I already know that, and this is not what I was asking.
I have never seen a hard disk or flash drive with a sector size of 2048
(only 512 or 4096) so the kernel must be correct and the Apple partition
table must be wrong. Despite this, you claim that the Apple partition
table is valid. Can you please explain why the wrong block size does not
matter ?

> It is quite a poor choice of parted to hop on the Apple Partition Map
> rather than onto the MBR partition map, which will be of interest for
> much more firmwares and operating systems.

Unfortunately this is not the only poor choice of parted and friends
based on libparted. At least it could provide an option to force the
partition table type to use, as fdisk or gdisk do.

> (I should really pester debian-cd more about that inappropriate option
> -isohybrid-apm-hfsplus in the xorriso run.)

The only storage media with a bloc size of 2048 I know about are optical
media such as CD and DVD.
Could the Apple partition map in the image be used when booting from an
optical disk on a Mac ?

[Curt]

On 2018-05-13, Mike Kupfer <mku...@alum.berkeley.edu> wrote:
>
> usb 1-2: device descriptor read/64, error -110
> ...
> usb 1-2: reset high-speed USB device number 2 using xhci_hcd
>
> I tried booting from a different USB port; that worked once, but on a
> subsequent attempt it failed with the same issue.
>

I've mounted usb sticks that spontaneously 'umounted' themselves while
producing similar I/O reset errors.

Which might conceivably explain why the installer would begin asking for
the cdrom; when it does so, whether the usb stick is mounted or not
might be a clue. Then again, mounted or not, if it can't be read the
distinction between mounted or unmounted is possibly of little import.

Maybe someone could boot the installer and in the early stages
effectuate a 'hard eject' of the thumb drive (by ripping it out by the
roots) to observe the manner in which the installer reacts (does it ask for
the insertion of the 'cdrom', or do tectonic plates shift along
unsuspected fault lines, eliminating the user entirely)?

[Thomas Schmitt]

Hi,

Pascal Hambourg wrote:
> I have never seen a hard disk or flash drive with a sector size of 2048
> (only 512 or 4096) so the kernel must be correct and the Apple partition
> table must be wrong.

Both are correct. The kernel reports what it perceives as hardware block
size ("physical"). The Apple Partition Map is free to use one of several
block sizes. (In case of a "mjg" isohybrid it is 2048 in order to give
room for the GPT header block at byte 512 to 1023. The first APM entry
then begins at byte 2048.)

It is the partition editor who is clueless. (One wonders how those editors
get to their hunchbacked representations from quite clear partition table
specs.)

> Can you please explain why the wrong block size does not matter ?

Block size matters in various aspects. But these aspects refer to own info
sources which do not necessarily tell the same number.
In a Debian x86 isohybrid we have the following block sizes:

- The MBR partition table assumes block size 512. To be used when converting
Logical Block Address and block count to the byte offsets of partitions.

- The El Torito Sector Count assumes block size 512. To be used when loading
the BIOS boot image into memory, or when accepting the EFI boot image as
EFI System Partition.

- The El Torito Load RBA (start block of the boot image) assumes block size
2048. To be used when loading or accepting the boot image.

- The Apple Partition Map announces block size 2048. To be used when
converting block addresses and counts of the APM to byte offsets.

- The ISO 9660 filesystem announces a block size of 2048. To be used when
converting block addresses and counts of the ISO metadata to byte offsets.

- Then we have the FAT filesystem in the EFI boot image. Probably it uses
a block size of 512 bytes.

Each interpreter of those structures needs to get told the matching block
size or needs to assume it correctly without being told.

> The only storage media with a bloc size of 2048 I know about are optical
> media such as CD and DVD.

Readers of ISO 9660 should obey the block size given in the Primary Volume
Descriptor. But all i know (including mine) will only work with logical
block size 2048.

The block size used with filesystem or partition table is quite independent
of the block size which the device assumes for its Logical Block Addresses.
(I think Linux demands multiples of 512 by a power of two: 512, 1024, 2048,
4096, ...)
The kernel converts the logical block addressies of filesystem or partition
table to the logical block addresses of the storage device.

(Many storage devices expose a logical block size of 512 while really having
larger storage chunks.)

> Could the Apple partition map in the image be used when booting from an
> optical disk on a Mac ?

Hardly. One can never be sure with firmware behavior, though.

Matthew J. Garrett ("mjg") introduced the APM into Fedora isohybrids in
order to mark a small HFS+ filesystem image file inside the ISO.
But Debian has no such HFS+ image, and thus does not serve that obscure
class of Apple machines which probably predate Apple's adoption of EFI.

In Fedora-Workstation-netinst-x86_64-23.iso one can see:

MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 0 845824
MBR partition : 2 0x00 0xef 180 10780
MBR partition : 3 0x00 0x00 684488 43168
MBR partition path : 2 /images/efiboot.img
MBR partition path : 3 /images/macboot.img
...
APM : N Info
APM block size : 2048
APM gap fillers : 0
APM partition name : 1 EFI
APM partition type : 1 Apple_HFS
APM start and size : 1 45 2695
APM partition path : 1 /images/efiboot.img
APM partition name : 2 EFI
APM partition type : 2 Apple_HFS
APM start and size : 2 171122 10792
APM partition path : 2 /images/macboot.img
...

Both partition tables mark the ISO files efiboot.img and macboot.img as
partitions. The addresses match when interpreted by appropriate block sizes:
180 = 45 * 4
684488 = 171122 * 4

When debian-cd adopted this layout, the HFS+ image was omitted, but the
option to mark the EFI image in Apple Partition Map staid with the xorriso
run. So we still see an APM in debian-cd amd64 and i386 ISOs.
E.g. in debian-9.4.0-amd64-xfce-CD-1.iso :

APM : N Info
APM block size : 2048
APM gap fillers : 0
APM partition name : 1 EFI
APM partition type : 1 Apple_HFS
APM start and size : 1 1835 208
APM partition path : 1 /boot/grub/efi.img


Matthew described his invention in
http://mjg59.dreamwidth.org/11285.html

[Thomas Schmitt]

Hi,

Mike Kupfer wrote:
> usb 1-2: device descriptor read/64, error -110

If we assume that 110 is a Linux errno, then this would be
ETIMEDOUT 110 /* Connection timed out */

> usb 1-2: reset high-speed USB device number 2 using xhci_hcd

That's what my USB attached DVD drives sometimes do after having gnawed
a long time on a read error.

> I have a copy of kern.log from the first (successful) boot, and I have
> the dmesg output from the second (failed) boot. If anyone wants to look
> at them, let me know.

I am interested. Especially whether there are messages from the ISO 9660
filesystem driver.

Curt wrote:
> I've mounted usb sticks that spontaneously 'umounted' themselves while
> producing similar I/O reset errors.
> Which might conceivably explain why the installer would begin asking for
> the cdrom;

The messages quoted by Kent could well indicate that the "CD-ROM" was found
but that rather a particular file was not readable or did not yield the
expected checksum.

It would be helpful if the error messages contained the name of the
failing software and the particular kind of read problem.

[Curt]

On 2018-05-13, Thomas Schmitt <scdb...@gmx.net> wrote:
>
> Curt wrote:
>> I've mounted usb sticks that spontaneously 'umounted' themselves while
>> producing similar I/O reset errors.
>> Which might conceivably explain why the installer would begin asking for
>> the cdrom;
>
> The messages quoted by Kent could well indicate that the "CD-ROM" was found
> but that rather a particular file was not readable or did not yield the
> expected checksum.

I was laboring under another erroneous impression for some reason, but
looking back at the OP he did say that the installer complained about
not being able to "read" the cdrom (not that there was none inserted).
Of course there was never a cdrom to read in the first place (which
could throw off lesser lights than some of you here in this forum), but
I guess we know 'it' means the thumb drive (because it can't or doesn't
care to distinguish between the two media).

A verbatim quote from Ms. Installer might or might not be edifying.

Perhaps it said something to this effect:

"There was an error reading data from the CD-ROM. Please make sure it
is in the drive. If retrying does not work, you should check the
integrity of your CD-ROM."

Apparently there is (or was) an option to check the integrity of the 'cdrom'
at that point (read failure) in the installer.

> It would be helpful if the error messages contained the name of the
> failing software and the particular kind of read problem.

Any gritty details would probably be in the logs.

>
> Have a nice day :)
>
> Thomas
>
>

--

[Thomas Schmitt]

Hi,

i wrote:
> > The messages quoted by Kent could well indicate that the "CD-ROM" was
> > found

Curt wrote:
> I was laboring under another erroneous impression for some reason, but
> looking back at the OP he did say that the installer complained about
> not being able to "read" the cdrom (not that there was none inserted).

Possibly it was me who brought up the idea of "not found". That was because
the started kernel needs to find the device with the ISO 9660 filesystem
that contains the expected files outside the initrd. A good occasion for
Linux not to like the USB stick after firmware and bootloader had no problem.

But i began to doubt this theory when Kent (OP) pointed to the Askubuntu
question as example of the messages he saw:
https://askubuntu.com/questions/127398/usb-drive-install-of-ubuntu-12-04-server-fails-cant-find-components-from-cd-r


> Of course there was never a cdrom to read [...]

> I guess we know 'it' means the thumb drive

> "There was an error reading data from the CD-ROM. [...]"

Actually it should talk about the ISO 9660 filesystem of the installation
medium. "CD-ROM" is sooo Y2K. Not modern any more and not yet retro.

Saying "Please have all installation diskettes ready" - that would be stylish.

Back to technical: It seems that a lower software layer bails out without
specific error indication and a thus clueless upper layer tries to say
something useful about this unspecific failure.

Can it be that debian-installer should be more verbose ?
For example with the description of its components ?
https://packages.debian.org/en/stretch/anna
says its purpose is to be not nearly apt, but to do nevertheless.

> Apparently there is (or was) an option to check the integrity of the 'cdrom'
> at that point (read failure) in the installer.

I would love to believe this. But probably the best check is
https://www.debian.org/CD/faq/#verify
on the system which copied the ISO image onto the installation medium.

> Any gritty details would probably be in the logs.

Where should the next victim of this problem look ? (It is not very rare,
as the web tells me via google.)

[David Wright]

On Fri 11 May 2018 at 15:13:04 (-0500), Kent West wrote:
> On Fri, May 11, 2018 at 2:59 PM, Pascal Hambourg <pas...@plouf.fr.eu.org>
> wrote:
>
> > Le 11/05/2018 à 20:33, Kent West a écrit :
> >
> >>
> >> I learned that EFI boot drives need to have a GPT partition table. On a
> >>
> >
> > This is not correct. The UEFI specification supports boot from a drive
> > with an MSDOS partition table. Otherwise why would there be an "EFI system
> > partition" type identifier (0xef) for MSDOS partition tables ? The Debian
> > installer hybrid image has an MSDOS partition table. (It also has Apple and
> > GPT partition tables, but they are bogus)
>
>
> That's good to know. I guess my source material (
> https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/)
> is wrong. Or I misunderstood it.

While a lot of the detail on that long page might be correct, there
are also statements there that don't seem to agree with reality.

It does make a statement that agrees with Pascal's:

"and the UEFI spec requires that UEFI-compliant firmwares be
capable of interpreting GPT (it also requires them to be capable
of interpreting MBR, for backwards compatibility)." (under "The
GPT (GUID partition table) format").

That said, there are other statements that are odd:

"I really can’t recommend strongly enough that you do not attempt
to mix UEFI-native and BIOS-compatible booting of
permanently-installed operating systems on the same computer, and
especially not on the same disk. It is a terrible terrible idea
and will cause you heartache and pain. If you decide to do it,
don’t come crying to me." (under "UEFI booting: background").
--
"Disk formats (MBR vs. GPT)

Here’s another very important consideration:

If you want to do a ‘BIOS compatibility’ type installation, you
probably want to install to an MBR formatted disk. If you want to
do a UEFI native installation, you probably want to install to a
GPT formatted disk."
--
"A specific example

To boil down the above: if you bought a Windows 8 or later system,
you almost certainly have a UEFI native install of Windows to a
GPT-formatted disk. This means that if you want to install another
OS alongside that Windows install, you almost certainly want to do
a UEFI-native installation of your other OS. If you don’t like all
this UEFI nonsense and want to go back to the good old world
you’re familiar with, you will, I’m afraid, have to blow away the
UEFI-native Windows installation, and it would be a good idea to
reformat the disk to MBR."

I can't reconcile that with the system here, a Windows 8→10 UEFI laptop
and GPT disk running linux in BIOS compatibility mode (here called
Legacy mode by Lenovo) booting from an MBR on an ATA disk:

# gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 976773168 sectors, 465.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2653F13F-2CD0-406F-B004-6AFFDBE18127
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4077 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 2050047 1000.0 MiB 2700 Basic data partition
2 2050048 2582527 260.0 MiB EF00 EFI system partition
3 2582528 4630527 1000.0 MiB ED01 Basic data partition
4 4630528 4892671 128.0 MiB 0C01 Microsoft reserved ...
5 4892672 347348991 163.3 GiB 0700 Basic data partition
6 347348992 429268991 39.1 GiB 8300 Linux-A
7 429268992 511188991 39.1 GiB 8300 Linux-B
8 511188992 883275775 177.4 GiB 8300 Linux-Home
9 883275776 883292159 8.0 MiB EF02 Linux-BIOS-Boot
10 883292160 892084223 4.2 GiB 8200 Linux-Swap
11 892086272 892803071 350.0 MiB 2700
12 892803072 894900223 1024.0 MiB 0700 Basic data partition
13 894900224 947329023 25.0 GiB 0700 Basic data partition
14 947329024 976773119 14.0 GiB 2700 Basic data partition
#

=> Grub2 (v1.99) is installed in the MBR of /dev/sda and looks at sector
883275776 of the same hard drive for core.img. core.img is at this
location and looks in partition 85 for . [from bootinfoscript].

The original sda5 was shrunk, and five new partitions added, all by
windows. Then the machine was booted in compatibility mode and linux
was installed. Finally, I told windows not to assign a drive letter
to the linux partitions (so that it does not keep offering to reformat
them at every encounter).

The result is a laptop that boots windows in UEFI Boot Mode and
linux in Legacy Mode. Windows knows nothing about the linux
installation except to ignore those partitions. Linux mounts
what it can of the windows installation, but in readonly mode.
Switching over involves going through the "BIOS Setup", reached
by a separate button (almost recessed).

All this makes me treat the information on that page with suspicion.
Perhaps the statement quoted here is self-referential (but lower than
95%, I hope): "You’ve probably read a lot of stuff on the internet
about UEFI. Here is something important you should understand: 95% of
it was probably garbage."

Cheers,
David.

[Mike Kupfer]

Thomas Schmitt wrote:

> Mike Kupfer wrote:

> > I have a copy of kern.log from the first (successful) boot, and I have
> > the dmesg output from the second (failed) boot. If anyone wants to look
> > at them, let me know.
>
> I am interested. Especially whether there are messages from the ISO 9660
> filesystem driver.

(full files sent off-list)

Hmm. The bad log has

[ 7.015708] usbcore: registered new interface driver uas
[ 25.552030] random: crng init done

In the good log, there are several other messages in between:

May 13 00:47:48 localhost kernel: [ 7.530496] usbcore: registered new interface driver uas
May 13 00:47:48 localhost kernel: [ 8.552616] scsi 2:0:0:0: Direct-Access General UDisk 5.00 PQ: 0 ANSI: 2
May 13 00:47:48 localhost kernel: [ 8.555906] sd 2:0:0:0: [sdb] 15728640 512-byte logical blocks: (8.05 GB/7.50 GiB)
May 13 00:47:48 localhost kernel: [ 8.559691] sd 2:0:0:0: [sdb] Write Protect is off
May 13 00:47:48 localhost kernel: [ 8.562143] sd 2:0:0:0: [sdb] Mode Sense: 0b 00 00 08
May 13 00:47:48 localhost kernel: [ 8.562329] sd 2:0:0:0: [sdb] No Caching mode page found
May 13 00:47:48 localhost kernel: [ 8.564222] sd 2:0:0:0: [sdb] Assuming drive cache: write through
May 13 00:47:48 localhost kernel: [ 8.616686] sdb: sdb1 sdb2
May 13 00:47:48 localhost kernel: [ 8.618956] sd 2:0:0:0: [sdb] Attached SCSI removable disk
May 13 00:47:48 localhost kernel: [ 8.842431] ISO 9660 Extensions: RRIP_1991A
May 13 00:47:48 localhost kernel: [ 8.854480] loop: module loaded
May 13 00:47:48 localhost kernel: [ 8.904560] squashfs: version 4.0 (2009/01/31) Phillip Lougher
May 13 00:47:48 localhost kernel: [ 9.769518] ip_tables: (C) 2000-2006 Netfilter Core Team
May 13 00:47:48 localhost kernel: [ 10.083646] random: crng init done

The other thing I noticed when comparing the logs is that the initial
USB-related messages are different, like the devices are being
recognized in a different order. I don't know if that's significant or
not.

> It would be helpful if the error messages contained the name of the
> failing software and the particular kind of read problem.

I'm afraid I didn't record the exact messages that I got during my
original attempts to install.

With the live image failures that I'm getting now, the console messages
are

[ 30.010394] usb 1-1.2: reset high-speed USB device number 3 using ehci-pci


BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) Unable to find a medium containing a live file system

mike

[Pascal Hambourg]

Le 13/05/2018 à 17:18, David Wright a écrit :
> On Fri 11 May 2018 at 15:13:04 (-0500), Kent West wrote:
>>
>> That's good to know. I guess my source material (
>> https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/)
>> is wrong. Or I misunderstood it.
>
> While a lot of the detail on that long page might be correct, there
> are also statements there that don't seem to agree with reality.

Most of the statements you quoted agree with my (admittedly limited)
experience with UEFI. There is a difference between the theory
(specifications) and the reality (implementations), and some pieces of
software may have extra requirements beyond the sole UEFI specification.

> "I really can’t recommend strongly enough that you do not attempt
> to mix UEFI-native and BIOS-compatible booting of
> permanently-installed operating systems on the same computer, and
> especially not on the same disk. It is a terrible terrible idea
> and will cause you heartache and pain. If you decide to do it,
> don’t come crying to me." (under "UEFI booting: background").

I would not be as much adamant as the author, but my experience says :
it can work, but expect trouble.

Most of my early experience with UEFI boot comes from a rather old Intel
motherboard. Beside crippled UEFI support (no UEFI boot from USB or SATA
in AHCI mode), it had a couple of annoying requirements :
- boot in legacy mode only if the MBR contains a partition entry with
the boot flag set, regardless of whether the disk has a MSDOS or GPT
partition table. This behaviour is beyond any common BIOS standard, but
I have observed it on many other systems, mostly Dell and HP ;
- boot in EFI mode from GPT only if the protective partition entry in
the MBR has the boot flag unset. I admit this requirement is part of the
GPT specification, but really do not see the point in enforcing such a
minor detail.

Anyway, these two requirements put together make it impossible to boot
in both legacy and EFI mode from the same GPT disk with this
motherboard. However they allow to boot in both modes from the same
MSDOS disk. But who still wants to use MSDOS format nowadays ?

> "Disk formats (MBR vs. GPT)
>
> Here’s another very important consideration:
>
> If you want to do a ‘BIOS compatibility’ type installation, you
> probably want to install to an MBR formatted disk. If you want to
> do a UEFI native installation, you probably want to install to a
> GPT formatted disk."

I do not agree so much with this one when it comes to install GNU/Linux.
But it is an absolute requirement when installing Windows.

> "A specific example
>
> To boil down the above: if you bought a Windows 8 or later system,
> you almost certainly have a UEFI native install of Windows to a
> GPT-formatted disk. This means that if you want to install another
> OS alongside that Windows install, you almost certainly want to do
> a UEFI-native installation of your other OS. If you don’t like all
> this UEFI nonsense and want to go back to the good old world
> you’re familiar with, you will, I’m afraid, have to blow away the
> UEFI-native Windows installation, and it would be a good idea to
> reformat the disk to MBR."

I agree with the author. If you want to keep the existing EFI Windows
installation and have a convenient dual boot with GRUB, you'll have to
set up your favourite distribution to boot in EFI mode. If you want to
go back to legacy boot, including for Windows, you'll have to
repartition the disk to MSDOS format and reinstall Windows.

> I can't reconcile that with the system here, a Windows 8→10 UEFI laptop
> and GPT disk running linux in BIOS compatibility mode (here called
> Legacy mode by Lenovo) booting from an MBR on an ATA disk:

That is not very convenient, is it ? You cannot boot Windows boot
manager from GRUB nor you can boot GRUB from Windows boot manager and
must select the boot mode in the UEFI firmware setup whenever you want
to switch the operating system.

> Switching over involves going through the "BIOS Setup", reached
> by a separate button (almost recessed).

As expected.

[Thomas Schmitt]

Hi,

Mike Kupfer wrote:
> (full files sent off-list)

Received.

> Hmm. The bad log has
>
> [ 7.015708] usbcore: registered new interface driver uas
> [ 25.552030] random: crng init done

It is quite sparse from that point on:

[ 29.034207] usb 1-2: reset high-speed USB device number 2 using xhci_hcd

and then inactivity for 220 seconds which ended by unrelated messages.

Before the message at 7.015708 the two logs show a short sequence of
agreement: "Product", "Manufacturer", "SerialNumber".

The USB device is not reported in the bad log as SCSI device sdX.
In the good log this is done as "sdb" with partitions "sdb1" and "sdb2".
Then the filesystem is detected

[ 8.842431] ISO 9660 Extensions: RRIP_1991A

A Rock Ridge enhanced ISO with partitions is probably a bootable isohybrid
filesystem. So that's what we should expect to see.


Both logs have the "error -110" and do not report the stick's serial number.
The inactivity before both error messages was 3.5 to 4 seconds.
The bad log has two suspicious lines before that range:
[ 6.997775] usb 1-2: New USB device found, idVendor=abcd, idProduct=1234
[ 7.000181] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Looks much like phony default values.
They are missing in the good log. Other kernel version ?

So it could be that the USB stick does not answer to this inquiry within a
short timeout threshold. But this seems not to hamper the further handling
of the device in the good log.

---------------------------------------------------------------------------

> With the live image failures that I'm getting now, the console messages
> are
>
> [ 30.010394] usb 1-1.2: reset high-speed USB device number 3 using
ehci-pci

This seems to be the worst event better than the stick falling out of the
USB socket.

> BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
> Enter 'help' for a list of built-in commands.
>
> (initramfs) Unable to find a medium containing a live file system

This matches the bad log. The stick is not registred as SCSI device and
thus not accessible for reading or writing data.

We shall note that this is not what Kent West, the OP, reports for his
problems. If his problem is hardware related, then it is more fuzzy
than yours.

[Mike Kupfer]

Thomas Schmitt wrote:

> The bad log has two suspicious lines before that range:
> [ 6.997775] usb 1-2: New USB device found, idVendor=abcd, idProduct=1234
> [ 7.000181] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> Looks much like phony default values.
> They are missing in the good log. Other kernel version ?

I'm not sure I understand the question. The good and bad logs are both
for the same kernel--they're different attempts to boot the same ISO
using the same USB stick.

> > BusyBox v1.22.1 (Debian 1:1.22.0-19+b3) built-in shell (ash)
> > Enter 'help' for a list of built-in commands.
> >
> > (initramfs) Unable to find a medium containing a live file system
>
> This matches the bad log. The stick is not registred as SCSI device and
> thus not accessible for reading or writing data.
>
> We shall note that this is not what Kent West, the OP, reports for his
> problems. If his problem is hardware related, then it is more fuzzy
> than yours.

This prompted me to go back to try to reproduce the CD-ROM thing. And
yes, I can reproduce a failure that sounds like the one that was
originally reported. If I select the installer (rather than the live
image) when booting from the USB stick, it gets through the 3 i18n
screens (language, location, keyboard layout), pauses, and then brings
up a screen with

-----8<-----8<-----
[!!] Detect and mount CD-ROM

No common CD-ROM drive was detected.

You may need to load additional CD-ROM drivers from removable media,
such as a driver floppy. If you have such media available now, insert it
and continue. Otherwise, you will be given the option to manually
select CD-ROM modules.

Load CD-ROM drivers from removable media?

<Yes> <No>
----->8----->8-----

This may be related to device initialization by the BIOS. I believe
there's a warning about this in the installation guide somewhere. But
even with my BIOS set to "Thorough" initialization, I still can get this
failure. It seems harder to get a successful boot (installer or live
image) if the system has been on recently.

In any case, I suspect that my live image boot failures are due to the
same root cause as the installer failure.

mike

PS. The BIOS setting I'm referring to is POST Behavior/Fastboot. The
choices are Minimal, Auto, and Thorough, with Minimal being the default
IIRC.

[David Wright]

In the case of GPT, I assume the partition entry with the boot flag set
is the protective MBR.

> - boot in EFI mode from GPT only if the protective partition entry
> in the MBR has the boot flag unset. I admit this requirement is part
> of the GPT specification, but really do not see the point in
> enforcing such a minor detail.
>
> Anyway, these two requirements put together make it impossible to
> boot in both legacy and EFI mode from the same GPT disk with this
> motherboard. However they allow to boot in both modes from the same
> MSDOS disk. But who still wants to use MSDOS format nowadays ?

Is it impossible, then, to change the boot flag in a protective MBR?

> > "Disk formats (MBR vs. GPT)
> >
> > Here’s another very important consideration:
> >
> > If you want to do a ‘BIOS compatibility’ type installation, you
> > probably want to install to an MBR formatted disk. If you want to
> > do a UEFI native installation, you probably want to install to a
> > GPT formatted disk."
>
> I do not agree so much with this one when it comes to install
> GNU/Linux. But it is an absolute requirement when installing
> Windows.

Yes, though I assume few people install Windows. It's more likely to
be pre-installed.

> > "A specific example
> >
> > To boil down the above: if you bought a Windows 8 or later system,
> > you almost certainly have a UEFI native install of Windows to a
> > GPT-formatted disk. This means that if you want to install another
> > OS alongside that Windows install, you almost certainly want to do
> > a UEFI-native installation of your other OS. If you don’t like all
> > this UEFI nonsense and want to go back to the good old world
> > you’re familiar with, you will, I’m afraid, have to blow away the
> > UEFI-native Windows installation, and it would be a good idea to
> > reformat the disk to MBR."
>
> I agree with the author. If you want to keep the existing EFI
> Windows installation and have a convenient dual boot with GRUB,
> you'll have to set up your favourite distribution to boot in EFI
> mode. If you want to go back to legacy boot, including for Windows,
> you'll have to repartition the disk to MSDOS format and reinstall
> Windows.

"convenient dual boot with GRUB" moves the goalposts.

> >I can't reconcile that with the system here, a Windows 8→10 UEFI laptop
> >and GPT disk running linux in BIOS compatibility mode (here called
> >Legacy mode by Lenovo) booting from an MBR on an ATA disk:
>
> That is not very convenient, is it ? You cannot boot Windows boot
> manager from GRUB nor you can boot GRUB from Windows boot manager
> and must select the boot mode in the UEFI firmware setup whenever
> you want to switch the operating system.

It's very convenient for me. It means I haven't had to interfere with
the way windows chooses to boot, or its configuration of the disk, at all.

> >Switching over involves going through the "BIOS Setup", reached
> >by a separate button (almost recessed).
>
> As expected.

Not by the author, who would have me reformat the disk as MBR and then
install windows…from where exactly?

Cheers,
David.

[songbird]

Pascal Hambourg wrote:
...

> I agree with the author. If you want to keep the existing EFI Windows
> installation and have a convenient dual boot with GRUB, you'll have to
> set up your favourite distribution to boot in EFI mode. If you want to
> go back to legacy boot, including for Windows, you'll have to
> repartition the disk to MSDOS format and reinstall Windows.

all i know is that if your bios doesn't boot in
UEFI mode and you don't know anything about what this
means you can end up installing Debian without UEFI
support and then it can be rather fun to get it back.

i managed to have grub do an install to a stable
partition without UEFI and i messed up the testing
setup i had. it took me some while to figure out
what went wrong and how to fix it. if you don't
really understand grub rescue commands and there
isn't a working system you can use to connect and
find help for the commands you need to enter it's
very frustrating.

the Debian UEFI pages helped a great deal but
there were other things i had to figure out coming in
cold to UEFI.

how to create a /boot/efi partition, what goes in
it, mounting it, clearing and putting in new efibootmgr
entries, etc.

refind was useful and at least it does what i expect
it to do. grub, i dislike how it assumed things i
didn't want to do. alas, i didn't know how different
UEFI was from bios mode.

i still haven't redone my efibootmgr entries but
refind doesn't care, i can create custom entries in
that config file and they work that is all i really
need at this point.


songbird

[Thomas Schmitt]

Hi,

i wrote:
> > [ 6.997775] usb 1-2: New USB device found, idVendor=abcd, idProduct=1234

> > They are missing in the good log. Other kernel version ?

> The good and bad logs are both
> for the same kernel--they're different attempts to boot the same ISO
> using the same USB stick.

Er. Suddenly i can see the "abcd" "1234" message in the good log too.
Sorry for the confusion.

Just to make it clear:
Was this kernel loaded from the very same USB stick which it then cannot
inquire properly and in some cases cannot read from ?

> This prompted me to go back to try to reproduce the CD-ROM thing. And
> yes, I can reproduce a failure that sounds like the one that was
> originally reported.

> [!!] Detect and mount CD-ROM
> No common CD-ROM drive was detected.

This does not match the complaint in the web to which Kent pointed us
as example.
The one by your machine is quite clear. The storage device with the
ISO filesystem cannot be found by the kernel.

In Kent's case it seems rather that some particular file cannot be
read from the already found storage device.

> This may be related to device initialization by the BIOS.

If the kernel was loaded by the bootloader from that USB stick, then
the subsequent failures in the kernel run could be caused by some bad
state of the USB stick when the bootloader starts the kernel.

If booting was done via EFI (and not Legacy BIOS mode) then we could
ask at grub-...@gnu.org, whether the problem is known in any way.
(With BIOS mode we'd need to go to quite dead sysl...@zytor.com )

How does the stick behave if you plug it into a running GNU/Linux system ?
Any problem messages in the log ? Is it usable as storage device ?

[Chris Ramsden]

Hmm, do you have any useful references?

I got a new Dell computer, shrunk the existing partitions down and
successfully installed grub2 and got a windows10/Linux multi boot using
grub. Then later I tried to upgrade my Linux and soon found that I was
getting error messages about grub not being able to find necessary
features on the boot device.

I tried to rebuild it with a clean install of Windows 10, reasoning that
if I could get it back to the original configuration, I could repeat the
original exercise. But alas, no, it remains stubbornly unable to install
grub2 alongside the windows bootloader. I got it to a state where I
could use the BIOS POST boot screen to choose a boot option, but this
wasn't the original successful arrangement where grub offered me the
Linux/windows loader choice.

I gave up, wiped windows and went through with a clean Linux install. I
don't really want windows that much, but it irks me that I haven't been
able to fathom out how to return to the original state in which it was
shipped. Your words hint at many things I became vaguely aware of but
totally failed to grasp. The other posters to this thread have at least
reassured me that it isn't easy or trivial to get right.

--
Chris

[Stefan Monnier]

> That said, there are other statements that are odd:

Not sure what you find odd about them:

> "I really can’t recommend strongly enough that you do not attempt
> to mix UEFI-native and BIOS-compatible booting of
> permanently-installed operating systems on the same computer, and
> especially not on the same disk. It is a terrible terrible idea
> and will cause you heartache and pain. If you decide to do it,
> don’t come crying to me." (under "UEFI booting: background").

Here he's just saying that there's a good chance you'll encounter
difficulties if you try that, and indeed in my experience the behavior
of such a config will depend on undocumented details.

> "Disk formats (MBR vs. GPT)
>
> Here’s another very important consideration:
>
> If you want to do a ‘BIOS compatibility’ type installation, you
> probably want to install to an MBR formatted disk. If you want to
> do a UEFI native installation, you probably want to install to a
> GPT formatted disk."

Same here (basically for the same reason: the behavior of the firmware
and OS when faced with a disk that has both a GPT and an MBR partitions
is largely unspecified and will vary depending on your system).

> I can't reconcile that with the system here, a Windows 8→10 UEFI laptop
> and GPT disk running linux in BIOS compatibility mode (here called
> Legacy mode by Lenovo) booting from an MBR on an ATA disk:

It's easy to reconcile: he doesn't say your setup is impossible or can't
work, he just recommends not to do that because you may encounter
unexpected difficulties. E.g. in theory an upgrade to your firmware or
to one of your OSes could break it, tho in practice you're probably OK
at least until you move that setup to another machine with
a different firmware.


Stefan

[David Wright]

Would I be correct in thinking that the BIOS POST boot screen is
what you get when you hit F12 sufficiently quickly after switch-on?
So are you choosing between UEFI and Legacy (compatibility) mode.
(I would like to know how Dell handles what I've been reporting on
with this Lenovo.)

> I gave up, wiped windows and went through with a clean Linux install. I
> don't really want windows that much, but it irks me that I haven't been
> able to fathom out how to return to the original state in which it was
> shipped. Your words hint at many things I became vaguely aware of but
> totally failed to grasp. The other posters to this thread have at least
> reassured me that it isn't easy or trivial to get right.

This is the scenario I was trying to avoid. As far as windows was
concerned, my mantra was Failure Is Not An Option.

Cheers,
David.

[David Wright]

On Mon 14 May 2018 at 09:14:23 (-0400), Stefan Monnier wrote:
> > That said, there are other statements that are odd:
>
> Not sure what you find odd about them:
>
> > "I really can’t recommend strongly enough that you do not attempt
> > to mix UEFI-native and BIOS-compatible booting of
> > permanently-installed operating systems on the same computer, and
> > especially not on the same disk. It is a terrible terrible idea
> > and will cause you heartache and pain. If you decide to do it,
> > don’t come crying to me." (under "UEFI booting: background").
>
> Here he's just saying that there's a good chance you'll encounter
> difficulties if you try that, and indeed in my experience the behavior
> of such a config will depend on undocumented details.

Yes, documentation of firmware is almost unknown in my experience
(since probably 30 years ago). That's why I took the least invasive
method that I could. Using a UEFI/UEFI approach means that you have
to explore the manufacturer's undocumented implementation of UEFI.
Plenty of horror stories there, including Pascal's.

> > "Disk formats (MBR vs. GPT)
> >
> > Here’s another very important consideration:
> >
> > If you want to do a ‘BIOS compatibility’ type installation, you
> > probably want to install to an MBR formatted disk. If you want to
> > do a UEFI native installation, you probably want to install to a
> > GPT formatted disk."
>
> Same here (basically for the same reason: the behavior of the firmware
> and OS when faced with a disk that has both a GPT and an MBR partitions
> is largely unspecified and will vary depending on your system).

Eh? I've yet to see a GPT disk that didn't have a protective MBR.
I thought that's the reason why GPT starts at block 1 and not
block 0: an MBR was designed into GPT from the start (no pun intended).

> > I can't reconcile that with the system here, a Windows 8→10 UEFI laptop
> > and GPT disk running linux in BIOS compatibility mode (here called
> > Legacy mode by Lenovo) booting from an MBR on an ATA disk:
>
> It's easy to reconcile: he doesn't say your setup is impossible or can't
> work, he just recommends not to do that because you may encounter
> unexpected difficulties. E.g. in theory an upgrade to your firmware or
> to one of your OSes could break it, tho in practice you're probably OK
> at least until you move that setup to another machine with
> a different firmware.

Not sure what you mean here. It's a laptop: nowt's going nowhere.

But in a page as long as this one is, I think the author is rather
dismissive of using Legacy mode at all. Perhaps the clue is here:

"Don’t do UEFI-native installs to MBR-formatted disks, or BIOS
compatibility installs to GPT-formatted disks (an exception to the
latter is if your disk is, IIRC, 2.2+TB in size, because the MBR
format can’t handle disks that big – if you want to do a BIOS
compatibility install to a disk that big, you’re kinda stuck with
the BIOS+GPT combination, which works but is a bit wonky and
involves the infamous ‘BIOS Boot partition’ you may recall from
Fedora 17).

I haven't been able to find anything infamous about the BIOS Boot
partition but it sounds as if the author had a bad experience at
sometime in the past which has affected their ability to view the
topic objectively.

Cheers,
David.

[Mike Kupfer]

Thomas Schmitt wrote:

> Hi,

>
> Just to make it clear:
> Was this kernel loaded from the very same USB stick which it then cannot
> inquire properly and in some cases cannot read from ?

Yes.

Oh, and I tried it again this morning, after the laptop had been off all
night, and this time it worked (live image boot). I then tried
rebooting and running the installer, and it failed in the usual place.

> > [!!] Detect and mount CD-ROM
> > No common CD-ROM drive was detected.
>
> This does not match the complaint in the web to which Kent pointed us
> as example.

Ah. Right.

> > This may be related to device initialization by the BIOS.
>
> If the kernel was loaded by the bootloader from that USB stick, then
> the subsequent failures in the kernel run could be caused by some bad
> state of the USB stick when the bootloader starts the kernel.
>
> If booting was done via EFI (and not Legacy BIOS mode) then we could
> ask at grub-...@gnu.org, whether the problem is known in any way.
> (With BIOS mode we'd need to go to quite dead sysl...@zytor.com )

I'm booting using EFI.

> How does the stick behave if you plug it into a running GNU/Linux system ?
> Any problem messages in the log ? Is it usable as storage device ?

If I plug the stick into the laptop when it's running, the stick is
mounted okay. I ran sum(1) on all the files, using "find ... -type f".
I didn't see any problems in kern.log. I poked around with cd and ls a
bit and didn't notice anything odd, but then I'm not familiar with the
contents of a hybrid image ISO.

Thanks for all your help on this.

mike

[Chris Ramsden]

On 2018-05-14 14:55, David Wright wrote:
> Would I be correct in thinking that the BIOS POST boot screen is
> what you get when you hit F12 sufficiently quickly after switch-on?
> So are you choosing between UEFI and Legacy (compatibility) mode.
> (I would like to know how Dell handles what I've been reporting on
> with this Lenovo.)

Yes. F12. Then you get to choose between the various bootable devices
and partitions. But this is BIOS or firmware (not Grub) doing it for you.

>> I gave up, wiped windows and went through with a clean Linux install. I
>> don't really want windows that much, but it irks me that I haven't been
>> able to fathom out how to return to the original state in which it was
>> shipped. Your words hint at many things I became vaguely aware of but
>> totally failed to grasp. The other posters to this thread have at least
>> reassured me that it isn't easy or trivial to get right.
> This is the scenario I was trying to avoid. As far as windows was
> concerned, my mantra was Failure Is Not An Option.
>
> Cheers,
> David.

--
Chris

[Thomas Schmitt]

Hi,

Stefan Monnier wrote:
> > the behavior of the firmware
> > and OS when faced with a disk that has both a GPT and an MBR partitions
> > is largely unspecified and will vary depending on your system).

David Wright wrote:
> I've yet to see a GPT disk that didn't have a protective MBR.
> I thought that's the reason why GPT starts at block 1 and not
> block 0: an MBR was designed into GPT from the start (no pun intended).

Indeed. The byte format of GPT is specified in UEFI. The Protective MBR is
the magic number of GPT. This implies that one cannot have a valid GPT and
MBR partitions with payload on the same disk.

But the code area of the Protective MBR can well be used to boot via BIOS.
EFI explicitely specifies that these bytes shall be ignored by EFI.
And as said previously, BIOS does not expect any partitions. Such expectation
could be in the MBR code, though. Most probably such code won't work with
GPT because of no bootable/active MBR partition.

============================================================================
Quotes from UEFI 2.4:

5 GUID Partition Table (GPT) Disk Layout
[...]

5.2 LBA 0 Format
LBA 0 (i.e. the first logical block) of the hard disk contains either
• a legacy Master Boot Record (MBR) (see Section 5.2.1)
• or a protective MBR (see Section 5.2.2 [actually 5.2.3]).

5.2.1 Legacy Master Boot Record (MBR)
A legacy MBR may be located at LBA 0 (i.e. the first logical block) of
the disk if it is not using the GPT disk layout (i.e., if it is using
the MBR disk layout).
[...]

5.2.2 OS Types
Unique types [of MBR partitions] defined by this specification (other
values are not defined by this specification):
• 0xEF (i.e., UEFI System Partition) defines a UEFI system partition.
• 0xEE (i.e., GPT Protective) is used by a protective MBR (see 5.2.2)
to define a fake partition covering the entire disk.

5.2.3 Protective MBR
A Protective MBR may be located at LBA 0 (i.e. the first logical block)
of the disk if it is using the GPT disk layout. The Protective MBR
precedes the GUID Partition Table Header to maintain compatibility with
existing tools that do not understand GPT partition structures.

Table 15. Protective MBR
Mnemonic | Byte | Byte | Content
| Offset | Length |
---------------------------------------------------------------
Boot Code | 0 | 440 | Unused by UEFI systems.
Unique MBR Signature | 440 | 4 | Unused. Set to zero.
Unknown | 444 | 2 | Unused. Set to zero.
Partition Record | 446 | 16*4 | [partition table]
Signature | 510 | 2 | Set to 0xAA55 [...]

The MBR partition table is described in table 15 as
Array of four MBR partition records. Contains:
• one partition record as defined Table 16; and
• three partition records each set to zero.

Table 16 then states

Protective MBR Partition Record protecting the entire disk
[...]
Mnemonic |Byte |Byte | Description
|Offset|Length|
----------------------------------------------------------------
BootIndicator| 0 | 1 | Set to 0x00 to indicate a non-bootable
| | | partition. [...]
StartingCHS | 1 | 3 | Set to 0x000200 [...]
OSType | 4 | 1 | Set to 0xEE (i.e., GPT Protective)
EndingCHS | 5 | 3 | Set to the CHS address of the last logical
| | | block on the disk. Set to 0xFFFFFF if it is
| | | not possible to represent the value in this
| | | field.
StartingLBA | 8 | 4 | Set to 0x00000001 (i.e., the LBA of the GPT
| | | Partition Header).
SizeInLBA | 12 | 4 | Set to the size of the disk minus one. Set to
| | | 0xFFFFFFFF if the size of the disk is too
| | | large to be represented in this field.

============================================================================

[Stefan Monnier]

> Yes, documentation of firmware is almost unknown in my experience
> (since probably 30 years ago). That's why I took the least invasive

It's documented to the extent that it says "implements UEFI" and that
UEFI is documented.

>> Same here (basically for the same reason: the behavior of the firmware
>> and OS when faced with a disk that has both a GPT and an MBR partitions
>> is largely unspecified and will vary depending on your system).
> Eh? I've yet to see a GPT disk that didn't have a protective MBR.

Exactly: what happens when a GPT disk has a real MBR (rather than a
protective MBR) is "you're on your own".

>> It's easy to reconcile: he doesn't say your setup is impossible or can't
>> work, he just recommends not to do that because you may encounter
>> unexpected difficulties. E.g. in theory an upgrade to your firmware or
>> to one of your OSes could break it, tho in practice you're probably OK
>> at least until you move that setup to another machine with
>> a different firmware.
> Not sure what you mean here. It's a laptop: nowt's going nowhere.

Take the disk out, put it in another machine (laptop or desktop, it
doesn't matter).


Stefan

[Thomas Schmitt]

Hi,

Mike Kupfer wrote:
> I'm booting using EFI.

So this could be a problem with GRUB about how it leaves the USB stick
after having loaded kernel and initrd from it.

The volatility of success and failure still gives me riddles. But if you
can reproduce failure within a bearable number of tries, then it could
be indeed a topic for grub-devel mailing list.

> > How does the stick behave if you plug it into a running GNU/Linux system ?
> > Any problem messages in the log ? Is it usable as storage device ?

> If I plug the stick into the laptop when it's running, the stick is

> mounted okay. [...] didn't notice anything odd, but then I'm not familiar

> with the contents of a hybrid image ISO.

They have about the best verification support you can wish for.

https://www.debian.org/CD/faq/#verify
proposes to learn the number of sectors

/sbin/isosize -x <device>

and to then compute the checksum of that amount of data

dd if=<device> count=<sector count> bs=2048 | sha512sum

which is to be compared with the .iso image's line in file SHA512SUMS
from where you downloaded the ISO.

[David Wright]

On Mon 14 May 2018 at 18:13:08 (+0200), Thomas Schmitt wrote:
> Hi,
>
> Mike Kupfer wrote:
> > I'm booting using EFI.
>
> So this could be a problem with GRUB about how it leaves the USB stick
> after having loaded kernel and initrd from it.
>
> The volatility of success and failure still gives me riddles. But if you
> can reproduce failure within a bearable number of tries, then it could
> be indeed a topic for grub-devel mailing list.

I read it as meaning that the USB stick works as a live system
(first boot), but not as an installer (second boot).

Which makes me wonder about earlier discussions (maybe months ago)
concerning whether installing Debian from a live stick can be
problematical (though I don't remember the nature of the problems
people had). I've never tried it.

Cheers,
David.

[Thomas Schmitt]

Hi,

David Wright wrote:
> I read it as meaning that the USB stick works as a live system
> (first boot), but not as an installer (second boot).

Hm ... re-reading Mike's mails ...

Mike Kupfer wrote in his first mail:

> > > using a netinst image and a live image.

and today:

> > If I select the installer (rather than the live
> > image) when booting from the USB stick, it gets through the 3 i18n
> > screens (language, location, keyboard layout), pauses, and then brings

> > up a screen with [error message]

Mike, is this true ?
Does the live image always succeed and the netinst image always fail ?
Is always the same USB stick affected ? (Or do you have one for each ISO ?)
How do you actually "select" between installer and live ?

Which netinst ISO and which live ISO exactly do you use ?

> Which makes me wonder about earlier discussions (maybe months ago)
> concerning whether installing Debian from a live stick can be
> problematical

But in Mike's case the live image is the (more) successful one.
The failure is more brutal than the past installation problems from
live ISOs. Probably the known ones have been solved meanwhile.

The main riddle with Mike's report is why the kernel cannot bring up
the USB stick as storage device although GRUB loaded the kernel from
there.

[Mike Kupfer]

Thomas Schmitt wrote:

> Does the live image always succeed and the netinst image always fail ?
> Is always the same USB stick affected ? (Or do you have one for each ISO ?)
> How do you actually "select" between installer and live ?

I made some changes between the initial attempts and what I'm doing to
test now; apologies for not being clear about it.

Right now, I'm using the live image ISO
(debian-live-9.2.0-amd64-xfce.iso). When it boots, I get a screen with
various selections. To boot the live image, I select "Debian GNU/Linux
Live"; to boot the text-based installer, I select "Debian Installer". I
don't think I have a netinst image on a USB stick at the moment, but I
could set one up if you think that would be a useful data point.

Since the OP's problems were with 9.4, I haven't tried a more recent
ISO, but I'll download an image and test it. (If this turns out to be
fixed in 9.4, I will be very embarrassed and will happily buy y'all
a meal if you're ever in the Bay Area.)

Neither the live image or the text installer is consistent. For
example, I just now started the text installer, after the laptop had
been powered off for a few hours. The first attempt got past the usual
failure point. At the screen for selecting a network device, I
power-cycled the laptop and tried again to run the text installer. This
time it failed at the usual place.

So if there's any pattern here, it's that the first attempt, when the
laptop has been off for awhile, is likely to succeed. Subsequent
attempts, without waiting for a few hours, are likely to fail. This
seems to be true for both the live image and the text installer.

mike

[Pascal Hambourg]

Le 14/05/2018 à 02:02, David Wright a écrit :
> On Sun 13 May 2018 at 19:08:48 (+0200), Pascal Hambourg wrote:
>>
>> Most of my early experience with UEFI boot comes from a rather old
>> Intel motherboard. Beside crippled UEFI support (no UEFI boot from
>> USB or SATA in AHCI mode), it had a couple of annoying requirements :
>> - boot in legacy mode only if the MBR contains a partition entry
>> with the boot flag set, regardless of whether the disk has a MSDOS
>> or GPT partition table. This behaviour is beyond any common BIOS
>> standard, but I have observed it on many other systems, mostly Dell
>> and HP ;
>
> In the case of GPT, I assume the partition entry with the boot flag set
> is the protective MBR.

Yes, as the protective GPT partition entry is the only non-empty entry.

>> - boot in EFI mode from GPT only if the protective partition entry
>> in the MBR has the boot flag unset. I admit this requirement is part
>> of the GPT specification, but really do not see the point in
>> enforcing such a minor detail.
>>
>> Anyway, these two requirements put together make it impossible to
>> boot in both legacy and EFI mode from the same GPT disk with this
>> motherboard. However they allow to boot in both modes from the same
>> MSDOS disk. But who still wants to use MSDOS format nowadays ?
>
> Is it impossible, then, to change the boot flag in a protective MBR?

Of cours not, but it is even less convenient than switching the boot
mode in the firmware setup : boot the system, switch the boot flag,
reboot the system...

[Pascal Hambourg]

Le 14/05/2018 à 16:33, David Wright a écrit :
>
> "Don’t do UEFI-native installs to MBR-formatted disks, or BIOS
> compatibility installs to GPT-formatted disks (an exception to the
> latter is if your disk is, IIRC, 2.2+TB in size, because the MBR
> format can’t handle disks that big – if you want to do a BIOS
> compatibility install to a disk that big, you’re kinda stuck with
> the BIOS+GPT combination, which works but is a bit wonky and
> involves the infamous ‘BIOS Boot partition’ you may recall from
> Fedora 17).
>
> I haven't been able to find anything infamous about the BIOS Boot
> partition but it sounds as if the author had a bad experience at
> sometime in the past which has affected their ability to view the
> topic objectively.

AFAICS the BIOS boot partition is a rather widely unknown topic.
Actually a BIOS boot partition is not required for BIOS boot on GPT. It
is only recommended or required in order to install GRUB in some
situations. But GRUB is not the only available bootloader.

[Pascal Hambourg]

Le 14/05/2018 à 17:50, Thomas Schmitt a écrit :
>
> And as said previously, BIOS does not expect any partitions.

In theory. All it should expect is the MBR signature 0xAA55. But as said
previously, experience taught me that many BIOS implementations wrongly
expect an MBR partition entry with the boot flag set in order to boot
from the disk.

[Mike Kupfer]

Thomas Schmitt wrote:

> Mike Kupfer wrote:

> > If I plug the stick into the laptop when it's running, the stick is
> > mounted okay. [...] didn't notice anything odd, but then I'm not familiar
> > with the contents of a hybrid image ISO.
>
> They have about the best verification support you can wish for.
>
> https://www.debian.org/CD/faq/#verify
> proposes to learn the number of sectors
>
> /sbin/isosize -x <device>
>
> and to then compute the checksum of that amount of data
>
> dd if=<device> count=<sector count> bs=2048 | sha512sum
>
> which is to be compared with the .iso image's line in file SHA512SUMS
> from where you downloaded the ISO.

Hmm. I used sha256sum instead of sha512sum, but I otherwise followed
the above instructions. The checksum from the dd pipeline does not
match the checksum of the original .iso file.

It does, however, match the checksum of the first partition. That is,

isosize -x /dev/sdb

gives me a sector count of 950976 and a block size of 2048. And

dd if=/dev/sdb count=950976 bs=2048 | sha256sum

gives me the same result as

sha256sum /dev/sdb1

mike

[Mike Kupfer]

Mike Kupfer wrote:

> Since the OP's problems were with 9.4, I haven't tried a more recent
> ISO, but I'll download an image and test it. (If this turns out to be
> fixed in 9.4, I will be very embarrassed and will happily buy y'all
> a meal if you're ever in the Bay Area.)

I just tried 9.4, using a different USB stick. It fails in the same way
(both live image and text installer) as 9.2. I guess I'll try a netinst
image next, but that'll have to wait until Wednesday or Thursday.

best regards,
mike

[songbird]

https://wiki.debian.org/UEFI
https://wiki.debian.org/GrubEFIReinstall

and

http://www.rodsbooks.com/refind/

> I got a new Dell computer, shrunk the existing partitions down and
> successfully installed grub2 and got a windows10/Linux multi boot using
> grub. Then later I tried to upgrade my Linux and soon found that I was
> getting error messages about grub not being able to find necessary
> features on the boot device.

yep. do not install stable after testing is all i
can recommend at this point (based upon my experience)
and of course keep a working/verified netinst image
handy.

> I tried to rebuild it with a clean install of Windows 10, reasoning that
> if I could get it back to the original configuration, I could repeat the
> original exercise. But alas, no, it remains stubbornly unable to install
> grub2 alongside the windows bootloader. I got it to a state where I
> could use the BIOS POST boot screen to choose a boot option, but this
> wasn't the original successful arrangement where grub offered me the
> Linux/windows loader choice.
>
> I gave up, wiped windows and went through with a clean Linux install. I
> don't really want windows that much, but it irks me that I haven't been
> able to fathom out how to return to the original state in which it was
> shipped. Your words hint at many things I became vaguely aware of but
> totally failed to grasp. The other posters to this thread have at least
> reassured me that it isn't easy or trivial to get right.

i was rather surprised by it all but i'd been living
in the dark ages for years anyways.


songbird

[David Wright]

On Mon 14 May 2018 at 11:56:11 (-0400), Stefan Monnier wrote:
> > Yes, documentation of firmware is almost unknown in my experience
> > (since probably 30 years ago). That's why I took the least invasive
>
> It's documented to the extent that it says "implements UEFI" and that
> UEFI is documented.

I can only speak for machines that I have access to. The earliest PCs
I remember came with booklets that had illustrations of the various
CMOS screens. Since then, I've usually had to transcribe the settings
manually into files that I archive (in case of battery failure etc.)

In this particular instance (the Lenovo), its PDF says:

How can I change the boot mode?
There are two boot modes: UEFI and Legacy Support. To change the boot
mode, start the BIOS setup utility and set boot mode to UEFI or Legacy
support on the boot menu.

and that's it, apart from implying that the Legacy mode is useful:

When do I need to change the boot mode?
The default boot mode for your computer is UEFI mode. If you need to
install a legacy operating system, such as Windows (that is, any operating
system before Windows 8), Linux or DOS, etc on your computer, you must
change the boot mode to Legacy support. The legacy operating system,
such as Windows, Linux or DOS, etc cannot be installed if you don't
change the boot mode.

> >> Same here (basically for the same reason: the behavior of the firmware
> >> and OS when faced with a disk that has both a GPT and an MBR partitions
> >> is largely unspecified and will vary depending on your system).
> > Eh? I've yet to see a GPT disk that didn't have a protective MBR.
>
> Exactly: what happens when a GPT disk has a real MBR (rather than a
> protective MBR) is "you're on your own".

As Thomas has just pointed out, there's no such thing. I didn't try
to read anything into your non-idiomatic use of the plural in "a disk
that has both a GPT and an MBR partitions". There will be any number
of partitions in the GPT, and just one in the MBR's partition table.

AIUI what's unspecified is what you choose to put in the code section
of the MBR. In my case, and probably for most people here, it will be
Stage 1 of Grub. And if one has any sense, that will be pointing to a
BIOS boot partition, which is one of the first things I set up on a
GPT disk so that I can use it in my old BIOS machines. It makes things
far more bullet-proof if Grub knows there's a safe place to put its
later Stages. I suspect that systems without ones are likely to be the
ones causes much of the reported trouble.

Where's the author's discussion of all this? My beef is that these
aspects are obfuscated with talk of magic code, magic space, magic
locations and "special sauce". That's the kind of stuff I'm placing in
*their* garbage, even if it doesn't make up 95%.

> >> It's easy to reconcile: he doesn't say your setup is impossible or can't
> >> work, he just recommends not to do that because you may encounter
> >> unexpected difficulties. E.g. in theory an upgrade to your firmware or
> >> to one of your OSes could break it, tho in practice you're probably OK
> >> at least until you move that setup to another machine with
> >> a different firmware.
> > Not sure what you mean here. It's a laptop: nowt's going nowhere.
>
> Take the disk out, put it in another machine (laptop or desktop, it
> doesn't matter).

At that point, I would assume windows is dead because it's not
licenced for its new home. At which point, a decision could be made
on the basis of whether the new machine could boot Grub through the
MBR. If that's not the case, then obviously one is going to have to
boot with UEFI from, say, a stick and grub-install in UEFI mode.

If that change is made correctly, the disk should be bootable in its new
location. In addition, if it were moved back to its original location,
it should now be possible to boot both windows and linux using EUFI mode.

But these are the issues I have been keen to avoid (successfully)
by booting in different modes for the two OSes. As I stated, there
was to be no disturbance of the windows system.

Cheers,
David.

[David Wright]

On Mon 14 May 2018 at 23:39:06 (+0200), Pascal Hambourg wrote:
> Le 14/05/2018 à 16:33, David Wright a écrit :
> >
> > "Don’t do UEFI-native installs to MBR-formatted disks, or BIOS
> > compatibility installs to GPT-formatted disks (an exception to the
> > latter is if your disk is, IIRC, 2.2+TB in size, because the MBR
> > format can’t handle disks that big – if you want to do a BIOS
> > compatibility install to a disk that big, you’re kinda stuck with
> > the BIOS+GPT combination, which works but is a bit wonky and
> > involves the infamous ‘BIOS Boot partition’ you may recall from
> > Fedora 17).
> >
> >I haven't been able to find anything infamous about the BIOS Boot
> >partition but it sounds as if the author had a bad experience at
> >sometime in the past which has affected their ability to view the
> >topic objectively.
>
> AFAICS the BIOS boot partition is a rather widely unknown topic.

I'm happy to publicise it a little, then, and without using terms
like infamous and wonky.

> Actually a BIOS boot partition is not required for BIOS boot on GPT.
> It is only recommended or required in order to install GRUB in some
> situations. But GRUB is not the only available bootloader.

No.

It's difficult to divine which bootloaders the author is familiar with.
My own experience of the last twenty years is limited to Lilo and Grub.
I remember reading about coreboot some years ago, but I doubt I'm ever
going to meet it. Are you suggesting others?

Cheers,
David.

[David Wright]

Sorry, I thought impossible meant not possible. I agree that booting
up a system (that happens to be set to the wrong state) just to change
the boot flag and then reboot is a lot less convenient than the
process I use (which always lets you boot into the correct system
first time, and only requires action when you want to change over).

But the fact that you've experienced a crippled mobo does not IMHO
mean that some of the scenarios you could use on less crippled systems
shouldn't be discussed—nay, be treated as near impossible—in a web
page that's meant to be throwing light on the matter. As written, the
author just pours special magic sauce around, whatever that is.

Cheers,
David.

[Thomas Schmitt]

Hi,

i wrote:
> > And as said previously, BIOS does not expect any partitions.

Pascal Hambourg wrote:
> experience taught me that many BIOS implementations wrongly
> expect an MBR partition entry with the boot flag set in order to boot from
> the disk.

You are right (and those firmwares are wrong).

I had to introduce a special option in xorriso in order to cater for this:
https://savannah.gnu.org/bugs/?46716
http://lists.gnu.org/archive/html/bug-xorriso/2015-12/threads.html
with the final conclusion described in
https://lists.gnu.org/archive/html/bug-grub/2016-03/msg00087.html

Summary:
- Boot flag on MBR partition of type 0xEE is bad on several EFI
implementations.
- No MBR partition with boot flag is bad on some very few BIOS
implementations.
- Compromise is to set the boot flag on a dummy partition of type 0x00.
This is barely UEFI-compliant because the specs say that a partition of
type 0x00 shall be regarded as non-existent.

Debian isohybrids for BIOS do not have this problem because MBR partition 1
has the boot flag set, anyway. But grub-mkrescue ISOs might be in need of
xorrisofs option
--mbr-force-bootable
if the owner suffers from one of those rare BIOSes.
(It was decided that from the view of GRUB this is a workaround for a
specs violation by the firmware. iSo --mbr-force-bootable is not applied
by default.)

[Thomas Schmitt]

Hi,

Mike Kupfer wrote:
> I used sha256sum instead of sha512sum, but I otherwise followed
> the above instructions. The checksum from the dd pipeline does not
> match the checksum of the original .iso file.

That's not good.
Especially we do not have to show up at grub-devel as long as the test
USB stick might have problems as storage device.

> dd if=/dev/sdb count=950976 bs=2048 | sha256sum
> gives me the same result as
> sha256sum /dev/sdb1

That's normal with Debian ISOs for "i386" and "amd64". The first partition
marks the whole ISO.


But the checksum should really match the one of the ISO image file and
the one listed in the *SUMS file.
Please verify that your ISO image file yields the checksum that is listed
in *SUMS.

> I just tried 9.4, using a different USB stick. It fails in the same way

Does the checksum of partition 1 (or truncated base device) match *SUMS ?

[Pascal Hambourg]

Le 15/05/2018 à 08:51, Thomas Schmitt a écrit :
>
> Summary:
> - Boot flag on MBR partition of type 0xEE is bad on several EFI
> implementations.
> - No MBR partition with boot flag is bad on some very few BIOS
> implementations.

Not so few in my experience.

> - Compromise is to set the boot flag on a dummy partition of type 0x00.
> This is barely UEFI-compliant because the specs say that a partition of
> type 0x00 shall be regarded as non-existent.

I never thought about this. It may be a workaround with my old UEFI
motherboard. Thanks for the tip, I will try it and report.

[Pascal Hambourg]

Le 15/05/2018 à 03:23, David Wright a écrit :
>
> In this particular instance (the Lenovo), its PDF says:

(...)

> The default boot mode for your computer is UEFI mode. If you need to
> install a legacy operating system, such as Windows (that is, any operating
> system before Windows 8)

This is so wrong. Windows has supported EFI since Vista.

> AIUI what's unspecified is what you choose to put in the code section
> of the MBR. In my case, and probably for most people here, it will be
> Stage 1 of Grub.

Note : "stage 1" is a GRUB legacy thing. GRUB 2 calls it "boot image".

> And if one has any sense, that will be pointing to a
> BIOS boot partition, which is one of the first things I set up on a
> GPT disk so that I can use it in my old BIOS machines. It makes things
> far more bullet-proof if Grub knows there's a safe place to put its
> later Stages. I suspect that systems without ones are likely to be the
> ones causes much of the reported trouble.

Moreover, GRUB BIOS cannot be installed without embedding (in a BIOS
boot partition or any other suitable location) when the /boot filesystem
is anywhere else than a plain partition (LVM, RAID, crypto...) or a
filesystem type which may move block around such as btrfs, defeating the
hardcoded blocklists.

[Pascal Hambourg]

Le 15/05/2018 à 03:37, David Wright a écrit :
>
>> But GRUB is not the only available bootloader.
>
> No.
>
> It's difficult to divine which bootloaders the author is familiar with.
> My own experience of the last twenty years is limited to Lilo and Grub.

Same here. I would not trust LILO any more now because it uses hardcoded
blocklists to read files, just like GRUB does when embedding is not
possible, which is considered unreliable.

> I remember reading about coreboot some years ago, but I doubt I'm ever
> going to meet it. Are you suggesting others?

AFAIK, Coreboot is a replacement for platform firmware (BIOS), not an
installable boot loader.

I have never used it, but it seems that syslinux supports GPT and EFI.
Also I have read positive reports about rEFInd for EFI boot.

[mess-mate]

Your're right.
I've installed windows 10first a mont ago, after that last debian and
rEFIND.
All run smootless without any problem and at the first install.
Of course, partitions on GPT and UEFI.

[Pascal Hambourg]

The trick indeed worked with my old motherboard ! Good to know, thanks.

However there are some caveats :
- I had to use the old fdisk version from Wheezy because newer versions
and other partition editors would not allow to set the boot flag on an
empty partition entry.
- parted silently erased the flag when I used it to edit the GPT table.
I expected it, because parted had already reset a hybrid MBR I had set
up with gdisk to a standard protective MBR.

[Thomas Schmitt]

Hi,

i wrote:
> > - Compromise is to set the boot flag on a dummy partition of type 0x00.
> >    This is barely UEFI-compliant because the specs say that a partition of
> >    type 0x00 shall be regarded as non-existent.

Pascal Hambourg wrote:
> - I had to use the old fdisk version from Wheezy because newer versions and
> other partition editors would not allow to set the boot flag on an empty
> partition entry.

You never know what kind of user-friendly abstraction a partition editor will
apply. So portable and future-safe is only the manipulation on byte level.

The layout of the MBR partition table can be learned from
https://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
"Structure of a classical generic MBR"
https://en.wikipedia.org/wiki/Master_boot_record#Partition_table_entries

Partition entries are 16 bytes of size and number 1 begins at offset 446.
The boot/active flag sits in the first byte of the partition entry.
The value of this byte is either 0 or 128 (= 0x80 in hex). We want 128.
Further we need to write values for partition start and size.

# Choose target disk or image file
disk=/dev/sdX

# Make a backup of your MBR by
backup_path="$HOME"/mbr_backup_of_"$(basename "$disk")"
dd if="$disk" bs=512 count=1 conv=notrunc of="$backup_path"

# Restore the backup to $disk would be done like this:
# dd if="$backup_path" bs=512 count=1 conv=notrunc of="$disk"
# If $disk is the system disk: Have a rescue system ready which can
# access the backup file and the system disk.

# Choose partition and compute start of its entry
partno=2
offset=$(expr 430 + "$partno" '*' 16)

# First set the partition entry to all zeros. So we can just hop over
# those bytes which shall stay 0. This spares us the plight to handle
# 0-bytes by shell commands.
dd if=/dev/zero bs=1 count=16 conv=notrunc seek="$offset" of="$disk"

# The boot/active flag
# (The gesture $'\xHH' is described in man bash section QUOTING.)
echo -n $'\x80' | dd bs=1 count=1 conv=notrunc seek="$offset" of="$disk"

# Start C/H/S = 0/0/1
offset=$(expr "$offset" + 2)
echo -n $'\x01' | dd bs=1 count=1 conv=notrunc seek="$offset" of="$disk"

# End C/H/S = 0/0/1
offset=$(expr "$offset" + 4)
echo -n $'\x01' | dd bs=1 count=1 conv=notrunc seek="$offset" of="$disk"

# Sector count = 1 (start LBA is 0)
offset=$(expr "$offset" + 6)
echo -n $'\x01' | dd bs=1 count=1 conv=notrunc seek="$offset" of="$disk"

The task would also be a good execercise for Programmer's First C Program.

[Mike Kupfer]

Thomas Schmitt wrote:

> Mike Kupfer wrote:
> > I used sha256sum instead of sha512sum, but I otherwise followed
> > the above instructions. The checksum from the dd pipeline does not
> > match the checksum of the original .iso file.
>
> That's not good.
> Especially we do not have to show up at grub-devel as long as the test
> USB stick might have problems as storage device.
>
>
> > dd if=/dev/sdb count=950976 bs=2048 | sha256sum
> > gives me the same result as
> > sha256sum /dev/sdb1
>
> That's normal with Debian ISOs for "i386" and "amd64". The first partition
> marks the whole ISO.

That is true for the 9.4 netinst ISO, but there seems to be something
odd going on with the live image ISOs.

alto$ sudo dd if=debian-live-9.4.0-amd64-xfce.iso of=/dev/sdb bs=32K
[sudo] password for kupfer:
59554+0 records in
59554+0 records out
1951465472 bytes (2.0 GB, 1.8 GiB) copied, 455.702 s, 4.3 MB/s
alto$ sudo isosize -x /dev/sdb
sector count: 952848, sector size: 2048

But 952848 * 2048 = 1951432704, not 1951465472.

If I calculate the sector count from the size of the .iso file

1951465472 / 2048 = 952864

and use that with dd piped to sha256sum, I do get the correct checksum:

dd if=/dev/sdb count=952864 bs=2048 | sha256sum
5d6f990bdd618902f9c0ce4b090d6547a77d89c3bab87136e6adf9a9378f4f39 -

That matches the value for the .iso, and it matches the value in
SHA256SUMS.

I saw this when testing multiple sticks, including the Verbatim stick
mentioned below, so I don't think this is a hardware glitch. And I
observed something similar with the 9.2 Xfce live image ISO.

> > I just tried 9.4, using a different USB stick. It fails in the same way

After running several tests, with 4 different sticks, 2 different ISOs
(the 9.4 netinst ISO and the 9.4 Xfce live image ISO), and 2 different
computers (the Latitude and my desktop), I believe the problem is, in
fact, the USB stick. And not just the first one that I had problems
with. It looks like there's something about that entire make/model
(MOSDART 8GB). 3 of the 4 sticks that I tested were MOSDART sticks, and
the 4th was a Verbatim stick. I had problems with all 3 MOSDART sticks,
with both ISOs and both computers. I had no problems with the Verbatim
stick with either ISO or computer, and that includes multiple tests
without any sort of "resting period" (with the power turned off).

Thomas, thanks again for all your help on this. I'll have to be more
careful the next time I'm buying storage.

mike

[Thomas Schmitt]

Hi,

Mike Kupfer wrote:
> there seems to be something odd going on with the live image ISOs.

> [...]

> alto$ sudo dd if=debian-live-9.4.0-amd64-xfce.iso of=/dev/sdb bs=32K

> [...]

> 1951465472 bytes (2.0 GB, 1.8 GiB) copied, 455.702 s, 4.3 MB/s
> alto$ sudo isosize -x /dev/sdb
> sector count: 952848, sector size: 2048
> But 952848 * 2048 = 1951432704, not 1951465472.

That's indeed undesirable.
Interestingly debian-live-9.3.0-amd64-xfce.iso shows no such difference.

It looks like 9.4 was padded up to a full multiple of 64 KiB, whereas
the isosize of 9.3 was already aligned to 64 KiB.

My guess for now is that the reason is the multi-session preparation
which xorriso does by default if used with its native command set.
IIRC (after 10 years) this preparation includes session alignment to
full 32 blocks - 64 KiB.
The producer of Debian Live ISOs, live-wrapper, is one of the very few
producers of bootable ISOs which uses that command set. The others use
the mkisofs emulation, which omits the multi-session preparation by
default.

I will have to investigate deeper and probably ask the live-wrapper
maintainer to disable the multi-session preparation by command
-compliance "no_emul_toc"

Thanks for pointing this out.

> I believe the problem is, in
> fact, the USB stick. And not just the first one that I had problems
> with. It looks like there's something about that entire make/model
> (MOSDART 8GB).

Let's hope they are not some of those fake capacity sticks. They pretend
to have their nominal storage capacity, offer the promised number of
logical blocks, but map them to a smaller number of physical blocks.
This makes them look as if they take all data. But in the end some of
the earlier written physical blocks get overwritten by later written
blocks with a different logical block address.

[David Wright]

On Wed 16 May 2018 at 00:35:33 (+0200), Pascal Hambourg wrote:
> Le 15/05/2018 à 03:23, David Wright a écrit :
> >
> >In this particular instance (the Lenovo), its PDF says:
> (...)
> > The default boot mode for your computer is UEFI mode. If you need to
> > install a legacy operating system, such as Windows (that is, any operating
> > system before Windows 8)
>
> This is so wrong. Windows has supported EFI since Vista.

I offer no defence of what Lenovo choose to write. What I do know is
that in order to make this laptop available to myself, I had to be
able to install linux without disturbing the Windows/GPT/UEFI setup
already present.

Reading the page at
https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/
would have been most discouraging, so I'm glad I didn't chance upon
it and, in the light of my experience, I consider some of *it* to be
garbage, a word it's happy to use to condemn 95% of what's written
about UEFI elsewhere on the internet.

But then, I also chose to ignore what's written in the Debian
Installation Guide §3.6.3:
"Booting from a disk with GPT is only possible in native UEFI mode".

> >AIUI what's unspecified is what you choose to put in the code section
> >of the MBR. In my case, and probably for most people here, it will be
> >Stage 1 of Grub.
>
> Note : "stage 1" is a GRUB legacy thing. GRUB 2 calls it "boot image".

I wasn't aware that the "Stage 1" name was eliminated. I don't find
"boot image" adding any clarity to the process. Anyway, someone needs
to tell https://en.wikipedia.org/wiki/GNU_GRUB which says:

Version 2 (GRUB)
Stage 1: boot.img is stored in the master boot record (MBR) or
optionally in any of the volume boot records (VBRs), and addresses
the next stage by an LBA48 address (thus, the 1024-cylinder
limitation of GRUB legacy is avoided); at installation time it is
configured to load the first sector of core.img.

> >And if one has any sense, that will be pointing to a
> >BIOS boot partition, which is one of the first things I set up on a
> >GPT disk so that I can use it in my old BIOS machines. It makes things
> >far more bullet-proof if Grub knows there's a safe place to put its
> >later Stages. I suspect that systems without ones are likely to be the
> >ones causes much of the reported trouble.
>
> Moreover, GRUB BIOS cannot be installed without embedding (in a BIOS
> boot partition or any other suitable location) when the /boot
> filesystem is anywhere else than a plain partition (LVM, RAID,
> crypto...) or a filesystem type which may move block around such as
> btrfs, defeating the hardcoded blocklists.

Agreed: that's why I'm making sure I add a BIOS boot partition to
any disk I convert from MBR to GPT, allowing it to be a candidate
for booting from.

Cheers,
David.

[David Wright]

On Wed 16 May 2018 at 00:23:43 (+0200), Pascal Hambourg wrote:
> Le 15/05/2018 à 03:37, David Wright a écrit :
> >
> >>But GRUB is not the only available bootloader.
> >
> >No.
> >
> >It's difficult to divine which bootloaders the author is familiar with.
> >My own experience of the last twenty years is limited to Lilo and Grub.
>
> Same here. I would not trust LILO any more now because it uses
> hardcoded blocklists to read files, just like GRUB does when
> embedding is not possible, which is considered unreliable.
>
> >I remember reading about coreboot some years ago, but I doubt I'm ever
> >going to meet it. Are you suggesting others?
>
> AFAIK, Coreboot is a replacement for platform firmware (BIOS),

Yes, that interested me because at the time I had a lot of
(unpowerful) PCs dotted around, some relatively inaccessible.
One of the attractions of coreboot was not having to sit in
front of each machine editing CMOS screens. But AIUI it was
designed for farms of the sort of PCs that were out of my league.

> not an installable boot loader.

I read somewhere that it could directly load linux instead of,
say, Grub, but I don't know anything about the technicalities.

> I have never used it, but it seems that syslinux supports GPT and EFI.

I haven't knowingly used that since I had PCs that booted into
DOS first, back in the last millennium.

> Also I have read positive reports about rEFInd for EFI boot.

One for the future, when I get my very own machine that's UEFI. At
the moment my newest PCs are a 2008 BIOS laptop and a couple of 2006
desktops. The latter boot GPT disks fine. The laptop will have to live
out the rest of its days configured as is, running jessie and wheezy;
too much of it is broken for anything else to be installed.

Cheers,
David.

[Teemu Likonen]

I am replying to an old thread which described the same or similar
problem that I had. I have a solution which may be useful to store in
the mailing list archives. The thread started with this message:

https://lists.debian.org/debian-user/2018/05/msg00391.html


* 2018-05-11 11:11:42-0500, Kent West wrote:

> I have a Dell Latitude E7250 laptop. I'm trying to install Debian to
> it using a USB stick.

> The real problem is that after going through the first three or four
> screens, the install halts, complaining about not being able to read
> the CD-ROM.

I have Dell Latitude E6230 and the same problem: The installer boots and
I can choose country and language settings. Then the installer tries to
detect and mount the installer drive but fails to detect it. I confirmed
that in installer's console: command "ls /dev" lists only the main hard
disk device files /dev/sda and its partitions.

The solution that worked for me: I removed the installer USB stick and
inserted it again. Then the installer finds the device and proceeds with
the install. Technically there is now device file /dev/sdb and its
partitions.

--
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450