Deb10 installer can't install grub

[Dave Sherohman]

I've got a new server and am currently fighting with the Debian 10
installer (build 20190702) in my attempts to get it up and running.
After much wailing and gnashing of teeth, I managed to get it to stop
complaining about being unble to mount /boot/efi and complete the
"Partition disks" step successfully, but now I'm completely stuck on the
"Install GRUB" step.

The GUI installer shows the error:

Unable to install GRUB in dummy
Executing 'grub-install dummy' failed.

Checking the syslog output on virtual console 4 shows a bunch of
os-prober activity (as expected), then finally:

Installing for x64_64-efi platform.
grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`.
error: Running 'grub-install --force "dummy"' failed.

I assume that the "canonical path" it's looking for is a /dev/sda-type
device name, but I have no idea how to assign one of those to an nvme
drive. (And I thought that kind of name was supposed to have been
banished in favor of "predictable" names by now anyhow.)

Several of the old-style names are already in use; the installer is
showing sda for the USB stick that the installation was booted from and
sdb-sdi for eight large disks, but the operating system is to boot and
run from the drives /dev/nvme0n1 and /dev/nvme1n1.

What do I need to do to get this working?

Also, would that solution also work for md devices as well as for nvme
devices? I had previously tried putting UEFI onto a RAID1 mirror
between the two nvme drives, but got a similar error from grub about not
being able to find the canonical path of /dev/md1.

--
Dave Sherohman

[Dan Ritter]

/dev/nvme0n1 is your first NVMe drive; /dev/nvme1n1 is your
second. Add a p1 to each to reference the first partition.

If you want to install to an mdadm RAID, that's do-able with
grub as well.

In the installer, you want expert mode. Does the disk
partitioner recognize both nvme0n1 and nvme1n1 ?

You'll want to create the following partitions on each,
identically:

1 efi - type efi
2 boot (or boot/root) - type MDADM volume
3 root, if using separate boot - type MDADM volume
4 swap - type MDADM volume

Then you go to the mdadm setup and create MDADM RAID1 devices
out of each pair of boot, root and swap.

After install to those RAID1 devices, grub will want to install.
Do that to the boot sector of /dev/nvme0n1, and then again to
the boot sector of /dev/nvme1n1.

After a reboot into the new OS, you should set up your storage
disks.

If this doesn't work, tell us where in the process it stops, and
what the error is there.

-dsr-

[The Wanderer]

On 2021-03-02 at 07:54, Dan Ritter wrote:

> You'll want to create the following partitions on each,
> identically:
>
> 1 efi - type efi
> 2 boot (or boot/root) - type MDADM volume
> 3 root, if using separate boot - type MDADM volume
> 4 swap - type MDADM volume
>
> Then you go to the mdadm setup and create MDADM RAID1 devices
> out of each pair of boot, root and swap.

Out of interest, how would you modify this for someone who intends to
run LVM on top of the RAID-1 being defined here, and wants to define as
many of the partitions as possible (but at least /) at the LVM level
rather than the mdadm level?

I'm guessing that the answer would be to define the EFI partition here
directly, then define one further partition as an appropriate type, set
up LVM inside that, and define the further partitions from there.

I'm not positive that that's the correct approach, however, especially
given that I suspect that /boot will need to be visible to GRUB and may
thus need to be outside the LVM.


This isn't purely academic interest; I'm not moving forward right now
because I can't currently afford all the components, but I'm planning a
new computer build as soon as that changes, and one part of the design
I've decided on is a pair of M.2 SSDs in RAID-1.

I remember having had enough trouble getting GRUB installed and booting
with a pair of SATA SSDs in RAID-1 when I built my current machine,
which doesn't use GPT or EFI boot, and I'd prefer to avoid having to
juggle that on the fly again - especially not given that the need to
cater to the partitioning requirements of EFI boot is likely to make
things more complicated.

--
The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw

[Dave Sherohman]

On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote:
> In the installer, you want expert mode. Does the disk
> partitioner recognize both nvme0n1 and nvme1n1 ?

Yes. The list of disks shown in the partitioner is currently:

RAID device #1
RAID device #2
/dev/nvme0n1
/dev/nvme1n1
SCSI2 (0,0,0) (sda) - the USB stick I booted from
SCSI1 (0,0,0) (sdb) - the first storage drive
... the other seven storage drives ...

> You'll want to create the following partitions on each,
> identically:
>
> 1 efi - type efi
> 2 boot (or boot/root) - type MDADM volume
> 3 root, if using separate boot - type MDADM volume
> 4 swap - type MDADM volume

Partitioning is identical on both nvme drives, consisting of:

1.0 MB free space
2.0 GB ESP (bootable)
8.0 GB swap
1.9 TB RAID physical device
331.8 kB free space

> Then you go to the mdadm setup and create MDADM RAID1 devices
> out of each pair of boot, root and swap.

RAID Device #1 is 1.9 TB, ext4fs, and set to mount on /
RAID Device #2 is 2.0 GB, ESP, and bootable

> After install to those RAID1 devices, grub will want to install.
> Do that to the boot sector of /dev/nvme0n1, and then again to
> the boot sector of /dev/nvme1n1.

...and this is where it dies. I'm now in the text-mode "expert
install", and there is no option to manually specify a partition to
install GRUB on, only a yes/no "Force GRUB installation to the EFI
removable media path?" prompt. Regardless of which answer I choose, I
get "Unable to install GRUB in dummy" and syslog records "failed to get
canonical path of /dev/md2".

I also tried dropping to a shell and manually running `grub-installer
/dev/nvme0n1` and got the error "ls: /dev/nvme0n1/proc: Not a directory"
Manually running grub-installer for /dev/md2 and /dev/nvme1n1
(unsurprisingly) produced similar results.

--
Dave Sherohman

[The Wanderer]

On 2021-03-02 at 09:01, Dave Sherohman wrote:

> On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote:

>> Then you go to the mdadm setup and create MDADM RAID1 devices
>> out of each pair of boot, root and swap.
>
> RAID Device #1 is 1.9 TB, ext4fs, and set to mount on /
> RAID Device #2 is 2.0 GB, ESP, and bootable

So the EFI partitions themselves are also RAIDed? I wouldn't have
expected that to work.

>> After install to those RAID1 devices, grub will want to install.
>> Do that to the boot sector of /dev/nvme0n1, and then again to
>> the boot sector of /dev/nvme1n1.
>
> ...and this is where it dies. I'm now in the text-mode "expert
> install", and there is no option to manually specify a partition to
> install GRUB on, only a yes/no "Force GRUB installation to the EFI
> removable media path?" prompt. Regardless of which answer I choose, I
> get "Unable to install GRUB in dummy" and syslog records "failed to get
> canonical path of /dev/md2".
>
> I also tried dropping to a shell and manually running `grub-installer
> /dev/nvme0n1` and got the error "ls: /dev/nvme0n1/proc: Not a directory"
> Manually running grub-installer for /dev/md2 and /dev/nvme1n1
> (unsurprisingly) produced similar results.

Is grub-installer specific to the Debian installer environment? I don't
find it at a glance in my installed system, or with 'apt-file search'; I
do find grub-install, which is a binary (where the error you quote seems
to indicate a script), and is what I seem to recall having used for the
manual installation of GRUB that finally got my current machine working
with a RAIDed boot-drive pair all those years ago. (Albeit not with EFI
boot partitioning, so that may not be directly comparable.)

If grub-installer is a script and is the correct thing to use, the first
thing I'd be inclined to do in your place is to examine that script and
see what it's trying to do at the point of the error, so that I can
figure out what it expects as input and how I can potentially try to
provide that.

[The Wanderer]

On 2021-03-02 at 09:20, Dave Sherohman wrote:

> On Tue, Mar 02, 2021 at 09:09:52AM -0500, The Wanderer wrote:
>
>> On 2021-03-02 at 09:01, Dave Sherohman wrote:

>>> RAID Device #1 is 1.9 TB, ext4fs, and set to mount on /
>>> RAID Device #2 is 2.0 GB, ESP, and bootable
>>
>> So the EFI partitions themselves are also RAIDed? I wouldn't have
>> expected that to work.
>

> That was my expectation as well, but Dan said it should work, so I went
> with it.

I didn't parse what he wrote that way. He said:

>>>> You'll want to create the following partitions on each,
>>>> identically:
>>>>
>>>> 1 efi - type efi
>>>> 2 boot (or boot/root) - type MDADM volume
>>>> 3 root, if using separate boot - type MDADM volume
>>>> 4 swap - type MDADM volume
>>>>

>>>> Then you go to the mdadm setup and create MDADM RAID1 devices
>>>> out of each pair of boot, root and swap.

Note that he specified to RAID the three listed MDADM partitions, but
did not mention the same for the EFI partition.

> Of course, I've also tried it with then not RAIDed and the results
> are identical, aside from the "canonoical path" error message in
> syslog changing from /dev/md2 to /dev/nvme0n1p1.

Not terribly surprising, but it was worth a check.

[Dave Sherohman]

On Tue, Mar 02, 2021 at 09:09:52AM -0500, The Wanderer wrote:

> On 2021-03-02 at 09:01, Dave Sherohman wrote:
>
> > On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote:
>
> >> Then you go to the mdadm setup and create MDADM RAID1 devices
> >> out of each pair of boot, root and swap.
> >
> > RAID Device #1 is 1.9 TB, ext4fs, and set to mount on /
> > RAID Device #2 is 2.0 GB, ESP, and bootable
>
> So the EFI partitions themselves are also RAIDed? I wouldn't have
> expected that to work.

That was my expectation as well, but Dan said it should work, so I went

with it. Of course, I've also tried it with then not RAIDed and the

results are identical, aside from the "canonoical path" error message in
syslog changing from /dev/md2 to /dev/nvme0n1p1.

> Is grub-installer specific to the Debian installer environment? I don't
> find it at a glance in my installed system, or with 'apt-file search'; I
> do find grub-install, which is a binary (where the error you quote seems
> to indicate a script), and is what I seem to recall having used for the
> manual installation of GRUB that finally got my current machine working
> with a RAIDed boot-drive pair all those years ago. (Albeit not with EFI
> boot partitioning, so that may not be directly comparable.)

Yes, it seems to be unique to the installer environment. I made that
same trip in reverse, first trying to run grub-install, and then having
to go looking when that didn't exist.

> If grub-installer is a script and is the correct thing to use, the first
> thing I'd be inclined to do in your place is to examine that script and
> see what it's trying to do at the point of the error, so that I can
> figure out what it expects as input and how I can potentially try to
> provide that.

Good thought. It's a shell script, so I'll see if I can make sense of
what it's doing in [filesystem]/proc.

--
Dave Sherohman

[Dave Sherohman]

On Tue, Mar 02, 2021 at 09:26:21AM -0500, The Wanderer wrote:
> I didn't parse what he wrote that way. He said:
>
> >>>> You'll want to create the following partitions on each,
> >>>> identically:
> >>>>
> >>>> 1 efi - type efi
> >>>> 2 boot (or boot/root) - type MDADM volume
> >>>> 3 root, if using separate boot - type MDADM volume
> >>>> 4 swap - type MDADM volume
> >>>>
> >>>> Then you go to the mdadm setup and create MDADM RAID1 devices
> >>>> out of each pair of boot, root and swap.
>
> Note that he specified to RAID the three listed MDADM partitions, but
> did not mention the same for the EFI partition.

Ah, right. When he mentioned RAIDing the boot, I took that to mean the
EFI partitions. Forgot that he as using a separate /boot.



So I took a look at the grub-installer script, and guess what I found as
a comment just before the `ls` that kills it?

---
# XXX cjwatson 2019-03-25: This is all far too complicated and fragile, and
# should be replaced with in-target or similar.

# Ensure proc is mounted in all the $chroot calls;
# needed for RAID+LVM for example
initial_proc_contents="$(ls $ROOT/proc)"
---

"Far too complicated and fragile", indeed!

--
Dave Sherohman

[The Wanderer]

On 2021-03-02 at 09:50, Dan Ritter wrote:

> The Wanderer wrote:
>
>> On 2021-03-02 at 07:54, Dan Ritter wrote:
>>
>>> You'll want to create the following partitions on each,
>>> identically:
>>>
>>> 1 efi - type efi
>>> 2 boot (or boot/root) - type MDADM volume
>>> 3 root, if using separate boot - type MDADM volume
>>> 4 swap - type MDADM volume
>>>
>>> Then you go to the mdadm setup and create MDADM RAID1 devices
>>> out of each pair of boot, root and swap.
>>
>> Out of interest, how would you modify this for someone who intends to
>> run LVM on top of the RAID-1 being defined here, and wants to define as
>> many of the partitions as possible (but at least /) at the LVM level
>> rather than the mdadm level?
>>
>> I'm guessing that the answer would be to define the EFI partition here
>> directly, then define one further partition as an appropriate type, set
>> up LVM inside that, and define the further partitions from there.
>>
>> I'm not positive that that's the correct approach, however, especially
>> given that I suspect that /boot will need to be visible to GRUB and may
>> thus need to be outside the LVM.
>

> grub can find a root in LVM, but needs /boot to be outside.
> /boot in an mdadm RAID1 is a good idea there.

So, three partitions, then?

1. efi
2. /boot
3. LVM

And define everything else inside the LVM. (In practice, "everything
else" here will be /, /var, /tmp, and possibly swap. /home will be on a
separate RAID array.)

It'll still probably be interesting figuring out what to specify as the
destination for the GRUB install, but if I'm lucky it might Just Work.

[Dan Ritter]

The Wanderer wrote:
> On 2021-03-02 at 07:54, Dan Ritter wrote:
>
> > You'll want to create the following partitions on each,
> > identically:
> >
> > 1 efi - type efi
> > 2 boot (or boot/root) - type MDADM volume
> > 3 root, if using separate boot - type MDADM volume
> > 4 swap - type MDADM volume
> >
> > Then you go to the mdadm setup and create MDADM RAID1 devices
> > out of each pair of boot, root and swap.
>
> Out of interest, how would you modify this for someone who intends to
> run LVM on top of the RAID-1 being defined here, and wants to define as
> many of the partitions as possible (but at least /) at the LVM level
> rather than the mdadm level?
>
> I'm guessing that the answer would be to define the EFI partition here
> directly, then define one further partition as an appropriate type, set
> up LVM inside that, and define the further partitions from there.
>
> I'm not positive that that's the correct approach, however, especially
> given that I suspect that /boot will need to be visible to GRUB and may
> thus need to be outside the LVM.

grub can find a root in LVM, but needs /boot to be outside.
/boot in an mdadm RAID1 is a good idea there.

-dsr-

[Charles Curley]

On Tue, 2 Mar 2021 09:50:22 -0500
Dan Ritter <d...@randomstring.org> wrote:

> > I'm not positive that that's the correct approach, however,
> > especially given that I suspect that /boot will need to be visible
> > to GRUB and may thus need to be outside the LVM.
>
> grub can find a root in LVM, but needs /boot to be outside.
> /boot in an mdadm RAID1 is a good idea there.

I'm not sure you need a separate /boot partition. I am using Bullseye
weekly builds, so I may have a more recent d-i and grub than you do.
Also this is on i386 on a small (60GB) drive, and a DOS type MBR.

I have one LVM partition, with swap and root in that. The LVM partition
starts at sector 2048, so I expect grub resides on the first 2048
sectors.

I suspect one can get an idea of things grub can do by
listing /boot/grub/i386-pc/.

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[Felix Miata]

Dan Ritter composed on 2021-03-02 07:54 (UTC-0500):

> Dave Sherohman wrote:

>> I've got a new server...

> You'll want to create the following partitions on each,
> identically:

> 1 efi - type efi

Not RAID I presume?

> 2 boot (or boot/root) - type MDADM volume

I never have /boot on RAID, but that's just me I suppose.

> 3 root, if using separate boot - type MDADM volume

Agree.

> 4 swap - type MDADM volume

Think about this:

To start with, RAID1 is marginally slower than ordinary filesystems on partitions.

Does a server use swap to hibernate?

Does a server /want/ to use swap for anything?

Does anyone want swap on any type of SSD?

If I was to put swap on NVME, I would not use a RAID, just like on my old RAID
systems on HDDs. If I had 32G or more RAM, I'd just let the kernel swap to file
instead of devoting discrete partition space to swap.
--
Evolution as taught in public schools, like religion,
is based on faith, not on science.

Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata *** http://fm.no-ip.com/

[Andrew M.A. Cater]

On Tue, Mar 02, 2021 at 05:57:37AM -0600, Dave Sherohman wrote:
> I've got a new server and am currently fighting with the Debian 10
> installer (build 20190702) in my attempts to get it up and running.
> After much wailing and gnashing of teeth, I managed to get it to stop
> complaining about being unble to mount /boot/efi and complete the
> "Partition disks" step successfully, but now I'm completely stuck on the
> "Install GRUB" step.
>
> The GUI installer shows the error:
>
> Unable to install GRUB in dummy
> Executing 'grub-install dummy' failed.
>
> Checking the syslog output on virtual console 4 shows a bunch of
> os-prober activity (as expected), then finally:
>
> Installing for x64_64-efi platform.
> grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`.
> error: Running 'grub-install --force "dummy"' failed.

Check the settings for the disk in BIOS equivalent - I had lots of problems
with an HP because I'd inadvertently told the HP software to take over
partitioning of the disks and put them into an array.

Andy C.

[Dave Sherohman]

General update on my situation:

Just before knocking off for the day, I tried telling the installer to
finish up without installing a boot loader, then used the netinst USB
stick as a rescue disk to boot my installed system. From there, I was
able to use the regular grub-install program (instead of the install
environment's grub-installer script).

grub-install ran with no errors and I now have grub in the (separate,
non-RAIDed) EFI partitions of both nvme drives... but it's still not
directly bootable. Trying to boot directly from nvme drops me into a
grub command line.

Thinking that this may be an initrd or similar issue, I tried
reinstalling the kernel image and grub-efi-amd64 packages, so that their
postinst scripts would run and rebuild initrd and reinstall grub to the
drives, but that had no effect. Still just getting the grub shell when
I boot from nvme.

Any tips on making use of the grub shell to make further progress, such
as getting the system to boot in non-rescue mode (i.e., not chrooted
from the installer)? The help information available in the grub shell
itself isn't terribly useful because it scrolls off the screen with no
(obvious) pager or scrollback buffer.

Alternately, suggestions for things I can try in the chroot environment
would also be good, since that's considerably less restrictive than the
installer environment I was trying to work within yesterday.


I guess the most obvious explanation for the current grub issues is that
grub isn't smart enough to boot an mdadm filesystem directly and I need
to repartition with a non-RAID /boot, but I don't consider that a
desirable solution, since it would then leave me with no /boot if the
device holding that partition dies.


(No new text below this point, just my original post for context.)

On Tue, Mar 02, 2021 at 05:57:37AM -0600, Dave Sherohman wrote:

--
Dave Sherohman

[Dave Sherohman]

So, after my last message, I ran across a general mdadm setup guide at
https://raid.wiki.kernel.org/index.php/Setting_up_a_(new)_system
and, in the "booting" section of that document, it said to add
GRUB_CMDLINE_LINUX="domdadm"
to /etc/default/grub. Didn't have that, so I added it, did the checks
mentioned in the document, and reinstalled grub to both EFI partitions.

Still didn't work.

Then I found https://linoxide.com/linux-how-to/grub-rescue-commands/ and
was able to start the installed kernel with the command sequence:

set root=(md/1)
linux (md/1)/vmlinuz root=/dev/md1
initrd (md/1)/boot/initrd.img-4.19.0-14-amd64
insmod normal
boot

This still ultimately died with an error saying the EFI partition had an
"Invalid FSINFO signature". Searches on that phrase mostly seemed to
talk about secure boot problems, so I double-checked the bios options
and secure boot was off, so that wasn't it.

So I went back to the linoxide.com document and went through all the
steps that I hadn't already done manually: Running os-prober and
update-grub. Then grub-install to the EFI partitions again, reboot,
and... success!


Based on this, I'm guessing that the original problem was that the
installer forgot to include mdadm support in its grub options, even
though it was configured with an mdadm boot device. And then I missed a
couple steps after adding mdadm support, so it didn't all get installed
to the EFI partitions correctly.

--
Dave Sherohman

[Marc Auslander]

On 3/3/2021 6:30 AM, Dave Sherohman wrote:

>
> Based on this, I'm guessing that the original problem was that the
> installer forgot to include mdadm support in its grub options, even
> though it was configured with an mdadm boot device. And then I missed a
> couple steps after adding mdadm support, so it didn't all get installed
> to the EFI partitions correctly.
>

One potential gotcha. When you boot from an mdadm file system
containing /boot/grub, grub will not write to the file system. In
particular, it will not update grub/grubenv even if you have a save_env
line in grub.cfg. So if you use grub-reboot to specify an alternate
line in grub.cfg, you need to reset grubenv afterwards. I do this in a
root @reboot cron job.

If you don't know what I'm talking about, you probably don't need to
worry about this.

[Andrei POPESCU]

On Ma, 02 mar 21, 14:01:52, Felix Miata wrote:
>
> To start with, RAID1 is marginally slower than ordinary filesystems on partitions.

This is true for some workloads, for others it can be significantly
faster.

https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/


Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

[Felix Miata]

Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200):

> Felix Miata wrote:

>> To start with, RAID1 is marginally slower than ordinary filesystems on partitions.

> This is true for some workloads, for others it can be significantly
> faster.

> https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/

I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in any of
the graphs there, and the subject of RAID1 barely touched, basically describing
its purely mirror topology and little else. I can't recall any graph there that
described performance of anything other than single disk, RAID6, and RAID10. Do
you know of, or can you provide a reference to, any way RAID1 performance can be
better than single disk?

[Dan Ritter]

Felix Miata wrote:
> Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200):
>
> > Felix Miata wrote:
>
> >> To start with, RAID1 is marginally slower than ordinary filesystems on partitions.
>
> > This is true for some workloads, for others it can be significantly
> > faster.
>
> > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/
>
> I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in any of
> the graphs there, and the subject of RAID1 barely touched, basically describing
> its purely mirror topology and little else. I can't recall any graph there that
> described performance of anything other than single disk, RAID6, and RAID10. Do
> you know of, or can you provide a reference to, any way RAID1 performance can be
> better than single disk?

Sure.

https://calomel.org/zfs_raid_speed_capacity.html

First two lines of the first table of results:

1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s
2x 4TB, mirror (raid1), 3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s

Which is as theory predicts: slight reduction in write
performance (write being dominated by disk cache -> disk, not
system -> disk cache), and approximate doubling in read
performance.

-dsr-

[to...@tuxteam.de]

On Wed, Mar 03, 2021 at 05:16:14PM -0500, Felix Miata wrote:
> Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200):
>
> > Felix Miata wrote:
>
> >> To start with, RAID1 is marginally slower than ordinary filesystems on partitions.
>
> > This is true for some workloads, for others it can be significantly
> > faster.
>
> > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/
>

> [...] Do

> you know of, or can you provide a reference to, any way RAID1 performance can be
> better than single disk?

I can just provide a hint: when accessing mirrored disks, reading is
done whenever the first disk finishes; writing whenever both disks
are done. So reading is potentially faster, writing potentially
slower.

That's about latency. The throughput shouldn't change too much (unless
you're combining very different disks, but that'd be asking for it).

Compose this with several layers of buffering and I doubt you'll be
able to make out any significant difference though.

No experimental data to back this up, alas.

> Evolution as taught in public schools, like religion,
> is based on faith, not on science.

OK. Now all bets wrt RAID are off, anyway.

Cheers
- t

[to...@tuxteam.de]

On Wed, Mar 03, 2021 at 05:40:39PM -0500, Dan Ritter wrote:
> Felix Miata wrote:
> > Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200):
> >
> > > Felix Miata wrote:

> > [...] Do

> > you know of, or can you provide a reference to, any way RAID1 performance can be
> > better than single disk?
>
> Sure.
>
> https://calomel.org/zfs_raid_speed_capacity.html
>
> First two lines of the first table of results:
>
> 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s
> 2x 4TB, mirror (raid1), 3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s
>
> Which is as theory predicts: slight reduction in write
> performance (write being dominated by disk cache -> disk, not
> system -> disk cache), and approximate doubling in read
> performance.

Thanks. Real data :)

The doubling in read throughput is somewhat surprising to me. Some
learning to do, it seems.

Cheers
- t

[Anssi Saari]

Dave Sherohman <da...@sherohman.org> writes:

> Any tips on making use of the grub shell to make further progress, such
> as getting the system to boot in non-rescue mode (i.e., not chrooted
> from the installer)? The help information available in the grub shell
> itself isn't terribly useful because it scrolls off the screen with no
> (obvious) pager or scrollback buffer.

There is a pager in Grub but it really isn't obvious. For the future
it's enabled by set pager=1. Why it's not on by default I don't know...

[Greg Wooledge]

to...@tuxteam.de (to...@tuxteam.de) wrote:
> > 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s
> > 2x 4TB, mirror (raid1), 3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s

> Thanks. Real data :)
>
> The doubling in read throughput is somewhat surprising to me. Some
> learning to do, it seems.

There are two copies of everything. So, if you need to read a big
file, you can read half of it from disk 1 and the other half from disk 2,
roughly simultaneously.

[to...@tuxteam.de]

Makes sense, thanks.

Cheers
- t

[Anssi Saari]

I have a vague memory of reading once that mdraid doesn't stripe reads
to a single file. I.e. with raid1 you get the double speed if you read
different files (or different parts of disk) but reading a single file,
no.

So is this different in zfs? I guess I could test this myself. The way
Bonnie++ was used for those results above is definitely parallel.

[Darac Marjal]

On 05/03/2021 09:25, Anssi Saari wrote:
> Greg Wooledge <gr...@wooledge.org> writes:
>
>> to...@tuxteam.de (to...@tuxteam.de) wrote:
>>>> 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s
>>>> 2x 4TB, mirror (raid1), 3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s
>>> Thanks. Real data :)
>>>
>>> The doubling in read throughput is somewhat surprising to me. Some
>>> learning to do, it seems.
>> There are two copies of everything. So, if you need to read a big
>> file, you can read half of it from disk 1 and the other half from disk 2,
>> roughly simultaneously.
> I have a vague memory of reading once that mdraid doesn't stripe reads
> to a single file. I.e. with raid1 you get the double speed if you read
> different files (or different parts of disk) but reading a single file,
> no.

RAID won't know what's a file or not. It works below the file system
layer, so only concerns itself with blocks. For any particular block, a
RAID1 array has N copies of that block to read from. It makes no sense
to read all of those copies - imagine if you were really paranoid and
had a 10-disk RAID array. Instead, the driver will choose one of the
constituent drives and read the block from that drive. So, if your file
is small, then yes, you won't see any performance benefit. But if your
file is larger than a block, or if you want to access more than one file
at once, then RAID can read the second block from a different drive.
Remember that  the I in RAID stands for Inexpensive. RAID assumes that
the disk is slow, the bus is faster and the applications are impatient  :)

>
> So is this different in zfs? I guess I could test this myself. The way
> Bonnie++ was used for those results above is definitely parallel.

ZFS will work in much the same manner, except when performing a "scrub".
During a scrub, all copies of all blocks are read and compared. If a
block differs across the mirrors, then ZFS has a decision to make: if
there are more identical copies than different copies, ZFS assumes that
the identical copies are good (hence why having a three-way mirror is
recommended). Alternatively, ZFS uses checksums on stored data. If the
block doesn't match the checksum, then ZFS can also recover that data.
More details here
https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integrity
>

[Andy Smith]

Hello,

On Fri, Mar 05, 2021 at 10:04:34AM +0000, Darac Marjal wrote:
> So, if your file is small, then yes, you won't see any performance
> benefit. But if your file is larger than a block, or if you want
> to access more than one file at once, then RAID can read the
> second block from a different drive.

However, Linux MD RAID-1 will only ever achieve the read performance
of a single device per thread, so in order to get the read
performance benefit of multiple copies you either need to use
multiple reading threads / processes or else use RAID-10 instead.

https://www.spinics.net/lists/raid/msg47240.html

ZFS does not suffer from this.

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[Andrei POPESCU]

On Mi, 03 mar 21, 17:16:14, Felix Miata wrote:
> Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200):
>
> > Felix Miata wrote:
>
> >> To start with, RAID1 is marginally slower than ordinary filesystems on partitions.
>
> > This is true for some workloads, for others it can be significantly
> > faster.
>
> > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/
>
> I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in any of
> the graphs there, and the subject of RAID1 barely touched, basically describing
> its purely mirror topology and little else.

Quote from the article:

In our performance charts, we show a line from two disks through
eight disks for RAID10. The first datapoint, for two disks, is
obviously a simple RAID1. The datapoints for four, six, and eight
disks are RAID10. We draw a line through all points for ease of
interpretation—but there is no three, five, or seven disk RAID10 in
the actual data, for hopefully obvious reasons.

[Dave Sherohman]

On Wed, Mar 03, 2021 at 10:28:34AM -0500, Marc Auslander wrote:
> One potential gotcha. When you boot from an mdadm file system containing
> /boot/grub, grub will not write to the file system. In particular, it will
> not update grub/grubenv even if you have a save_env line in grub.cfg. So if
> you use grub-reboot to specify an alternate line in grub.cfg, you need to
> reset grubenv afterwards. I do this in a root @reboot cron job.
>
> If you don't know what I'm talking about, you probably don't need to worry
> about this.

I don't *know* know what you're talking about, but I think I get the
gist of it. I'm not doing anything fancy enough to have to deal with
the details of grubenv and grub-reboot, but your initial comment about
grub refusing to write to a mdadm filesystem is consistent with what I
was seeing, and it's nice to have confirmation that it actually does
work that way.

--
Dave Sherohman

[Nicholas Geovanis]

Dave the issues you describe dealing with mdadm setup, well that's why I eventually

just switched to LVM-based mirroring and striping. Early on, LVM wasn't supported as a boot

volume, but the original code base had supported that on HP-UX and AIX. So years ago now,

that restriction was lifted. Another mdadm issue is just the use of it in a kind of

"oral tradition": it's not that setup is hard, it just still seems like magic spells after all this time.

--
Dave Sherohman