Unplanned Maintenance - 27 Sep 2023

[Shawn Webb]

Hey all,

Our server room unexpectedly lost power. So I'm going to take this as
an opportunity to install new 2.5Gbps NICs in our servers.

Our infrastructure is currently down. I'll let everyone know when it's
back up.

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

[Shawn Webb]

On Wed, Sep 27, 2023 at 07:36:28PM -0400, Shawn Webb wrote:
> Hey all,
>
> Our server room unexpectedly lost power. So I'm going to take this as
> an opportunity to install new 2.5Gbps NICs in our servers.
>
> Our infrastructure is currently down. I'll let everyone know when it's
> back up.

Hey all,

Most of the infrastructure is back up. Unfortunately, I'm going to
need to rebuild the 14-STABLE build VMs. If you need/want to download
HardenedBSD 14-STABLE, please use one of our mirrors:
https://hardenedbsd.org/content/mirrors

[Ladar Levison]

I tried creating a new HBSD 14 vagrant box, with a config that is almost identital to the one used for HBSD 13, but the install process is failing. It appears broken, but I can't be sure.

From what I could tell, when I provide a config script to BSD install, it is skipping over the 'mount' step. That's the step where the target disk gets mounted as '/mnt' . Despite the failure, the installer continues as if nothing is wrong, only to fail when  '/mnt' runs out of space... (I had it setup with a small in-memory rw allocation).

I tried a every disk configuration I could find. MBR and GPT tables, auto parition, and manully paritioned. And nothing seems to work. The disk DOES get partitioned, it just doesn't get mounted. After the install fails, you can run 'bsdinstall mount' and the tartget will get mounted, but using the script directive to automate the process always failed.

I'm wondering if someone ekse has a working install config I can try? And if this problem is a bug, or a byproduct of something new feature/or hanged in v14?

Note that I haven't tried creating a box for FreeBSD 14 yet, so I don't know if the problem is upstream or not. You can find install config I'm using here:

https://raw.githubusercontent.com/lavabit/robox/91348aa7476bd1ae5d61288f42d46c12076efefc/http/generic.hardenedbsd.vagrant.cfg

Also, is there still an archive of the HBSD 12 packages? Either a permanent archive I can point the HBSD 12 vagrant boxes at, or one that I can download and preserve/mirror before it disappears forever?

L~

> (my until it runs out of the disk space (the I provide a config script to  params to

[Shawn Webb]

On Wed, Sep 27, 2023 at 09:30:00PM -0500, 'Ladar Levison' via HardenedBSD Users wrote:
> I tried creating a new HBSD 14 vagrant box, with a config that is almost
> identital to the one used for HBSD 13, but the install process is failing.
> It appears broken, but I can't be sure.
>
> From what I could tell, when I provide a config script to BSD install, it is
> skipping over the 'mount' step. That's the step where the target disk gets
> mounted as '/mnt' . Despite the failure, the installer continues as if
> nothing is wrong, only to fail when  '/mnt' runs out of space... (I had it
> setup with a small in-memory rw allocation).

How small is small? I'm unsure less than 2GB would be sufficient.

>
> I tried a every disk configuration I could find. MBR and GPT tables, auto
> parition, and manully paritioned. And nothing seems to work. The disk DOES
> get partitioned, it just doesn't get mounted. After the install fails, you
> can run 'bsdinstall mount' and the tartget will get mounted, but using the
> script directive to automate the process always failed.
>
> I'm wondering if someone ekse has a working install config I can try? And if
> this problem is a bug, or a byproduct of something new feature/or hanged in
> v14?
>
> Note that I haven't tried creating a box for FreeBSD 14 yet, so I don't know
> if the problem is upstream or not. You can find install config I'm using
> here:
>
> https://raw.githubusercontent.com/lavabit/robox/91348aa7476bd1ae5d61288f42d46c12076efefc/http/generic.hardenedbsd.vagrant.cfg

Unfortunately, I don't use Vagrant, so I'm not of much use there.
Regardless of Vagrant use, though, you might have stumbled across a
bug in bsdinstall. If you can look at the fourth virtual terminal
(Ctrl + Alt + F4), you'll see bsdinstall's debug output. I believe
that debug output is also saved as a file somewhere, but I don't
currently remember where.

If you can find the bsdinstall log file and send it over, I'd be happy
to help take a look.

>
> Also, is there still an archive of the HBSD 12 packages? Either a permanent
> archive I can point the HBSD 12 vagrant boxes at, or one that I can download
> and preserve/mirror before it disappears forever?

Unfortunately, we do not have the capability to archive package builds
for unsupported releases. The package repo for HardenedBSD 12-STABLE
no longer exists.

[Shawn Webb]

On Wed, Sep 27, 2023 at 09:45:09PM -0400, Shawn Webb wrote:
> On Wed, Sep 27, 2023 at 07:36:28PM -0400, Shawn Webb wrote:
> > Hey all,
> >
> > Our server room unexpectedly lost power. So I'm going to take this as
> > an opportunity to install new 2.5Gbps NICs in our servers.
> >
> > Our infrastructure is currently down. I'll let everyone know when it's
> > back up.
>
> Hey all,
>
> Most of the infrastructure is back up. Unfortunately, I'm going to
> need to rebuild the 14-STABLE build VMs. If you need/want to download
> HardenedBSD 14-STABLE, please use one of our mirrors:
> https://hardenedbsd.org/content/mirrors

I restarted the package builds:

13-STABLE:
ttps://hbsd-pkg-13-stable-01.hardenedbsd.org/build.html?mastername=hbsd13-default&build=2023-09-28_15h48m40s

15-CURRENT:
https://hbsd-pkg-current-01.hardenedbsd.org/build.html?mastername=current-default&build=2023-09-28_15h53m38s

Missing is 14-STABLE since I need to rebuild that VM. I hope to start
that next week.

[Ladar Levison]

On 9/29/23 10:07 AM, Shawn Webb wrote:

On Wed, Sep 27, 2023 at 09:30:00PM -0500, 'Ladar Levison' via HardenedBSD Users wrote:

How small is small? I'm unsure less than 2GB would be sufficient.

Using v13 config, /mnt was being setup as a 100 MB memory partition, but that should be replaced by the 128 GB virtual disk during install.

I tried a number of permutations, including increasing the memory /mnt memory limit to 1 GB and removing the memory partition setup altogether.

If you can find the bsdinstall log file and send it over, I'd be happy
to help take a look.

I saved the debug logs out to a file, and looked at them, but they didn't provide much help, other than to say the install had run out of space. Like I said, I think the issue is earlier, where it would normally mount the new disk, but I didn't see any errors earlier in the log.

I'll see if I can grab an error log when I get a chance.

Unfortunately, I don't use Vagrant, so I'm not of much use there.
Regardless of Vagrant use, though, you might have stumbled across a
bug in bsdinstall. If you can look at the fourth virtual terminal
(Ctrl + Alt + F4), you'll see bsdinstall's debug output. I believe
that debug output is also saved as a file somewhere, but I don't
currently remember where.

This isn't a vagrant issue. At this point it's a problem with an automated install on QEMU. All of the hypervisors (Vbox/VMware, etc) were failing, and I assumed because of the same issue, but I only looked closely at the QEMU install, since I have that platform setup locally.

That's why I was asking if you had a working bsdinstall script that you used for testing, that you know works, which I could try running?

L~

[Ladar Levison]

Hi Shawn,

See the attached log files. The bsdinstall_log file is the standard debug log, and as I mentioned previously, it doesn't provide any useful information. Slightly more helpful is the installer.log file, which captures the raw text written to the console, but it's a bit jumbled. Despite the jumble you still see where it runs 'scriptedpart da0' and succeeds. Then, further on, it fails when it runs out of space. I'm also going to include a screenshot of the VM console showing the error on the console afer the install fails.

I did a little more testing. If I run 'bsdinstall autopart' and click enter a couple of times to partition the disk, it gets partitiioned and I can then run 'bsdinstall mount' to get the drives mounted. I don't know for sure, but this appears to write an fstab file out, which doesn't get written when using scriptedpart. Unfortunately, I don't know how to run the scripted installer from this point.

I can also run 'bsdinstall' and setup the system interactively, without any problems.

L~

[Ladar Levison]

On 10/1/23 8:04 PM, 'Ladar Levison' via HardenedBSD Users wrote:

Despite the jumble you still see where it runs 'scriptedpart da0' and succeeds. Then, further on, it fails when it runs out of space. I'm also going to include a screenshot of the VM console showing the error on the console afer the install fails.

I did more digging and managed to isolate the bug/problem to the '/usr/libexec/bsdinstall/scriptedpart'utility... that program should be writing out the partition table it creates to an fstab file once it's done with the disk allocation, but isn't. Without the fstab file, the  target drive doesn't get mounted.

I tried various strategies to get the fstab file written out, but nothing seemed to work. I was able to confirm the version of scriptedpart on the HBSD v13 install media works correctly. I also tried increasing the temp filesystem on /mnt to 4GB and when I do that, the install finishes, but HBSD is installed onto a temp filesystem, and gets lost on remount/reboot. But that confirms the problem is with the scriptedpart, and not a problem with some other part of the install.

I also tracked down this code snippet inside the /usr/libexec/bsdinstall/script file:

# Make partitions
rm -f $PATH_FSTAB
touch $PATH_FSTAB
if [ "$ZFSBOOT_DISKS" ]; then
	bsdinstall zfsboot
else
	bsdinstall scriptedpart "$PARTITIONS"
fi
bsdinstall mount

As you can see bsdinstall mount is getting executed properly, it just fails to do anything without a PATH_FSTAB file to work with.

The scriptedpart tool is compiled, and I don't have time to track down the code for it, but I suspect it's an upstream FreeBSD installer bug.

Finally, I was able to successfully complete an automated install of HBSD v14 by setting up a unionfs and replacing the 'scriptedpart' binary, with from the HBSD v13 install media (note this workaround requires you to also copy the libdialog.so.9 dependency)...

Shawn, this is a sufficiently awful (and unmaintainable) fix. so I'd like to avoid using it. I know your busy rebuilding infrastructure, but let me know when you have time to take a look, and estimate when it wil get fixed?

L~

[Shawn Webb]

Part of me wonders if this issue exists in FreeBSD, too. Would you be
able to test your script on a FreeBSD 14 snapshot installer image?

If it's only reproducible in HardenedBSD, I'd be happy to help fix it.
After getting the 14-STABLE build infrastructure back online (hoping
by the end of this coming weekend), I'll make this my first priority.

I suspect that if it is a bug in HardenedBSD's bsdinstall (our changes
are minimal, mainly dealing with changing branding from FreeBSD to
HardenedBSD), it would likely take me a few days to fix it.

[Ladar Levison]

On 10/2/23 8:49 AM, Shawn Webb wrote:

Part of me wonders if this issue exists in FreeBSD, too. Would you be
able to test your script on a FreeBSD 14 snapshot installer image?

I don't have an automated config for FreeBSD 14 yet. I usually wait until a distro makes their release official before adding it to the build pipeline. But I'll give it a shot if time permits.

If it's only reproducible in HardenedBSD, I'd be happy to help fix it.
After getting the 14-STABLE build infrastructure back online (hoping
by the end of this coming weekend), I'll make this my first priority.

I feel your pain. If I wasn't already overloaded this week I'd try and help. Certainly let me know if you want me to update the rsync mirror config, to, for example, start mirroring the packages in addition to the installers.

I suspect that if it is a bug in HardenedBSD's bsdinstall (our changes
are minimal, mainly dealing with changing branding from FreeBSD to
HardenedBSD), it would likely take me a few days to fix it.

Like I said, I didn't track down the code files that are used to build this utility. But I'm guessing it doesn't involve much code. You might start by comparing the v13 version of the file to the v14 file, since I've already confirmed the former version works.

I did a bit of string analysis on the binaries looking for clues and/or possible options for making it work. What found is that partedit shares a lot of strings/logic including its use of the FSTAB_PATH variable. I was able to run that tool and save out drive geometry just fine using the v14 binaries (alas, manually creating an fstab file for an auto install didn't seem to work though).

It's possible a minor change, perhaps a misaligned patch, has something out of alignment in scriptedpart which is why it isn't writing out the fstab. Either way, it might help to also compare scriptedpart with partedit since they probably share some copy/paste code between them.

L~

[Ladar Levison]

On 10/2/23 9:41 AM, 'Ladar Levison' via HardenedBSD Users wrote:

On 10/2/23 8:49 AM, Shawn Webb wrote:

Part of me wonders if this issue exists in FreeBSD, too. Would you be
able to test your script on a FreeBSD 14 snapshot installer image?

I don't have an automated config for FreeBSD 14 yet. I usually wait until a distro makes their release official before adding it to the build pipeline. But I'll give it a shot if time permits.

I tried out FreeBSD 14.0-BETA4, and it failed with the exact same error. I have to admit I was a little surprised. I figured lots of people must be running automated FreeBSD installs. Which led me to think that if there was a bug, it would have been found and/or fixed. Just to be sure, I also tried the 14.0-BETA1 installer. It failed as well. (13.2 guest images get built as part of the robox pipeline, so I know they work).

Attached are screenshots of the install error (it's identical), and a second screenshot which shows the properly partitioned drive, along with an empty fstab file, thus confirming the bug.

I'll open a FreeBSD bug report. I don't know how closely you work with the FreeBSD team, but 14.0 is due to release at the end of month, so perhaps we should email some of the of release engineering team members directly?

P.S. While not as critical, I can confirm also these two bug reports with the scriptedpart utility, are affecting affect the HBSD installer. Running scriptedpart with DEFAULT and with an empty PARTITION value were two of the first variations I tried while tracking down this bug. I didn't realize at the time the segfaults were a separate bug.

Bug 273220 - "scriptedpart DEFAULT" will segfault with unformatted disks
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273220

Bug 273218 - scriptedpart will segfault if not provided with any arguments
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273218

[Shawn Webb]

On Mon, Oct 02, 2023 at 02:10:36PM -0500, Ladar Levison wrote:
>
> On 10/2/23 9:41 AM, 'Ladar Levison' via HardenedBSD Users wrote:
> > On 10/2/23 8:49 AM, Shawn Webb wrote:
> > > Part of me wonders if this issue exists in FreeBSD, too. Would you be
> > > able to test your script on a FreeBSD 14 snapshot installer image?
> >
> > I don't have an automated config for FreeBSD 14 yet. I usually wait
> > until a distro makes their release official before adding it to the
> > build pipeline. But I'll give it a shot if time permits.
> >
> I tried out FreeBSD 14.0-BETA4, and it failed with the exact same error. I
> have to admit I was a little surprised. I figured lots of people must be
> running automated FreeBSD installs. Which led me to think that if there was
> a bug, it would have been found and/or fixed. Just to be sure, I also tried
> the 14.0-BETA1 installer. It failed as well. (13.2 guest images get built as
> part of the robox pipeline, so I know they work).

Thanks for testing on FreeBSD! Since the bug stems from there, I'll
let them come to a solution that best works for them.

>
> Attached are screenshots of the install error (it's identical), and a second
> screenshot which shows the properly partitioned drive, along with an empty
> fstab file, thus confirming the bug.
>
> I'll open a FreeBSD bug report. I don't know how closely you work with the
> FreeBSD team, but 14.0 is due to release at the end of month, so perhaps we
> should email some of the of release engineering team members directly?

You might want to fire off an email to freebsd-current{AT}freebsd.org
and/or freebsd-stable{AT}freebsd.org. Given that they're actively in
the 14.0 release engineering cycle, they're closely monitoring
reports on those mailing lists.

>
> P.S. While not as critical, I can confirm also these two bug reports with
> the scriptedpart utility, are affecting affect the HBSD installer. Running
> scriptedpart with DEFAULT and with an empty PARTITION value were two of the
> first variations I tried while tracking down this bug. I didn't realize at
> the time the segfaults were a separate bug.
>
> Bug 273220 - "scriptedpart DEFAULT" will segfault with unformatted disks
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273220
>
> Bug 273218 - scriptedpart will segfault if not provided with any arguments
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273218

If you have any other reports to file, it might be worth filing them
in FreeBSD's BugZilla before reaching out to their mailing lists.

[Ladar Levison]

On 10/2/23 9:41 AM, 'Ladar Levison' via HardenedBSD Users wrote

I'll open a FreeBSD bug report. I don't know how closely you work with the
FreeBSD team, but 14.0 is due to release at the end of month, so perhaps we
should email some of the of release engineering team members directly?

On 10/2/23 8:49 AM, Shawn Webb wrote:

You might want to fire off an email to freebsd-current{AT}freebsd.org
and/or freebsd-stable{AT}freebsd.org. Given that they're actively in
the 14.0 release engineering cycle, they're closely monitoring
reports on those mailing lists.

If you have any other reports to file, it might be worth filing them
in FreeBSD's BugZilla before reaching out to their mailing lists.

I opened up the bug report. I'll have to let someone else pick up the issue for now. While it was a tricky puzzle, and served as a nice distraction/procrastination, which forced me to reverse engineer/study the bsdinstall code, I ran out of time to work on it last week.

I'm moving the Robox HardenedBSD targerts to a away from tracking release versions, to a single ta4rget, aka 'hardenedbsd13' went to 'hardenedbsd' which I was supposed to use the 14.0 release. But this bug means I wasn't able to get a working install, so I've been releasing v13 to that target. What's nice about not having to track a specific release, is that I can wait until the bugs are fixed, before migrating the that target to 14.0.

I only investigated the failure to initially confirm it wasn't an issue with my installer config, and then once I confirmed it was indeed a bug, I wanted to make sure it was reported along with enough details that someone could find and fix it. I know I personally find it hard to act when an issue gets reported, but the bug report lacks any useful information that I can use to fix it.

L~

[Shawn Webb]

On Wed, Sep 27, 2023 at 09:45:09PM -0400, Shawn Webb wrote:

> On Wed, Sep 27, 2023 at 07:36:28PM -0400, Shawn Webb wrote:
> > Hey all,
> >
> > Our server room unexpectedly lost power. So I'm going to take this as
> > an opportunity to install new 2.5Gbps NICs in our servers.
> >
> > Our infrastructure is currently down. I'll let everyone know when it's
> > back up.
>
> Hey all,
>
> Most of the infrastructure is back up. Unfortunately, I'm going to
> need to rebuild the 14-STABLE build VMs. If you need/want to download
> HardenedBSD 14-STABLE, please use one of our mirrors:
> https://hardenedbsd.org/content/mirrors

The 14-STABLE build infrastructure is back online. The mirrors have
been updated with the latest build. A new package build is running.

[Ladar Levison]

Hi Shawn,

On 10/2/23 2:52 PM, 'Ladar Levison' via HardenedBSD Users wrote:

I opened up the bug report. I'll have to let someone else pick up the issue for now. While it was a tricky puzzle, and served as a nice distraction/procrastination, which forced me to reverse engineer/study the bsdinstall code, I ran out of time to work on it last week.

It looks like the installer bug was fixed last week, just in time to make it into the 14.0=RC2 install media. I tested the FreeBSD 14.0 RC2 installer and it's working again. 

So I thought I'd follow up and make sure you knew it was time to resync with the upstream code,. And see if you needed me to test an updated HardenedBSD 14 installer?

L~

[Shawn Webb]

Hey Ladar,

Thanks for following up.

We sync with FreeBSD every six hours, so we've had the fix in our
source tree since not too long after the fix was introduced.

We build every first and fourteenth of every month, so you should see
FreeBSD's fix in the next build. If you'd like to go ahead and skip
the wait line and do your own build, let me know how it goes. :-)

I really do appreciate your assistance in filing the report and doing
causal analysis. If there's anything you need from me, let me know.

[Ladar Levison]

On 10/23/23 9:24 PM, Shawn Webb wrote:

We sync with FreeBSD every six hours, so we've had the fix in our
source tree since not too long after the fix was introduced.

We build every first and fourteenth of every month, so you should see
FreeBSD's fix in the next build. If you'd like to go ahead and skip
the wait line and do your own build, let me know how it goes. :-)

I got tired of ISO files disappearing during a pipeline build, so I wrote a little pre-flight script function which updates the install media for distros with frequent updates automatically (Arch/Gentoo/HBSD/CentOS stream/etc). 

I run that update logic pretty often, to keep the templates updated. So I should see the new media as soon as it hits the mirror servers.

That said, I didn't realize the install media was built on a twice-monthly schedle/ Thanks for explaining. I'll look for the build-4 ISOs after the 1st, and if they work, update my scripts so I can start shipping HBSD 14 VMs.

L~