Status of Infrastructure

[Shawn Webb]

Hey all,

I'd like to say thank you to the community for standing by with
patience as we perform emergency maintenance on our infrastructure.

I'd like to take just a brief moment to tell everyone where we are,
how we got here, and where we're going.

An attempted upgrade failed of our nightly build server, a server
which provides a pristine copy of build artifacts from
jenkins.hardenedbsd.org and installer.hardenedbsd.org. It is the
source-of-truth location for nightly builds and supported releases
alike. Attempting to reboot into the last known-working ZFS Boot
Environment (BE) failed due to a regression FreeBSD introduced into
the bootloader from 11-STABLE to 12-STABLE: the bootloader in
12-STABLE does not support booting from root ZFS pools that utilize
partionless schemes (ie, /dev/ada0 rather than /dev/ada0p1).

Given the delicate nature of bootloaders, a decision was reached that
we should rebuild the server by performing a reinstallation of
12-STABLE with an entirely new ZFS pool, one that used partioned
disks. This decision necesitates the backup of existing data. Since
we've never performed an initial backup before, we needed now to
perform the initial backup.

We had to back up 1.52TB of data from a datacenter in New York City,
New York, USA to a datacenter in Maryland, USA. The Maryland
datacenter has limited bandwidth. As of Monday, 26 Aug 2019, the
backup completed.

That brings us to today. Today, I purchased hard drives of sufficient
capacity that would allow us to both reproduce the issue and test
restoration procedures. I plan to start this work on Saturday, 31 Aug
2019.

Once testing and refining the restoration procedures results in
success, we will perform the same restoration procedures on the
nightly build server. This means restoring that entire 1.52TB over the
limited pipe again. Downloading took multiple days. It's a safe bet
uploading will as well.

So, if all goes well, we're looking at another 1.5 weeks of downtime.

I have published a build of the last 12-STABLE/amd64 release for
general consumption:

https://hardenedbsd.org/~shawn/2019-08-24_12-stable_amd64/

The build artifacts are signed with the same signature referenced in
my email signature below.

I will be kicking off a new package build for 12-STABLE/amd64 soon.
I'm paying attention to some tmpfs work being done in upstream
FreeBSD, the source of the package build kernel panics mentioned
previously.

Please pay attention to this thread. I'll use this thread to keep
everyone updated.

Thank you, everyone, for your patience and understanding.

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 443-546-8752
Tor+XMPP+OTR: lat...@is.a.hacker.sx
GPG Key ID: 0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2

[Shawn Webb]

I arrived at the Maryland datacenter early this morning to deploy the
new hard drives and start the restoration test procedure.

We're now a day ahead of schedule. :)

Thanks,

[Shawn Webb]

The restoration steps proved successful. I reinstalled HardenedBSD
12-STABLE/amd64 on the server. I've handed the server over to Oliver
for him to perform further configuration tasks.

[Uwe Trenkner]

Thank you, Shawn and Oliver! Your work is highly appreciated!

Good luck and best regards
Uwe

[Shawn Webb]

I have now started on the new build scripts. The previous scripts were
showing signs of major bit rot. The new scripts serve to simplify and
unify work going forward.

Ideally, I would like to be able to add these new tools I'm building
to the ports tree. Meaning, one could reproduce our build
infrastructure by using `pkg install`. These new build scripts will
reflect this desire.

I'm hoping to deploy beta versions of the build scripts by the end of
this week, enabling the build of 13-CURRENT/amd64 and
13-CURRENT/arm64.

Moving on to other subjects. There is currently a merge conflict in
the hardened/12-stable/master branch in the hardenedBSD.git repo. The
merge conflict is due to my reversion of FreeBSD's ASR MFC to
12-STABLE. I plan to resolve the merge conflict this week, bringing
FreeBSD's ASR into HardenedBSD 12-STABLE in similar fashion as was
done for 13-CURRENT.

As on 13-CURRENT, HardenedBSD users will be able to choose at runtime
which implementation to use: FreeBSD's ASR or HardenedBSD's ASLR. The
default will be HardenedBSD's ASLR.

Once the merge conflict is resolved, I will enable nightly builds of
12-STABLE/amd64 from the hardenedBSD.git repo. Oliver will likely want
to wait at least a week before cutting a new stable release from the
hardenedBSD-stable.git repo. For more information on the repos we use,
please visit our wiki[1].

If all goes as planned, we should be back up and running within the
next two weeks.

I appreciate the patience as we take this opportunity to ensure
smoother operation of our infrastracture going forward. We are putting
in place a foundation which will enable us to scale our
infrastructure with efficient ease.

[1]: https://github.com/HardenedBSD/hardenedBSD/wiki

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

[Uwe Trenkner]

Thank you for all the work and for sharing the latest status.

As you do not mention 11-STABLE at all: Can we assume that this branch
has been dropped from HardenedBSD?

Best regards

Uwe

[heda...@gmail.com]

Hi Shawn,

It's been three weeks now since your last status report, and it seems the build server is still offline. How are are things going ?

Is there anything we can do to help you and Oliver to get back on tracks ? I read money was not (yet) a problem; is it still true ?

Thanks,

Marin.

[Shawn Webb]

Hey Marin,

I'm working on the infrastructure as I have time.

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 443-546-8752

Tor+XMPP+OTR: lat...@is.a.hacker.sx
GPG Key ID: 0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2

On Sun, Oct 06, 2019 at 04:37:08AM -0700, heda...@gmail.com wrote:
> Hi Shawn,
>
> It's been three weeks now since your last status report, and it seems the
> build server is still offline. How are are things going ?
> Is there anything we can do to help you and Oliver to get back on tracks ?
> I read money was not (yet) a problem; is it still true ?
>
> Thanks,
>
> Marin.
>

> Le lundi 16 septembre 2019 07:34:31 UTC+2, Shawn Webb a ??crit :

[Shawn Webb]

I've made public the new build scripts:
https://github.com/lattera/hbsd_build

Note that these scripts are under heavy development. The first thing
to do is to config-ify the build. Writing the config functions
shouldn't take too long.

I'm not soliciting feedback or patches for the scripts right now.
However, I wanted to make the repo public so our community can track
the progress.

Once the scripts are in good enough shape to use (early this week, I'm
hoping), I will transfer this repo to the official HardenedBSD
account. It's at this point that I will accept contributions to this
particular codebase.

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 443-546-8752
Tor+XMPP+OTR: lat...@is.a.hacker.sx
GPG Key ID: 0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2

[Shawn Webb]

On Wed, Aug 28, 2019 at 08:38:19PM -0400, Shawn Webb wrote:

Hey all,

I wanted to update everyone on the status of our infrastructure. I
have great news to share!

The new build scripts have progressed along nicely. The one task left
to do is make a couple configuration variables dynamic and I can
deploy the scripts to production.

Tomorrow, Sunday, 20 Oct, 2019, I plan to head over to the Maryland
datacenter and deploy three Dell R420 servers. These servers will act
as the new build servers, replacing the build functionality of the
previous single build server, kindly leased to us for free by New York
Internet (NYI).

We will continue to utilize the NYI server. It will serve solely as
the primary mirror and rsync master. The three Dell R420 servers will
perform regular builds of supported branches, publishing the builds to
the NYI server.

The build scripts are structured such that they can be made into a
port. I envision being able to scale our infrastructure out by
performing `pkg install hbsd-build-scripts` and installing build
config(s).

Thus, if one build server experiences hardware failures, another build
server can be stood up easily and efficiently.

I plan to run the build scripts by hand for around a week to ensure
proper function. Once I'm satisfied with the stability, I will
automate the build via cron. The build scripts were meant to be run
via cron, anyways.

Tomorrow, HardenedBSD should begin publishing new builds of
13-CURRENT/amd64. Please note that while I work on the build scripts,
the directory structure may change. Builds may appear and disappear.
Until further notice is given, these builds should be considered
experimental.

I will update everyone once the web server is up and the first build
is published.

Thank you very much for your patience and understanding.

[Carlos Lopez]

Really good news ... Many thanks for your impressive work Shawn

--
Regards,
C. L. Martinez

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@hardenedbsd.org.

[Shawn Webb]

Hey all,

We now have two build servers chugging along nicely. Builds of
11-stable/amd64 and 13-current/amd64 have been published. Again, since
I'm actively and heavily developing these scripts, I'm reserving the
right to change the directory structure. I'll let everyone know once
things have settled down and are cemented in place.

But, if you want to get your hands dirty and move some electrons, you
can find the builds here: http://ci-01.nyi.hardenedbsd.org/pub/

Thanks,

[Shawn Webb]

Hey all,

Our two build servers, ci-01.md.hardenedbsd.org and
ci-02.md.hardenedbsd.org, are running fine and dandy. Below is a
bulleted list of what each server currently builds and when.

Our amd64 package build server is currently working on a
13-current/amd64 package build.

While my puppy Vader sleeps, I'm working on resolving a build issue
with HardenedBSD 12-STABLE/amd64. With any luck, I will have the issue
resolved today. Once resolved, I will kick off a new official build.

I've decided to keep the current directory structure on our master
mirror (ci-01.nyi.hardenedbsd.org).

Builds from the hardenedbsd.git repo will be published in
/pub/hardenedbsd. Builds from hardenedbsd-stable.git will be published
in /pub/hardenedbsd-stable.

Bulleted list of our build infrastructure:

* ci-01.md.hardenedbsd.org: Runs HardenedBSD 13-CURRENT/amd64. Builds
HardenedBSD 13-CURRENT/amd64 and HardenedBSD 13-CURRENT/arm64.
* ci-02.md.hardenedbsd.org: Runs HardenedBSD 12-STABLE/amd64. Builds
HardenedBSD 12-STABLE/amd64 and HardenedBSD 11-STABLE/amd64.
* ci-01.nyi.hardenedbsd.org: "Demoted" from build server to master
mirror. Receives builds and redistributes to secondary mirrors.