HEADS UP: GitLab down

[Shawn Webb]

Hey all,

Our oligarchic overlords with their legion of AI bots have decided
that our GitLab isn't powerful enough to line their pockets with our
code.

I've now powered off our self-hosted GitLab and disabled the autosync.
I'll pull an all-nighter tonight to see if we can switch to Radicle.
If that fails, I'm not sure what to do. It's evident that we need a
whole new fleet of servers just to handle the load and ain't nobody
got funds for that.

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Signal Username: shawn_webb.74
Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

[Shawn Webb]

On Mon, Apr 20, 2026 at 03:01:05PM +0000, Shawn Webb wrote:
> Hey all,
>
> Our oligarchic overlords with their legion of AI bots have decided
> that our GitLab isn't powerful enough to line their pockets with our
> code.
>
> I've now powered off our self-hosted GitLab and disabled the autosync.
> I'll pull an all-nighter tonight to see if we can switch to Radicle.
> If that fails, I'm not sure what to do. It's evident that we need a
> whole new fleet of servers just to handle the load and ain't nobody
> got funds for that.

GitLab is back online, but in read-only mode. I will look at switching
to Radicle tonight. If Radicle still is not an option, I'm at a loss
for what to do next. I will not return to GitHub as our
source-of-truth.

[Shawn Webb]

On Mon, Apr 20, 2026 at 03:42:45PM +0000, Shawn Webb wrote:
> On Mon, Apr 20, 2026 at 03:01:05PM +0000, Shawn Webb wrote:
> > Hey all,
> >
> > Our oligarchic overlords with their legion of AI bots have decided
> > that our GitLab isn't powerful enough to line their pockets with our
> > code.
> >
> > I've now powered off our self-hosted GitLab and disabled the autosync.
> > I'll pull an all-nighter tonight to see if we can switch to Radicle.
> > If that fails, I'm not sure what to do. It's evident that we need a
> > whole new fleet of servers just to handle the load and ain't nobody
> > got funds for that.
>
> GitLab is back online, but in read-only mode. I will look at switching
> to Radicle tonight. If Radicle still is not an option, I'm at a loss
> for what to do next. I will not return to GitHub as our
> source-of-truth.

The server is offline again due to new hardware failures. I'm running
Dell's integrated Hardware Diagnostics. It's already reporting bad
memory. Once this diagnostic run is completed, I will boot into
memtest86 to attempt to find which exact stick of RAM is dead.

This server hosts the following VMs:

1. GitLab
2. Vaultwarden
3. DNS
4. Radicle

After all the diagnostics are completed, including memtest86, I will
boot the server back up and attempt to migrate VMs 2-4 to a different
server. All our other servers are resource-constrained, so this might
not be possible.

Thanks,

[Shawn Webb]

I forgot that our main nginx vm is on the same dead server. Our
package repos and the rest of the dev/build infrastructure are
currently unavailable.

I'm sorry for the inconvenience and hassle.

[Shawn Webb]

Our nginx VM is back up! So our installers and our package repos are
available once again (and any non-GitLab web service).

I'm currently backing up our Radicle VM, which will likely take all
night. After that completes, I'll move that VM to another host.

GitLab will not come back online for at least a few days. Our src and
ports repos are now officially available on Radicle. The RIDs are
posted below.

1. RID for src: rad:z2HLHXgL1xevBNQsf8BmQW7MpJmtm
2. RID for ports: rad:z2XrdvALg77ycnuZRXgScb27yb3wM

Once the mostly-dead server is fully backed up, I will bring GitLab
back online. It will not be online permanently. It'll be online only
to help us migrate to Radicle fully.

From an infrastructure perspective, once the VMs are backed up, I will
work to find out which exact piece of hardware is failing on the
server. There's at least one dead stick of RAM and IPMI reported a
failure on the PCI bus.

From a development perspective, my next goal will be to implement
Radicle support in the ports tree. Think: USE_GITLAB, but USE_RADICLE.
This will be required for downloading port distfiles. Unfortunately,
building ports-mgmt/pkg currently requires our GitLab to be online.

I will keep everyone informed. Thank you, everyone, for your patience
and understanding.

I would like to end this status update with a sincere thank you to the
Radicle team and community. They came to our rescue in our dire moment
of need, even at the cost of time, bandwidth, and storage. Also
OpenBSD's qbit (Aaron Bieber) helped perform initial testing, and I'm
grateful for his help, too.

Thanks,

[Shawn Webb]

Things have been progressing little-by-little. Our initial test seemed
to work okay. Now that we're trying to deploy Radicle to a dedicated
seed node, we're experiencing issues. It seems hosting HardenedBSD's
src and ports trees is indeed proving to be a challenge for Radicle.

That said, we're still moving along with the test. I suspect there
will need to be some bug fixes, performance improvements, and
reliability improvements to Radicle. We're almost there!