Bugzilla HA How to?

[Laci D]

Does anyone have experience setting up Bugzilla in a highly available architecture?
Basically to have two servers behind one hostname, either active-active or active-passive setup.

[Thorsten Schöning]

Guten Tag Laci D,
am Mittwoch, 10. März 2021 um 21:23 schrieben Sie:

> Does anyone have experience setting up Bugzilla in a highly available architecture?
> Basically to have two servers behind one hostname, either
> active-active or active-passive setup.

I can't remember a such specific discussion on this list yet.

Because there are so many different possibilities to implement HA, it
might be a better idea if you start with telling people about what
technologies you have experience with, how automated you NEED things
to be, which downtime you really CAN NOT accept for which reason etc.

E.g., in the easiest case Bugzilla supports using a shadow database
already, which makes at least that part some kind of fault tolerant
for at least reads.

https://bugzilla.readthedocs.io/en/5.0/administering/parameters.html#shadow-database

OTOH, Bugzilla stores things in the file system as well, so a shadow
database wouldn't cover everything anyway. Though, depending on your
needs, file systems can be pretty efficiently mirrored with somewhat
low admin maintenance using simply RSYNC and a cronjob or by sending
snapshots for file systems like BTRFS and ZFS using a cron job. I'm
successfully using ZREP with Ubuntu 16.04 for some years in some kind
of active-passive setup.

http://www.bolthole.com/solaris/zrep/

So you don't necessarily need clusters like Proxmox, shared block
devices like with DRBD or alike. It heavily depends on your needs.

https://pve.proxmox.com/wiki/Main_Page
https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device

Mit freundlichen Grüßen

Thorsten Schöning

--
AM-SoFT IT-Service - Bitstore Hameln GmbH i.G.
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister für IT und TK

E-Mail: Thorsten....@AM-SoFT.de
Web: http://www.AM-SoFT.de/

Tel: 05151- 9468- 0
Tel: 05151- 9468-55
Fax: 05151- 9468-88
Mobil: 0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH i.G., Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB neu - Geschäftsführer: Janine Galonska


Für Rückfragen stehe ich Ihnen sehr gerne zur Verfügung.

Mit freundlichen Grüßen

Thorsten Schöning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil:
Webseite: https://www.am-soft.de

AM-Soft IT-Service - Bitstore Hameln GmbH i.G. ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister für IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

Büro Dallgow-Döberitz
Tel: 03322 507 020

Büro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service - Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

Büro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491 876 735 7

Büro Liebenwalde
Tel: 033054 810 00

HSA - das Büro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT – Consulting GmbH
NL Piesteritz
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore Gütersloh GmbH
Gütersloh
Tel: 05245 920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Geschäftsführer Janine Galonska

[Laci D]

I was thinking to do something like:
- point bugzilla.hostame.com to bugzilla-01.hostame.com
- stand up a bugzilla-02.hostame.com server which syncs /usr/local/www/bugzilla/data/ from 01 to 02 (either with rsync or with zfs snapshots, the servers are/will be on FreeBSD)

During a maintenance window we'd be doing:
- set bugzilla-01.hostame.com read only (if bugzilla has such an option, if not change DB connection to a read only replica)
- sync /usr/local/www/bugzilla/data/ from 01 to 02
- point bugzilla.hostame.com to bugzilla-02.hostame.com
- do required maintenance on bugzilla-01.hostame.com (patching, migrating, upgrading, etc)
- set bugzilla-02.hostame.com read only (if bugzilla has such an option, if not change DB connection to a read only replica)
- sync /usr/local/www/bugzilla/data/ from 02 to 01
- set bugzilla-01.hostame.com read/write
- point bugzilla.hostame.com to bugzilla-02.hostame.com

What do you think, does this look feasible?

Viel dank!
Laci

[Thorsten Schöning]

Guten Tag Laci D,
am Donnerstag, 11. März 2021 um 15:41 schrieben Sie:

> - stand up a bugzilla-02.hostame.com server which syncs

> /usr/local/www/bugzilla/data/ from 01 to 02[...]

That directory is an implementation detail of Bugzilla and might
change at any time. So syncing that only into another arbitrary older
or even newer version of Bugzilla is a risk in theory. Taking all or
nothing should be the better approach.

> - set bugzilla-01.hostame.com read only (if bugzilla has such an
> option, if not change DB connection to a read only replica)

Bugzilla can be put into maintenance mode, but NOT into read-only.
Read-only is used automatically for such operations IF shadow
databases are configured.

> shutdownhtml
https://bugzilla.readthedocs.io/en/5.0/administering/parameters.html#general

> What do you think, does this look feasible?

First of all I still don't see against what your HA-setup should
actually protect and your should REALLY reconsider if such a
complexity is necessary to start with at all. Keep in mind that things
heavily depend on the number of outtakes you have e.g. because of
updates and those weren't too many in the near past at all.

Said that, your setup won't work at all: You forgot the database.
Upgrading Bugzilla sometimes means new database schema and when you
only update/test one instance, the other is per definition NOT able to
use the same schema anymore/yet. OTOH, if you have multiple different
databases for Bugzilla, you have actual data in different databases
and need to merge that at some point. That is neither trivial nor HA.

You won't change that problem with database clusters or stuff as well:
As long as you are running two different versions of Bugzilla, you can
not assume to use the same database for both.

So, as said: What are you actually trying to protect against? Better
start with simple backups to be able to revert in case of problems.

[Laci D]

"So, as said: What are you actually trying to protect against? Better start with simple backups to be able to revert in case of problems."

It is more like a planned failover, current server upgrade takes about two hours (OS upgrade, reboots, etc.) while Bugzilla is unavailable, I'd like to cut this planned downtime as short as possible. This is the main goal.

[Thorsten Schöning]

Guten Tag Laci D,
am Donnerstag, 11. März 2021 um 17:29 schrieben Sie:

> It is more like a planned failover, current server upgrade takes
> about two hours (OS upgrade, reboots, etc.) while Bugzilla is
> unavailable, I'd like to cut this planned downtime as short as
> possible. This is the main goal.

How do you do it for other hosted services? If you don't, don't care
for Bugzilla too much as well at least for the beginning. HA is REALLY
complex and nothing to start with only based on advices on this list.
Otherwise pretty much do what you are already doing with other
services, Bugzilla is only one more.

There simply is no golden bullet for you in my opinion: A cluster with
shared storage for Bugzilla and its database will allow you most
likely to simply take one of the hosts offline for maintenance.
Bugzilla would be served by the other nodes and all would be fine.

OTOH, those clusters shouldn't be underestimated, even with tools like
Proxmox making things pretty easy in THEORY. Additionally, when it's
about upgrading Bugzilla, the shared storage/database means that you
need to take all Bugzilla instances offline anyway, upgrade all of
them and back online again only after everything succeeded.

I don't think I can provide any more advices on this. :-) If at all,
you might ask on HA-specific mailing lists/forums, because the
problems to solve for Bugzilla are not too Bugzilla-specific: Some
files and a database. Sounds VERY common with web apps... ;-)

[Laci D]

Thank you for your advice!

[S Bob]

1) Use PostgreSQL as the db engine

2) Setup a Hot Standby database node via PostgreSQL built in streaming
replication

3) install Bugzilla on both nodes

4) use a DNS failover method (or something similar) to perform the
following for a primary node failure:

a) Promote the PostgreSQL Hot Standby
b) Update the DNS / Proxy / URL so now connections are directed to the
secondary site
c) send an alert so someone can investigate and repair the HA cluster

> _______________________________________________
> support-bugzilla mailing list
> support-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/support-bugzilla
> PLEASE put support-...@lists.mozilla.org in the To: field when you reply.

[Thorsten Schöning]

Guten Tag S Bob,
am Donnerstag, 11. März 2021 um 21:54 schrieben Sie:

> a) Promote the PostgreSQL Hot Standby
> b) Update the DNS / Proxy / URL so now connections are directed to the secondary site
> c) send an alert so someone can investigate and repair the HA cluster

A Hot Standby for Postgres doesn't provide data Bugzilla stores in
the local file system. So if you need shared/clustered/... storage for
that anyway, you can simply use that for Postgres as well. DRBD e.g.
simply provides block storage usable for those purposes.

OTOH, how do you deal with upgrading all of your Bugzilla-instances
in a schema-incompatible way? If you take down the master, stop the
replication and host users from the Hot Standby-Bugzilla, you need to
make that read-only or newly added data in that schema needs to be
merged into the master schema again.

How does one put Bugzilla into read-only mode? I'm only aware of
"shutdownhtml". Would a read-only Bugzilla still be HA anyway? :-)