[Notification] Planned Maintenance - mozillakerala.org

[shine at mozilla]

Hello,

This is to inform that there will be a planned maintenance of the
server that hosts mozillakerala.org over the weekend ( April 25-26,
2020 ).

The main intention is to attempt an operating system upgrade from
Debian 8 ( Jessie ) to Debian 10 ( Buster ). The server hasn't had any
updates for a very long time.

TL; DR
-----
1. Planned maintenance activity on mozillakerala.org server over the weekend
2. Upgrade operating system to Debian Buster
3. Database engine migration to mariaDB
4. Downtime of 10 - 30 minutes
5. Other follow-up activities if time permits
===============

Long version
---------------

I know ( and acknowledge ) that the server has not seen any love at
all, both in terms of updates or usage, in the past years. I wanted to
change that, at least in terms of updates, with this activity.

The plan is to follow the normal upgrade path. Since the server is
running Debian, it would be stable enough to even skip the version in
between altogether and jump straight to the latest version, but I
don't want to take the risk.
The action plan is as follows :

1. Stop web server ( apache2, php5-fpm )
2. Stop database server ( mysql )
3. Take a database dump ( just to be safe )
4. Update sources.list to point to stretch
5. apt update; apt dist-upgrade
5.1. if all goes well, attempt start of apache2 without php
5.1.1. verify static landing page ( mozillakerala.org ) is active
6. if that goes well, repeat steps 1, 4 and 5 with repo pointing to buster
9. once we are on buster, then start php-fpm and mysql

That would conclude the main activity. If all of the above steps go
well ( which generally doesn't happen that easy in the linux server
world ), then I have another bunch of activities to do in my mind :

1. migrate database engine to mariadb
2. migrate web server to nginx
3. clean up existing domains list
3.1 fix unknown ServerName redirections
4. set up Transport Layer Security ( TLS )
5. upgrade Piwik to Matomo
6. set up some basic server hardening

Downtime time window
--------------------
If the operating system upgrade goes well, there should only be an
intermittent downtime of around 5 - 10 minutes for the main website (
mozillakerala.org ) considering there is good internet network speed
on the server. The blog, however, would be down for further if the
migration to the mariadb database engine is also taken up. While mySQL
and mariaDB are data-compatible, there can be unknown factors that
could crash which could extend the downtime of the database engine
which in turn could extend the downtime of the blog. The other
follow-up activities can be done with the server running because they
do not need to keep the server down for operation.

If you have any objections about this activity, you have time until 9
AM on Saturday, April 25, 2020 to voice your opinions after which the
activity will assume to be unopposed and in motion.
If you have any other opinion about the action plan or the follow-up
activities; or if you'd like to add to the list of activities, please
reply to this email and we can discuss the feasibility of including it
in the action items.

I'd also like to point out that the above plan ( especially the follow
up action items ) is subject to change considering the possibility of
breakages or breakdowns during the main activity. The main target of
the activity would be to complete the operating system upgrade to get
to the latest versions of packages and system patches. The follow up
action items would be taken up only if the main activity goes through
successfully and there is enough time to tackle the rest of the follow
up action items.

~ shine

[Akshay S Dinesh]

I don't know if it counts as objection but I doubt your effort is worth it.
Especially the way mozilla wants to centralize groups on to its community
portal.

On Thu, Apr 23, 2020 at 9:51 PM shine at mozilla <shine....@gmail.com>
wrote:

> _______________________________________________
> community-kerala mailing list
> communit...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/community-kerala
>

[shine at mozilla]

Realizing that is precisely what prompted me to revive the server back
up. We've always been rebellious. So, why not some more fun?

So I guess, what I'm trying to say is - until mozilla decides to pull the
plug on the server itself, we should maintain as much of it as we can.


~ shine

[Gautam krishna.R]

Can we enable livepatch as well: https://ubuntu.com/livepatch so that it
will do critical updates automatically?

Thanks and Regards,

Gautam Krishna R

Software Engineer

Red Hat India (APAC) <https://www.redhat.com/>

Tower X, Magarpatta City, Pune

gau...@redhat.com
M: +91 9446335048 <+919446335048> | GitHub: git.io/gkr
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://www.redhat.com/>



On Fri, Apr 24, 2020 at 3:42 AM shine at mozilla <shine....@gmail.com>

[shine at mozilla]

I believe Livepatch is an Ubuntu service, but we run Debian.

Livepatch is a snap app and I definitely don't want to install snaps
on to the server. Let's just say I have a very strong prejudice
towards snaps.

~ shine

~ shine

On Fri, Apr 24, 2020 at 4:41 PM Gautam krishna.R
<r.gauta...@gmail.com> wrote:
>
> Can we enable livepatch as well: https://ubuntu.com/livepatch so that it will do critical updates automatically?
>
> Thanks and Regards,
>
> Gautam Krishna R
>
> Software Engineer
>
> Red Hat India (APAC)
>

> Tower X, Magarpatta City, Pune
>
> gau...@redhat.com

> M: +91 9446335048 | GitHub: git.io/gkr
>
> @RedHat Red Hat Red Hat

[shine at mozilla]

Hello,

The planned activity will be starting soon. While I'd like to give
commentary-style updates, I think that'd be spam to the rest of the
people in the list.

Hence, I'm not going to be sending minor updates to the
community-kerala list; but I'll send it to people already on the cc
list. If you'd like to get updates on this activity, please reply to
this email with 'Subscribe' in the email body.

Once you're on the list, and you think the commentary are too
'spammy', you can also send 'Unsubscribe' to unsubscribe from the
updates as well.

So, here goes. See you on the other side.

~ shine

On Fri, Apr 24, 2020 at 5:36 PM shine at mozilla

[shine at mozilla]

Hello,

This is to inform that the activity has been stalled due to an
outdated kernel version ( 2.6.32 ) that is used to boot the server
with. A ServerFault [1] question indicates that it is possible that
the server provider is using NetBoot to boot the machine ( because
there is no kernel in the filesystem ). This limits the option to
upgrade the kernel to the latest release. I've reached out to tad who
used to be the point of contact for the server to see if this issue
can be resolved in some way.

However, the good news is, I was able to upgrade the operating system
( with warnings about the outdated kernel ) to Debian Stretch. It's
not the greatest, but that's the closest I could get the system to.
The other good news is that Debian automatically upgraded mySQL to
mariaDB during the OS upgrade from Debian Jessie to Debian Stretch.
So, that's one follow-up task checked off of the list. I'll probably
pick some of the follow up tasks if I can until we get a response to
the kernel issue.

All existing services are back online in their previous good form.
I'll resume this activity once we have an update about the kernel
upgrade from the server provider.

Links
-----
[1] https://serverfault.com/questions/586171/cant-update-linux-kernel-on-vps

~ shine

On Sat, Apr 25, 2020 at 3:03 PM shine at mozilla