[Proposal] Point wiki.heatsynclabs.org at heatsynclabs.wiki

[Robert Bushman]

Proposal for June 13, 2024:

Q: Why do you think you're better at operations than Jeff?

A: I don't. Jeff has operations skills that I do not. I voted for Jeff
for operations both of the past two years, including the year that I had
been running against him. I can go into detail, but the short answer is
I can't do what Jeff does. I'm a fan, and I intend to vote for him again
this year, if he runs.

Q: OK, so why do you think you can run a wiki server?

A: MediaWiki happens to be in my specialized wheelhouse. I've been
adminning production LAMP stacks for 28 years. My largest fleet of
unmanaged hosts was 120 boxes at Apple. My largest fleet including
managed hosts and serverless was at Amazon where I was tech lead then
tech manager on a team with $6.5m of annual AWS spend. I mostly do data
pipelines and ML; but for 23 years I've always had at least one wiki
running, usually a few.

My skill at ops overall doesn't hold a candle to Jeff. I wouldn't even
say I'm necessarily better at MediaWiki. But I am quite good at it, and
I had the free time to get it live.

What I'm looking for:

1. Recent version of MW, regularly updated.
2. Nightly tarball of the recovery content, publicly accessible. (this
is a "run over by a bus" solution for non-hierarchical orgs - anyone can
pull nightly and clone the server at their pleasure)
3. IPv6

I have that live now, and I'm happy to host as long as desired. But I'm
not emotionally attached to running it. As soon as the above is live
elsewhere, I'll be happy to drop mine.

I'll put passwords to the server and database in the KeePass. The
install and recovery process is bone stock MW, their docs are better
than mine, but I'll make my notes available on both the HSL wiki and my
main wiki. If someone wants me to replicate it on another host, all I
need is root on a stock Deb 12 box with a provider that supports SMTP
and IPv6.

[Eric Ose]

A. Does this mean that links we had to wiki.heatsynclabs.org would all need to be forwarded to the new domain?

I'd kind of prefer not to have url change shenanigans.

B. Does this proposal as written put you in charge of the wiki?

I'm opposed to that because we have had too many problems in the past when new Operations Managers couldn't properly do all of the IT things related to HeatSync Labs things.

Eric Ose

Robot Ambassador

Sometimes cool things just happen, but usually you have to plan them.

--
You received this message because you are subscribed to the Google Groups "HeatSync Labs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to heatsynclabs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/heatsynclabs/a88ced94-6ff9-4343-bd57-4645be72f133%40traxel.com.

[Robert Bushman]

A: We will not have to update any links. It's just an update to the DNS.

B: I will not be in charge of the wiki. Admin privs would be available
for whoever wants or needs them, and the passwords will be in KeePass.


Oh - and - meanwhile: For existing updates, please continue updating the
current wiki at wiki.heatsynclabs.org. I am merging updates. If there is
anything you want merged faster than I notice it, please ping me.

On 5/10/24 10:47, Eric Ose wrote:
> A. Does this mean that links we had to wiki.heatsynclabs.org

> <http://wiki.heatsynclabs.org> would all need to be forwarded to the new

> <mailto:heatsynclabs%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/heatsynclabs/a88ced94-6ff9-4343-bd57-4645be72f133%40traxel.com <https://groups.google.com/d/msgid/heatsynclabs/a88ced94-6ff9-4343-bd57-4645be72f133%40traxel.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "HeatSync Labs" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to heatsynclabs...@googlegroups.com

> <mailto:heatsynclabs...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/heatsynclabs/CAEk_gvBv3eeAHZNNAmmQwN48c2uJh-9fU2%3DAX3kR_YjALagOmw%40mail.gmail.com <https://groups.google.com/d/msgid/heatsynclabs/CAEk_gvBv3eeAHZNNAmmQwN48c2uJh-9fU2%3DAX3kR_YjALagOmw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Jeff Sittler]

TL;DR version:

My main concerns are:
1. This would be moving out of the control of the lab by going to a personal cloud account
2. The increased cost of maintaining the hosting
3. Was handled without proper governance by the OPs team.  Afterall, we are the ones who maintain the infrastructure at the lab.

In my opinion, this proposal should be pulled back and let’s discuss this further.  Perhaps it is the same path that Bob is proposing, 

perhaps not, but I would like to see us come up with a more solid plan to move forward that does not cost the lab any additional 

financial burden at this time.

Unfortunately, I will be out of town for the May 23rd HYH.  I would like see this postponed until the following HYH (June 13th)  so 

I can be in attendance and available to discuss/answer any questions people may have.

Long version…

I would like to re-state the following top concerns/observations I have and the “guidelines” I follow when implementing solutions at 

the lab that relate to Operations.

1. From an Operations perspective, we need to make sure we are implementing some form of governance around how we host software and implement solutions for the lab.
2. We have had issues in the past that have caused problems with hosted solutions, so we need to make sure that we do our due diligence to prevent those scenarios from happening in the future.
3. We need to be cognizant about how HSL spends its finances and take advantage of using what resources we have available, or are provided, to us without additional financial hardship.


Now, as to addressing Bob’s responses:

> Q: Why do you think you're better at operations than Jeff?

A: What does this have to do with the Wiki?  Why are singling me out as the bad guy for having an opinion and voicing concerns that I 

have about it being hosted under a personal account and the cost of operating it?

> A: MediaWiki happens to be in my specialized wheelhouse. I've been
> adminning production LAMP stacks for 28 years. My largest fleet of
> unmanaged hosts was 120 boxes at Apple. My largest fleet including
> managed hosts and serverless was at Amazon where I was tech lead then
> tech manager on a team with $6.5m of annual AWS spend. I mostly do data
> pipelines and ML; but for 23 years I've always had at least one wiki
> running, usually a few.

As I mentioned above, we need to make sure we are implementing solutions that are properly governed by the lab.  Based on Bob’s answer, 

it sounds like he has experience in the software field and should understand the need for governance with situations like this compared to 

just going out and doing things on one’s own under the “do-ocracy” banner.  While I am a fan of going and doing things to make progress, 

doing so in a path that has potential detriment to HSL is not the correct way to do things.

> 1. Recent version of MW, regularly updated.

This can be handled by us, and in fact has already been setup to handle upgrades as needed on the new platform

> 2. Nightly tarball of the recovery content, publicly accessible. (this
> is a "run over by a bus" solution for non-hierarchical orgs - anyone can
> pull nightly and clone the server at their pleasure)

Again, the Database is already performing nightly backups.


> 3. IPv6

Why do we need IPv6?  What benefit is there from it?  While it is in use, it has not been widely adopted in the Internet community 

and just adds additional complexity.

Now, I would like to address some other comments from the Wiki Upgrade thread.

> We rely upon outside services. We have dependencies on GitHub, Google
> Groups, Google Calendar, Slack, DropBox, PayPal and more. Most of those
> have lock-in problems.

Yes, we do rely on well established cloud solutions that are experts in their field and do not cost HSL anything, or a very small fee for 

something we can not do ourselves (PayPal).  Google, GitHub, Slack, Dropbox, etc provide free services to us.  I know I do not want to 

be on the hook for handling credit card payments, I’d rather leave that to the banks/companies that specialize in that.

I am not against using services in the cloud when they benefit us and do not cause extra drain on our finances and are under HSL’s 

control.  We have had issues in the past where a cloud service is under someone’s personal account (with good intentions) and 

then something happens and we lose access to whatever they were doing and things break.  I would like to prevent that from happening 

again.

> This does not have lock-in. This is hosting, only. The server is vanilla
> Deb 12, and the backups are published nightly, ready to be restored by
> anyone who can do the work. Lift and shift is a snap.

While this might not be “vendor lock-in” in a sense, it being hosted on AWS, what is the process for moving it to another provider if 

we so choose? Could a non-technical person do it?

> The in-house network is flawed. We do not have the resilience of a
> hosted solution. When I tried to work with you on the upgrade, you said
> we don't have remote access. We don't have IPv6. It's fun to run our own
> machines, and they should be used for fun. They are not production-grade.
>
> They should not be used for mission critical infrastructure.

I disagree with the above statements, and in fact I feel it is a slap in the face of the team(s) that have kept things running for 10+ years (I know 

it is longer, I’m rounding down).  It has taken them a lot of sweat and time to keep the lab up and running and to make sure we are able to do 

what we can.  Now, is the infrastructure outdated and need some re-architecting?  Yes, but that does not mean we do not have the talent to 

do it, whether it is on-prem or in the cloud, this is something that should be discussed by the OPs team for a path forward that benefits everyone 

while making things easier to maintain.  As for the remote access, it is locked down for security sake.  Proper protocol is to not allow remote 

access directly to servers.

> I started asking for a Wiki upgrade 3.5 years ago. I've documented the
> process multiple times. I started discussing it with you 16 months ago.
> HSL has had the chance to do this, and has not.

I agree that I have dropped the ball on getting the Wiki upgraded in a timely manner and that is on me.  More discussion below.


> This past September, you were amenable to my approach. When I arrived,
> you said I could not have access to do the work. Then you said you
> didn't want to do it they way I do it. Then you asked me to walk you
> through it the way you want to do it.

1. You were wanting remote access to the server.  We do not just give anyone remote access to the servers.
2. You were wanting access to the ROOT user.  We do not just give anyone ROOT access to the servers.  That is a huge security risk.

As for “doing it my way”, perhaps there was some kind of disconnect, but I am not the Wiki expert and I had no “way” to do it.  I was 

deferring to your expertise.  As I told you, I had the Wiki upgraded and ready to migrate the data over and wanted your help with the 

best path to do that.  Unfortunately we seemed to be at a disagreement and did not get the data migrated.

> HSL owns ~[wiki.heatsynclabs.org](http://wiki.heatsynclabs.org/)~, which should be pointed at the new host,
> ASAP. heatsynclabs.wiki is a placeholder. If we can find a good path
> forward, I'm happy to transfer it.

Having a “new domain” again introduces another cost of at least $10/year that the lab had not been made aware of, along with the 

approximately $7/mo expense of the hosting the Wiki.  While they are small, all these small expenses add up. Again, while I appreciate 

you initially covering these small expenses, by your own admission of “I'm happy to pay for it for the foreseeable future” how long is that 

future?  What happens when that is no longer the case?  HSL will be on the hook for yet another nickel and dime expense that “we” did 

not sign up for.  Also, who is going to maintain it if/when you lose interest in maintaining it, or it becomes a bigger job than you want?

> The obvious reason that the wiki needed to be upgraded is that the current wiki is jacked, and has been as long as I've been here.

Aside from being outdated, how is the wiki “jacked”?  From my understanding, several years ago it was hacked and spam was added 

to the wiki, but it was my understanding that it has been remedied since then.

> nightly backups and a disaster recovery plan that gets tested regularly. I want to invest time and
> energy into making pages on the wiki, but I'm not comfortable doing that without a solid recovery plan.
>
> Second would be regular upgrades for security. Bugfixes for user-facing
> issues are nice; the security ones are important.

I agree with this as well and is something that should be handled accordingly.


> Bob is creating a new server because just installing updates on the old one is not feasible.

Actually it is very feasible and steps have already been preformed, as mentioned in my response above.  The last steps are:
1. migrate the data to the updated version of Mediawiki
2. point wiki.heatsynclabs.org from the old server to the new server (that again, is already up and running, just needs #1 above done)


> That is a legitimate concern. I would like the end result of this
> process to be a single wiki, administered by HSL.

This would still be the case no matter where it is hosted, in the cloud or on-prem.


> A hosted server gets us some resilience and network features that would
> be difficult or costly to replicate internally.

We already have the infrastructure in place for this and have been doing it for 10+ years.


> HSL should own and control the outside wiki as soon as it is able. I
> will be publishing a disaster recovery doc which will work as a
> migration doc. Deploying a replacement server is essentially the same as
> recovering from an HDD failure. As soon as the HSL server meets our
> needs, I'll kill the one hosted on my account.

Since the server hosted at the lab meets our needs, I would rather see an effort in getting this “ask” over the hill and coordinating getting 

the data migrated than to spend time working on a cloud instance and then moving it back to the lab.

> I agree with running it by HYH. That has been and remains my intent.
Was this discussed at a HYH?  I did not see it on any HYH agenda for discussion.  Perhaps I missed it.


My apologies for the long response, I wanted to address some of the more pressing comments I saw.

My main concerns are:
1. This would be moving out of the control of the lab by going to a personal cloud account
2. The increased cost of maintaining the hosting
3. Was handled without proper governance by the OPs team.  After all, we are the ones who maintain the infrastructure at the lab.

In my opinion, this should be pulled back and let’s discuss this further.  Perhaps it is the same path that Bob is proposing, perhaps not, 

but I would like to see us come up with a more solid plan to move forward that does not cost the lab any additional financial burden 

at this time.

-Jeff

[Eric Ose]

Jeff,

What I find interesting is that most of the previous operations team wanted the fewest number of things relying on our upstairs rack as possible.

That's at least what I think I remember over many years. I don't know all the factors behind that perspective.

Eric Ose
Robot Ambassador
It's just an idea unless there is a date and time included.

To unsubscribe from this group and stop receiving emails from it, send an email to heatsynclabs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/heatsynclabs/2b23411b-72eb-4915-8449-ce47181a5364n%40googlegroups.com.

[Robert Bushman]

> Unfortunately, I will be out of town for the May 23rd HYH.  I would like
> see this postponed until the following HYH (June 13th)  so
> I can be in attendance and available to discuss/answer any questions
> people may have.

That is a fair request. Even if it weren't, the vote on this proposal is
actually already scheduled for June 13th. I didn't submit it in time for
May 23.

> > Q: Why do you think you're better at operations than Jeff?
>
> A: What does this have to do with the Wiki?  Why are singling me out as
> the bad guy for having an opinion and voicing concerns that I
> have about it being hosted under a personal account and the cost of
> operating it?

First, I want to make sure you read my answer:

"A: I don't. Jeff has operations skills that I do not. I voted for Jeff
for operations both of the past two years, including the year that I had
been running against him. I can go into detail, but the short answer is
I can't do what Jeff does. I'm a fan, and I intend to vote for him again
this year, if he runs."

The reason for presenting that question and answer was to make it clear
to everyone that I am not impugning your abilities as the operations
board member.

Second, the reason I'm focusing on you is because you are the operations
board member. You were the person I tried to work with on this and you
don't need a proposal to make the change. You could just do it.

> As I mentioned above, we need to make sure we are implementing solutions
> that are properly governed by the lab.  Based on Bob’s answer,
> it sounds like he has experience in the software field and should
> understand the need for governance with situations like this compared to
> just going out and doing things on one’s own under the “do-ocracy”
> banner.  While I am a fan of going and doing things to make progress,
> doing so in a path that has potential detriment to HSL is not the
> correct way to do things.

There's an old joke in InfoSec about how you can secure any system by
pulling the plug. It's the same with governance. Good governance means
delegating, not just saying, "No."

That is true even in top-down corporate world, and it is especially true
at this non-hierarchical organization. We are not about authority and
gatekeeping. One of our core principles is "don't stop others from
making". The mission of board members and champions is to facilitate
volunteers who are moving the org forward.

> > 2. Nightly tarball of the recovery content, publicly accessible. (this
> > is a "run over by a bus" solution for non-hierarchical orgs - anyone can
> > pull nightly and clone the server at their pleasure)
>
> Again, the Database is already performing nightly backups.

That is a problem. Restoring the database alone does not restore the
wiki. I'm fairly certain that approach is what went wrong last time.

The MediaWiki documented backup and recovery process is what works. It
is how anyone can restore from the nightly. If a person is capable of
restoring from backup, they can pull the nightly and deploy a new
server. Once that happens, I suspect there will be insufficient
motivation to use the server I have running.

Not being able to restore using the MediaWiki standard recovery process
means that disaster recovery is not working.

> > 3. IPv6
> Why do we need IPv6?  What benefit is there from it?  While it is in
> use, it has not been widely adopted in the Internet community
> and just adds additional complexity

IPv6 has been widely adopted by network operators for years. It is the
path forward for things like IoT, and a rapidly increasing number of
hosts are being deployed as IPv6-only because of the cost of IPv4
addresses. Supporting IPv6 is a best practice.

> > This does not have lock-in. This is hosting, only. The server is vanilla
> > Deb 12, and the backups are published nightly, ready to be restored by
> > anyone who can do the work. Lift and shift is a snap.
>
> While this might not be “vendor lock-in” in a sense, it being hosted on
> AWS, what is the process for moving it to another provider if
> we so choose? Could a non-technical person do it?

The process is the same as disaster recovery:
1. Follow standard MediaWiki process to install MediaWiki.
2. Follow standard MediaWiki process to recover from nightly backup.
3. Point DNS to new server.

No, a non-technical person cannot do disaster recovery. That is true
regardless of who is hosting it or where it is.

> > The in-house network is flawed. We do not have the resilience of a
> > hosted solution. When I tried to work with you on the upgrade, you said
> > we don't have remote access. We don't have IPv6. It's fun to run our own
> > machines, and they should be used for fun. They are not
> production-grade.
> >
> > They should not be used for mission critical infrastructure.
>
> I disagree with the above statements, and in fact I feel it is a slap in
> the face of the team(s) that have kept things running for 10+ years (I know
> it is longer, I’m rounding down).  It has taken them a lot of sweat and
> time to keep the lab up and running and to make sure we are able to do
> what we can.  Now, is the infrastructure outdated and need some
> re-architecting?  Yes, but that does not mean we do not have the talent to
> do it, whether it is on-prem or in the cloud, this is something that
> should be discussed by the OPs team for a path forward that benefits
> everyone
> while making things easier to maintain.  As for the remote access, it is
> locked down for security sake.  Proper protocol is to not allow remote
> access directly to servers.

"I feel it is a slap in the face of the team(s) that have kept things

running for 10+ years ... Now, is the infrastructure outdated and need
some re-architecting? Yes"

Why is it a slap in the face for me to point out that the infrastructure
is outdated and needs to be re-architected, but not for you to do so?

"that does not mean we do not have the talent to do it"

I didn't say we don't, only that in the past 3.5 years that I've been
trying to get this done, we haven't.

"Proper protocol is to not allow remote access directly to servers."

I feel like "directly" is carrying all the weight in that sentence to
make it true. I've got no problem with a jump server. I have a problem
with having to be on-site to do administration. That is not a standard
practice.

> > This past September, you were amenable to my approach. When I arrived,
> > you said I could not have access to do the work. Then you said you
> > didn't want to do it they way I do it. Then you asked me to walk you
> > through it the way you want to do it.
>
> 1. You were wanting remote access to the server.  We do not just give
> anyone remote access to the servers.
> 2. You were wanting access to the ROOT user.  We do not just give anyone
> ROOT access to the servers.  That is a huge security risk.

I am not just anyone. I am the person who has documented this
repeatedly, been chasing it for more than 3 years, and the only person
who was done the upgrade and data migration successfully.

> As for “doing it my way”, perhaps there was some kind of disconnect, but
> I am not the Wiki expert and I had no “way” to do it.  I was
> deferring to your expertise.  As I told you, I had the Wiki upgraded and
> ready to migrate the data over and wanted your help with the
> best path to do that.  Unfortunately we seemed to be at a disagreement
> and did not get the data migrated.

You didn't follow my process. You didn't let me do what I offered to do.
You brought me in under the pretense of enabling me to do the work, then
told me to teach you how to do it. That was not what I had offered to do.

> > HSL owns ~[wiki.heatsynclabs.org](http://wiki.heatsynclabs.org/)~,
> which should be pointed at the new host,
> > ASAP. heatsynclabs.wiki is a placeholder. If we can find a good path
> > forward, I'm happy to transfer it.
>
> Having a “new domain” again introduces another cost of at least $10/year

Then don't transfer it. It is not needed. It is a placeholder. I'm just
offering to transfer it if HSL wants it.

> that the lab had not been made aware of, along with the
> approximately $7/mo expense of the hosting the Wiki.  While they are
> small, all these small expenses add up. Again, while I appreciate
> you initially covering these small expenses, by your own admission of
> “I'm happy to pay for it for the foreseeable future” how long is that
> future?  What happens when that is no longer the case?  HSL will be on
> the hook for yet another nickel and dime expense that “we” did
> not sign up for.  Also, who is going to maintain it if/when you lose
> interest in maintaining it, or it becomes a bigger job than you want?

I answered these questions already.

This is a rounding error in my monthly spend on hosted services. I've
been a paying HSL member for 13 years. I currently have three other
MediaWiki servers running, which have been up for years. It's not a big job.

And, again, anyone capable of running a MediaWiki server and handling
backup and recovery can pull the nightly backup and deploy a replacement
server any time they want.

> > A hosted server gets us some resilience and network features that would
> > be difficult or costly to replicate internally.
>
> We already have the infrastructure in place for this and have been doing
> it for 10+ years.

Are you saying the rack upstairs is as resilient and full featured as
AWS? Are you saying that despite having said that it is out of date and
needs to be re-architected?

> > HSL should own and control the outside wiki as soon as it is able. I
> > will be publishing a disaster recovery doc which will work as a
> > migration doc. Deploying a replacement server is essentially the same as
> > recovering from an HDD failure. As soon as the HSL server meets our
> > needs, I'll kill the one hosted on my account.
>
> Since the server hosted at the lab meets our needs, I would rather see
> an effort in getting this “ask” over the hill and coordinating getting
> the data migrated than to spend time working on a cloud instance and
> then moving it back to the lab.

It does not meet our needs. The old server is running a 12 year old
version of MediaWiki. The new server does not have the content from the
old one.

> > I agree with running it by HYH. That has been and remains my intent.
> Was this discussed at a HYH?  I did not see it on any HYH agenda for
> discussion.  Perhaps I missed it.

This was discussed at HYH, it is in the HYH notes.