[mathesar-developers] Installation Improvement project 0.1.4 feedback

[Mukesh Murali]

Hey all,

I've written up a final proposal for the [Installation plan](https://wiki.mathesar.org/en/projects/installation-improvements-plan-0_1_4). This should explain the terminologies used to give a general overview of the Installation plan and the changes to expect with the new installation process instead of going deep into the implementation details

These two projects aim to cover the details of the implementation and mostly concern @Anish Umale, @Ghislaine Guerin, @Rajat Vijay and @Brent Moran 

[Laying the groundwork for improving our installation process](https://wiki.mathesar.org/en/projects/installation-improvements-0_1_4) - This project contains codebase and infrastructure-related changes. It will lay the ground work necessary for updating the documentation which will be worked on after the upcoming cycle
[Implmentation details related to the documentation](https://wiki.mathesar.org/en/projects/installation-improvements-0_1_4) -  This is still a work in progress and is in a draft state. It will be worked on after the upcoming cycle.

I'd appreciate feedback on it. 

Thanks

[Mukesh Murali]

Hey all,

Please ignore the previous mail and read this one. I accidentally sent it without formatting the markdown.

This

I’ve written up a final proposal for the Installation plan. This should explain the terminologies used to give a general overview of the Installation plan and the changes to expect with the new installation process instead of going deep into the implementation details

These two projects aim to cover the details of the implementation and mostly concern @Anish Umale, @Ghislaine Guerin, @Rajat Vijay and @Brent Moran

Laying the groundwork for improving our installation process - This project contains codebase and infrastructure-related changes. It will lay the groundwork necessary for updating the documentation which will be worked on after the upcoming cycle
Implementation details related to the documentation - This is still a work in progress and is in a draft state. It will be worked on after the upcoming cycle.

I’d appreciate feedback on it.

Thanks

[Kriti Godey]

Both your project links link to the same page, Mukesh. I am assuming the second project was supposed to link to https://wiki.mathesar.org/en/projects/installation-documentation-improvement-2

I don't think the plan to split up the projects the way you've described makes sense. This means we won't get any user-facing improvements to installation until release 0.1.5 when the documentation gets updated. Instead, I'd like to see a plan that prioritizes user-facing changes and starts implementing the documentation outline in 0.1.4 (even if full implementation is delayed until 0.1.5).

For example, you could choose to implement the new "Install with Docker" and "Install with Debian" pages, and remove the Docker Compose / guided script installation in 0.1.4 and implement the PyPI / PaaS / Helm installations in 0.1.5. There might be other ways to split up the work too. But we absolutely need to get simplified installation out to users ASAP. It's the number one complaint we get, and we're a few months past launch and haven't made any significant improvements yet.

Also, please address the work that was supposed to go out in 0.1.3 and make sure you're making time for shipping those changes too.

[Mukesh Murali]

I updated the [Installation Improvement Project 0.1.4](https://wiki.mathesar.org/en/projects/installation-improvements-plan-0_1_4) based on the feedback

- Removed Installing using Helm charts and PyPI. It will be worked on in later cycles

- Added updating documentation for Debian Install and Docker image to this cycle

- Added removing docker-compose and Build from source documentation.

- Added @Sean Colsen as documentation reviewer and @Pavish Kumar Ramani Gopal as Product and Design reviewer (taken over from @Kriti Godey)

Please reply if there are concerns by Monday (2023-08-21)

[Mukesh Murali]

Follow up on the previous email

I moved most of the installation plan-related things from Installation Improvement Project 0.1.4 into the Installation Plan.

Reading through the Installation Plan should be enough to give an overview of the installation plan. It does not contain information on the work to be done in this cycle and is just used as a reference for the actual Installation Improvement Project 0.1.4 which mostly concerns, @Ghislaine Guerin, @Rajat Vijay, @Sean Colsen @Brent Moran as it has implementation details of the work we will be doing in this cycle to achieve the plan.

Please reply if there are concerns by Monday (2023-08-21)

[Dominykas Mostauskis]

I'll preface by saying that I've not been tracking all the various discussions surrounding installation. Most of the team seemed to be involved already, and the issue seemed complex. During last team meeting we agreed to provide feedback, so here it goes. I'm sure some of this was already discussed in installation discussions, so this might just be about bringing me up to date.

- What's our support and priority for installing without virtualization, containerization or debian packages? I would consider these installation methods to be nice to haves, but if I've a problem I want to be able to fall back to a good old command line step-by-step process. If I can't do that, that might be very frustrating, because then I have to debug the virtualization, containerization, or debian package solution just to get the thing working.

- I saw that downgrading our Python version requirement is part of the project. Why? Is it difficult or undesirable to ship our own Python version? Is setting up own python environment not standard practice anyway? Python is the primary tool we're using on the backend. I hope we don't force upon ourselves a lower version without a very good reason.

- In the "should we install things on the user db" email thread, we came to the agreement (it seems) that user UX could be improved if we optionally allow them to provision Mathesar with our internal schemata and a user that only has write access to those schemata (and whatever else the user decides to let us edit), thus making Mathesar safer. Seems like it's a bit late in the planning phase to get this into the next release. But, I'd like to know now that we see a clear path to this kind of installation. If anyone has concerns about this suggestion, please respond in the mentioned email thread, for the sake of focus. In this thread, I think we should only discuss how it fits into the wider installation plan. https://groups.google.com/a/mathesar.org/g/mathesar-developers/c/d6HQlCpEvJI/m/RzPPkWsFBQAJ

[Sean Colsen]

I've skimmed over the project page and the plan. I don't have any questions or changes to suggest.

[Mukesh Murali]

Thanks @Sean Colsen and @Dominykas Mostauskis

What’s our support and priority for installing without virtualization, containerization or debian packages? I would consider these installation methods to be nice to haves, but if I’ve a problem I want to be able to fall back to a good old command line step-by-step process. If I can’t do that, that might be very frustrating, because then I have to debug the virtualization, containerization, or debian package solution just to get the thing working.

Status:

• We will be completely removing the “Build from scratch” installation guide which contains information for the step-by-step process
• We might add it later if there is enough feedback from the community

Reasons:
Part of the current installation problem has to do with us assuming the user might want something instead of going for a minimal setup and asking for feedback from the user which ended up in a lot of wasted resources. Just adding the steps involved wouldn’t be a good UX. We would also need to explain why and when to do something which takes a lot of effort, especially for something granular like building from source. Also with lots of changes to the Installation process, the current “Build from scratch” guide is quite outdated and we again need to rethink how to explain certain things. So it is not a good use of our time now.

I saw that downgrading our Python version requirement is part of the project. Why?

We need it for supporting various versions of Debian Distro as we will be using the system Python when installing using Debian packages

Is it difficult or undesirable to ship our own Python version? Is setting up own python environment not standard practice anyway?

Massive pain for someone not used to installing Python. Half the installation problem is with installing and setting up Python. Look at this hackernews comment

I hope we don’t force upon ourselves a lower version without a very good reason.

We will be supporting only the actively maintained versions

we came to the agreement (it seems) that user UX could be improved if we optionally allow them to provision Mathesar with our internal schemata and a user that only has write access to those schemata (and whatever else the user decides to let us edit), thus making Mathesar safer

It would be even easier to add these changes after we implement the installation plan. I will comment on the email thread about it.

[Pavish Kumar Ramani Gopal]

It looks good to me. I do not have any concerns/questions on the project.

[Brent Moran]

Regarding installing from scratch, I think we could consider beefing up our developer installation README.md to make up for anyone who is turned off by the removal of that section of docs.

Regarding the deprecation of the docker compose I think it's worth at least documenting that if you are okay with losing your metadata, you can simply set up Mathesar using the new version and point it at your current DB where you had the user tables installed, etc. I don't think we need to add migration scripts at this point to make that less painful.

Otherwise, I think it looks fine.

[Dominykas Mostauskis]

Regarding Python version, which version would you like to support? I see the current Debian stable uses 3.11. And, regarding bundling of Python in the Debian package, I'm not sure I understood your response. The HN thread you linked to links to Nuitka, which could build us a system-python-independent binary, right? Of course, if we're targeting Debian stable, that's not necessary.

You tersely responded to my concern about DX being degraded by downgrading Python version by saying that it should be enough to use whatever a maintained Debian release supports, correct me if I misunderstood. I strongly disagree. A newer Python feature being available has saved me time numerous times. I think it's smarter to invest into making our distributions portable rather than our code. 

[Mukesh Murali]

Regarding Python version, which version would you like to support?

We will be supporting Python versions from 3.7 onwards

I see the current Debian stable uses 3.11

We cannot just consider the current stable. We also need to account for LTS releases and the current LTS, Buster uses Python 3.7.

You tersely responded to my concern about DX being degraded by downgrading the Python version by saying that it should be enough to use whatever a maintained Debian release supports, correct me if I misunderstood. I strongly disagree. A newer Python feature being available has saved me time numerous times.

We had a similar discussion and concluded to not update to the latest Python as we still need to support users using the old version of Python. At that point, we weren’t sure of supporting a lower version of Python but with our Debian releases dependent on the system Python, we have to extend our support to Python 3.7 too.
We can re-open that discussion to talk about this and if needed we can stop supporting lower Python versions. In case we decided to drop support for lower versions the users using old versions of Debian can install Mathesar as a Python module using pip (nstalling as Python module from PyPI) instead of using the Debian package and running it using a newer version of the Python interpreter, so I don't think it affects the installation plan.

I think it’s smarter to invest into making our distributions portable rather than our code.

We could use some scripts to make our code compatible with the old version (which is out of the scope of this project) or package a Python interpreter along with the code which is not an easy thing to do because

1. It increases the size of the binary due to adding a Python interpreter to the distribution
2. I briefly tried using some tools(cx_freeze, PyOxidier) which help you ship a Python interpreter and even though I was able to package our app. It wasn’t a pleasant experience when building (unexpected run errors on certain machines, incompatible with certain libraries especially with C libraries like psycopg , also most libraries don’t follow Python standards properly which leads to conflicts with standalone Python)
3. Using the binary was also an unpleasant user experience as the startup was very slow
   
   

The HN thread you linked to links to Nuitka, which could build us a system-python-independent binary, right

Maybe you are referring to the first comment. The parent comment is pointing out the fact that installing Python is not trivial. Any way the intent was to convey that we cannot always expect a majority of our users to install a specific version of Python in addition to the default system Python.

[Kriti Godey]

A few thoughts:

(1) I am not sure if this addresses the leftover work planned for 0.1.3

(2) Who is doing the documentation, discussion, and research work? The other work has people assigned.

(3) Why are we researching PaaS options in this cycle? Seems unrelated to the output planned, and will we have enough time?

(4) General question: will we have enough time to do all this?

Otherwise, looks good.

Dom, I don't think we should revisit our decision to support a wide range of Python and Postgres versions at this point. We discussed this extensively at the installation meeting and made a decision – that was the place to raise concerns. At a high level, we decided that we want to make Mathesar easy to deploy in a variety of environments since we are more likely to gain users if we work within people's existing self-hosting setups. There was a lot of surrounding context that I don't think we can easily describe in email, I'd recommend talking to Mukesh or Brent in a call (since they were both part of the installation call) if you want to learn more.

[Dominykas Mostauskis]

Sounds like other practical options were exhausted before deciding to target 3.7. That satisfies me. Thanks for the context, Mukesh and Kriti. 

[Mukesh Murali]

This update mostly concerns @Kriti Godey since she had some feedback on the project.

I made the following changes to the project

• Removed research work. We won’t be researching on list of PaaS to support in this cycle
• Made GitHub issues for all the tasks and added the link to the meta issue which tracks all these issues
• Assigned myself to the documentation and discussion work

Let me know if your concerns are addressed

[Kriti Godey]

Here's the project link for anyone in this thread who wants to look.

Mukesh, this looks good. Please update the project to approved.

Some notes:

- I removed myself as owner and fixed a typo.

-  It looks like the migration to MkDocs messed up the formatting of the timeline table and the status paragraph. Please fix.