MPF Installers Rework Project. Need your input & feedback

[Brian Madden]

Hi Everyone!

I'm Brian. I was really involved with MPF back in the day, though I've been away for awhile. Earlier this year I joined FAST Pinball full time, and as part of that am putting some time into MPF.

The MPF installers are pretty old now and need to be updated. My belief is that many people are not running the latest versions of MPF because getting everything installed is complicated, so once it's working, people are afraid to touch or change anything for fear of breaking it and never getting it to run again. :)

So I am rewriting and rebuilding the MPF installers from scratch. (Actually, MPF itself is simple to install. It's the MPF-MC which is the pain, since that requires those multimedia libraries (SDL2, Gstreamer) as well as it has code which needs to be compiled, and it has to integrate with Kivy, etc.

If anyone is familiar with Python installers, I will be using the most modern approaches, setuptools, pyproject.toml, the new build packager, leveraging GitHub Actions for CI/CD along with the cibuildwheel, etc.

You can follow along in the new-installers branch if you're interested:

https://github.com/missionpinball/mpf-mc/commits/new-installer

I am hoping to ultimately figure out how to ensure that the MC uses the GPU, that we can run it on a Raspberry Pi, etc. But for now I'm just working on the installation process itself, and getting all the various packages for the different platforms built.

My initial goals will be to support:

• macOS, Intel
• macOS, Apple Silicon
• Windows 10 / Windows 11, x64 only
• Linux, prob Ubuntu 18-04 +
• Raspberry Pi 4

It's also looking like Python 3.8 and 3.9 will be the targets for most. Python 3.10 will require some work (which can be done in the next few years, no hurry), and I think we can drop Python 3.7 (though I'll confirm which Python is in the Ubuntu LTS builds, keeping support for 3.7 probably is fine for now too).

Ultimately I just wanted to share that I'm working on this. (Along with Jan, Anthony, and Quinn who have contributed so much and I'm leaning on for help.) If anyone has any thoughts or feedback, suggestions, recommendations, or ideas, or if anyone wants to help, please reach out or reply here.

Thanks, and happy pinballing!

Brian

[Ralf G.]

I am hoping to ultimately figure out how to ensure that the MC uses the GPU, that we can run it on a Raspberry Pi, etc.

Brian, big Hooray to that, and many thanks for letting us know. I am no coder at all, but if you need a tester, let me know.

Ralf

[Todd Legere]

Hello Brian,

I have been mostly doing installs on linux hosts (Ubuntu/Lubuntu/Xbuntu 20.04, 21.04, 22.04, Raspberry Pi 3.b)

Once you get into Python 3.10 that were you see the builds failing for mpf-mc. I can say that mpf, mpf-mc builds ok in 21.04 but after that your out of luck.

Using Virtualbox and virtual machines I have been able to test many examples of O/S version with mpf installs.

On day I would like to pair down the linux version so that it only contains the neccessary modules and packages and does quick boots for my two cobra board systems.

Perhaps what may be nice is to have the installer have optional modules that uninstall packages and modules that are not necessary on the mpf host or an option that ask the question...

Is this a development machine or production machine.

If it is a development machine then you want the full suit of mpf, mpf-mc, mpf-terminal, etc, its gonna be a larger install.

If it is a production machine then strip off the linux desktop apps like libraoffice, fat desktop apps, etc.

Let me know if I can be of assistance of testing installs or providing feedback about profiling install choices (Development install or Production install).

I will be monitoring the github url. Thank you for exploring the issue of the installer.

[jabdoa]

Pipx would be one of the more modern ways to install python packages. Another variant would be docker images but that might not work with graphics on win and mac. I would just go with pipx and improve instructions on how to install dependencies. Maybe with some scripts (similar to linux). Then test those on some ci.

Jan

[Cole M]

I think a lot of people find the multiple versions and spotty compatibility within python confusing. Using a python virtual environment appears to be an important component of setting up a stable development system.  That's a step above what many casual users are comfortable setting up independently.  Either step by step instructions on this or including the compatible python and environment setup as part of the install would be helpful if such a thing is possible.  Also, we will need to remember to include either instructions or an option for the multimorphic hardware component.  I'm working on a fresh install on a ubuntu system this week.  If you have anything you want me to try, now is an ideal time.  I will happily give it a go on linux and try it out with FAST and MM boards.

[Jhon Smith]

Just tried new installation on Windows based on new guide (https://docs.missionpinball.org/en/latest/install/windows.html) . and got this error: 
Seams this was also in some old ones

[Brian Madden]

Thanks for sharing. I have not seen this before. Unfortunately I’m traveling this week and do not have a Windows computer with me, and my Mac laptop is an M1 so I can’t run Windows in a VM :(

So I might not be able to fully troubleshoot until I get home. But in the meantime, can you tell me more about your environment? Version of windows, version of python, and did you have MPF installed before, and did you remove or change anything before you tried to install the latest?

So far I’ve only tested on fresh machines, so I need to learn about how the upgrade process might work.

Thanks!

Brian

[Brian Madden]

Oh, also, for those files it couldn’t find. Can you verify that those files actually exist? (If you know how to do that?)

Thanks!

[Jhon Smith]

Hi, This was fresh windows 10 (64bit) install with Python 3.9  venv (reinstalled windows and added python 3.7 too) It not showing. It is showing this and nothing more.  

Checked there are files in "share" folder.  

I will try again maybe missed something. If not I will install windows 7 like on my Mac parallels where everything works. :)

[Brian Madden]

Wow I have no idea what the issue is. The automatic builds on github which build the wheels and packages follow this same process, and I tried on local Windows machines that were fresh installs of Windows 10 x64 and they all worked too with no issues. So I don’t know what it could be.

Depending on how proficient you are with Windows, the command prompt, and Python, you could try a few things.

The first would be to confirm that the correct version of python is on the path, like:

python --version

Then since this is a fresh install, maybe try using pip instead of pipx?

Then: (as three separate commands)

python -m pip install --upgrade pip

pip install --upgrade setuptools wheel

pip install mpf-mc --pre

This is the exact procedure I've been using locally on Windows. (You could try this all in a venv too if you want)

Brian

[Jhon Smith]

Finally works.  Managed to do this in 3rd clean install :)

System: clean Windows 10 64bit

Python 3.7  from python website

Last time followed steps: 

1) python -m pip install --upgrade pip

2) pip install --upgrade setuptools wheel

And added this line instead (pip install mpf-mc --pre) 

3) pip install mpf[all] mpf-mc

Seams in my case missing parts was in [ALL] section. 

[Jhon Smith]

Thank You!

[Brian Madden]

Awesome, glad to hear it. AND, thanks for testing this. I will update the procedures in the docs!

Brian

[Dan - aka BorgDog]

Hi Brian, thanks for the new installers.  Just removed all my old python/mpf stuff off my windows 11 machine (was on python 3.6 still) and ran through your new installers, everything worked fine, and my game on an older .56dev runs fine.

So a couple questions:

• is the install for monitor and language server still the same at this point?
• what is the procedure for upgrade now, I assume it runs through pipx? I don't see/can't find in docs.

Thanks,

Dan

[Brian Madden]

Thanks for the feedback Dan, and glad it worked! I didn't actually test on Win11, I was assuming it would work, but it's cool to have confirmation.

I did not look at the installers for the monitor or language server yet. I think those are not quite as high priority since I think those are still pretty easy to install. (Tell me if I'm wrong though.) I'll still get to them at some point in the next few weeks-ish? I think?

For updating moving forward, also a good question that I didn't document yet. :)

The procedure should be the same, essentially. That "--pre" flag will tell the installer (pipx in this case) to grab the latest prerelease version, so if you have 0.56.0.dev32 now and 0.56.0.dev33 is posted, then just running that command again will grab the latest and install it (along with any updated prerequisites). In fact you should only need to run the MPF-MC part each time because it should pull in the latest MPF. (The only reason they're separate in the install is because pipx can't take more than one package at a time from the command line and we need to get "mpf[all]" in there and also mpf-mc.

I'll update the docs now for this. Again thanks for testing and please post back if you have any issues.

Brian

[Dan - aka BorgDog]

hmmm.  OK followed the instructions for installing MPF Monitor, basically just  pip install PyQt5 and  pip install mpf-monitor --pre

now when I try to run the same game that worked earlier I get:

C:\Users\borgd\OneDrive\Pinball\__Homebrew\MPF_Descent>mpf both
Process Process-1:
Traceback (most recent call last):
  File "C:\Python\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Python\Python39\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python\Python39\lib\site-packages\mpf\commands\both.py", line 13, in _start_mc
    module = import_module('mpfmc.commands.mc')
  File "C:\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mpfmc'

if I recall correctly, running pip list did not show mpf and mpf-mc, but pipx list showed them. after installing monitor as above, mpf now shows in pip list.  should I have used pipx to install monitor instead of pip? (actually have no idea what pipx is)

OK, tried to figure this out, but managed to bork things worse so going to start over again.  

Will wait for your feedback before trying monitor again, and to answer your question having monitor working is pretty important, at least to me.

thanks for all your work, and no rush

-Dan

[Brian Madden]

Ah ok.. I didn't think about it, but seeing this it make sense.

pipx is basically pip + that python virtual environment stuff rolled into one simple tool. So when you install something with pipx, it installs it into a virtualized / isolated environment (so that nothing else conflicts or breaks it), and then when you run the "mpf" command in the background it's activating that virtual env. (So no more of that activation command like if you manually used python venv.)

When you installed mpf monitor with pip, that was in the normal python environment, which didn't have mpf (since it was in the virtual one and didn't see it), so mpf-monitor probably installed mpf into the non-virtual environment too. If you do pip uninstall mpf-monitor (and any other components that show up in pip list), then maybe run "pipx --reinstall-all" (there are two dashes before that like the other pipx flags in case the google smashes them together.) You may also have to do that pipx ensurepath again too?

Anyway so yeah, moving forward we'll need to use pipx to install mpf-monitor and the other things too. I'll take a look at this tomorrow morning and see what I can figure out, should be pretty straightforward esp since the monitor and other things do not require any compiling.

Thanks for helping test all this out! I'll post back tomorrow.

Brian 

[Brian Madden]

Ok, looks like the process to install a new package into an existing environment is pretty straightforward. Just run this:

pipx install mpf mpf-monitor

That command is saying "install the mpf-monitor into the mpf virtual environment"

So give that a shot? I cannot get it fully working here because I have my M1 mac, and it looks like PyQt5 (the graphical window library that mpf-monitor uses) does not have a version for M1 mac yet.

I have to leave soon so I don't know if this is an easy or hard fix, or maybe the newer version of PyQt6 does and we just need to update the monitor for it. But for Intel mac or linux or windows I think that's all you should need?

You could also use:

pipx install mpf-monitor --pip-args="--pre"

to install the dev branch of the monitor.

I'll revisit this again tomorrow morning to get it properly documented and figured out. The good news is the monitor should work with the new stuff except for worst case M1 macs.

[Dan - aka BorgDog]

gave me an error on that command, I believe it is inject instead of install.. I'm going to wipe out and try all again, done too many things mine is all messed up

[Dan - aka BorgDog]

OK, here is what I did that seems to get everything working, including monitor and the MPF window issue that is in the other thread. Basically inject everything into mpf venv after installing that one first.

install python

command window:

pip install --user pipx 

python -m pipx ensurepath

close and restart command window:

pipx install mpf[all] --pip-args="--pre" --verbose --include-deps

pipx inject mpf mpf-mc --pip-args="--pre" --verbose --include-deps --include-apps

pipx inject mpf PyQt5 --verbose --include-deps --include-apps

pipx inject mpf mpf-monitor --pip-args="--pre" --verbose --include-deps --include-apps

again this is on windows 11, with python 3.9.13.  I am far from an expert at all this, so don't know if that is really the right way to do it, but it all seems to be working now.

[Brian Madden]

This is fantastic and a huge help, thanks!

And based on what I am seeing & learning, this does seem like the right way to do it. I’ll update the instructions tomorrow to include this.

Also I noticed that PyQt6 is pretty compatible with PyQt5, so I can update mpf-monitor for that and also update the installer to match the new installer style of mpf and mpf-mc and then make a nice simple all in one install plan. (Something like “install mpf dev workstations” or something which would do MPF, MPF-MC, MPF Monitor, and all the other tools.)

But at least we have it working now, thank you, a load off my mind!

Brian

[A Kaljunen]

Hey. I have readed this interesting topic and followed those updated instructions and need to ask, that there is little difference about those two cases. After restarting command window here says, that ( pipx install mpf[all]....) and updated instructions says, that  ( pipx install "mpf[cli]"....)? What might does those apostrophes do and what and why it is different than here? I have had problem with my MPF MC and might this help for me that get MC work.

Aclac

[cobra18t]

Brian,

Any ideas on Aclac's MC not starting issue here:  https://groups.google.com/g/mpf-users/c/OwL7Q6Ig_Fk 

I believe he used the installer but is having an import error on audio_interface.pyx. 

[A Kaljunen]

My problems seems to be solved, when I downgraded  Mission Pinball Framework Media Controller v0.56.0-dev34 to  Mission Pinball Framework Media Controller v0.56.0-dev33. Now my MC working anyway as expected:)

[Quinn Capen]

I will have some time next week to look at the Windows build process to see why dev34 is not working properly. It appears the build process is correct, but the custom audio libraries cannot locate the supporting dlls (sdl2, gstreamer, etc) after being installed.

Quinn

[grastle flowpat]

Howdy,

I was wondering if there is a beta of the new Linux installer to test?  I had a couple of goes in VMs, even managed to compile a 3.9 python with SSL to get pip working, but no luck doing things the way I thought they should go, the old installer went down a twisted maze of requirements on modern distros, so I left that alone.

The link to https://github.com/missionpinball/mpf-mc/commits/new-installer in original post gives a 404.

Thank you

[Brian Madden]

I went through the steps to also build a Linux installer, but I haven't written up any instructions. And I forget if I tested it locally. You can see the latest runs and artifacts here: https://github.com/missionpinball/mpf-mc/actions/runs/3033818604

And you can see the pypi files here (https://pypi.org/project/mpf-mc/#history) so you should be able to install with pip? You can check out the build and testing scripts that run the builds to see other things that might be needed: https://github.com/missionpinball/mpf-mc/blob/dev/.github/workflows/build_wheels.yml

As for the Windows installer issues, I will try to get some time in the next few days. Unfortunately I have an M1 Mac so I can't even run Windows in a VM on this thing, I need to fire up my old desktop computer. The difference between dev33 and dev34 was only to remove the hard dependency on MPF, (because at FAST Pinball we are creating our own MPF distribution so we need mpf-mc to work with anything that uses the MPF API versus only a package called "mpf"). But I have no idea why 33 would work and 34 wouldn't. Literally it was just a single line deleted in a requirements file: https://github.com/missionpinball/mpf-mc/commit/9ca0336071b22b66068ec4abcb31d712063d2968

Finally, what page is that broken link from? That link was to the "new-installer" branch of mpf-mc where I was figuring out the new installer, but it's no longer there because it's been merged into the main (dev) branch now.

[Brian Madden]

I think I realized what the installer issue is when using the latest (dev34) instead of dev33.

The solution will be to install mpf first and then install mpf-mc into the mpf pipx container. (So, switching the order of the install.) I will test and update the docs.

The technical reason, for those interested, is because pipx only works with python apps that have console entry points, which mpf-mc does not have, but mpf does. So removing mpf as a prereq means that mpf-mc will no longer install via pipx. So instead we’ll just install mpf first with pipx, and then install mpf-mc (and mpf-monitor and anything else) into the mpf pipx environment.

Brian

[grastle flowpat]

Thank you for al the great info, and hard work. The "broken link" was from your original post :-)