Request for comments: Calamares Python API

[Teo Mrnjavac]

Hello everyone,

this email is for those of you who have already used the Calamares Python API
and wish to shape its future.

The Calamares Python API was first developed about two years ago, way before
many other essential features of Calamares had fallen into place. Over these
two years Calamares has transitioned from a very rough proof of concept to the
fairly complete system installer framework we have today, but the Python API
has seen very few changes.


Based on the past two years of use, including bug reports and conversations
with some developers and system integrators among you, I've identified a
number of issues.

* Python modules don't support translations. This is bad, because they cannot
output user-visible strings. This in turn blocks many improvements in progress
reporting and other things.

* The Calamares module format is somewhat unpythonic and arbitrary. Currently
the module developer has to implement a run() function, and this function
isn't enforced by an interface or any other Python-understandable mechanism.
On a related note, there is no way to pass instance arguments from the
Calamares side to the module (besides the libcalamares.job.configuration
mechanism).

* Since the entry point of the module is just an arbitrary global scope
function, Calamares injects a number of objects into the Python module's
environment before calling run(). These objects are then known to module
developers as libcalamares.job, libcalamares.utils and
libcalamares.globalstorage. While it makes sense for
libcalamares.globalstorage to live in the global scope, libcalamares.job is
something that would be much more meaningful as "self" in the Python module.

* Testing is harder than it should be. This is a consequence of the previous
two points. On a related note, does anyone use testmodule.py? Does it work? I
know that it doesn't work for some more complex modules, but have you found it
useful at least for simpler ones?

* Limited scope. The Calamares Python API technically only allows the module
developer to write a Python Job. Should this be extended to allow multi-job
modules? GUI modules? Control flow scripting (i.e. replacing the "sequence"
item from settings.conf)? You tell me.

* Lack of documentation. I don't have a good excuse for this.


I have a number of ideas on how to address these issues, but everything is up
for discussion at this point. This is a mid-long-term effort. Ideally, in the
Calamares Python API v2, creating a module shouldn't mean writing a run()
function and relying on globals, but inheriting from an abstract base class
Job or Module.

Please help me expand this list. Have you encountered design elements in the
Calamares Python API that bother you or simply wish to see improved?


PS. whenever I mention Python GUI modules, people say "PyQt". This is not what
PyQt is for. We need to use Python to script a C++/Qt5 application, i.e. an
embedding solution, not a solution to develop a Python application with a Qt5
GUI. While allegedly a dirty trick might exist to achieve the former, normally
PyQt deals with the latter.

Cheers,
--
Teo Mrnjavac
http://teom.org | t...@kde.org

[artoo]

Hi Teo,


thanks for adding me.

Does it mean I can push to calamares-manjaro?

Before I go in some thoughts on modules, I noticed workflow for distro
specific modules could be improved.

Perhaps have distro subfolders and chose distro in cmake/make or something?


Now, modules, the immediate need in my view is error handling.

Example:

If a module fails, the target root still has api fs folders
mounted(sys,proc,run,dev).

Atm, I think each module would have to implement a umount call, the only
module which has such thing is umount module.

This make testing a module a bit painful, and the end user might try
again to install, but hdd setup will fail due to still mounted api fs.

So I think, this should be perhaps one of the first steps for new python
api.


I like the idea to create gui with python. This way, one could provide
some settings to a module in gui.

There are plenty of idea how to make use of it.

Interfaces, that each module has to implement sounds also good idea.

Another, in my view quite important feature would be to add a initsystem
type detection in welcome module or so.

Each module could query or even set it it to do init specifc things
instead of having init specific modules.


I haven't tried the testmodule.py yet, I looked at code, and figured it
would not work for the modules I working on.

Last, is there probably a better mechanism to configure modules?

While settings.conf is fine, it also has disadvantages, eg netinstall
module would be nice to activate in gui optionally.


So far my thoughts


Regards

Marcus

[Teo Mrnjavac]

On giovedì 18 agosto 2016 11:01:39 CEST artoo wrote:
> Hi Teo,
>
>
> thanks for adding me.
>
> Does it mean I can push to calamares-manjaro?
>

You can now. Manjaro related things (including the calamares-manjaro repo) are
handled by Philip Müller.

> Before I go in some thoughts on modules, I noticed workflow for distro
> specific modules could be improved.
>
> Perhaps have distro subfolders and chose distro in cmake/make or something?
>

I don't think that would work well, because (1) distro-specific modules don't
live upstream, and the whole point of Calamares is avoiding forks. If every
distro maintained and further developed their own fork, we risk regressing to
the whole Cnchi/Thus mess. Also, (2) there is no reason to make this choice at
build time in CMake, since Python modules can easily be mixed/matched at the
package level after Calamares-upstream is already built and shipped.

However, I do realize that Calamares configuration, modules and branding may
end up all over the place on some distros. We can try to address that.

>
> Now, modules, the immediate need in my view is error handling.
>
> Example:
>
> If a module fails, the target root still has api fs folders
> mounted(sys,proc,run,dev).
>
> Atm, I think each module would have to implement a umount call, the only
> module which has such thing is umount module.
>
> This make testing a module a bit painful, and the end user might try
> again to install, but hdd setup will fail due to still mounted api fs.
>
> So I think, this should be perhaps one of the first steps for new python
> api.
>

This is a very good point: adding a mechanism which allows modules to specify
a sort of catch clause, which gets called by the job executor in case of fatal
error anywhere in the pipeline. For this to work, Python modules ought to keep
an internal state. Noted.

>
> I like the idea to create gui with python. This way, one could provide
> some settings to a module in gui.
>
> There are plenty of idea how to make use of it.
>
> Interfaces, that each module has to implement sounds also good idea.
>
> Another, in my view quite important feature would be to add a initsystem
> type detection in welcome module or so.
>
> Each module could query or even set it it to do init specifc things
> instead of having init specific modules.
>

I'm not sure I understand this. Do you mean a way to detect whether we're
installing systemd or openrc?

>
> I haven't tried the testmodule.py yet, I looked at code, and figured it
> would not work for the modules I working on.
>
> Last, is there probably a better mechanism to configure modules?
>
> While settings.conf is fine, it also has disadvantages, eg netinstall
> module would be nice to activate in gui optionally.
>

There is no support for optional module activation yet. For this to work, the
"sequence" item in settings.conf would need to be replaced by a more powerful
mechanism, like a Python script.