Reminder: hotfix fast path needed

[Jan Kiszka]

Hi all,

just to remind you of a long-standing community desire:

Patches that carry "Fixes:" tag and address build breakages should be on
a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
almost on a daily basis, but those are just two recent examples.

The good news in this: More and more folks are actually testing recent
versions, finding such issues and providing feedback.

Thanks,
Jan

[1]
patchwork.isar-build.org/project/isar/patch/20251017085344.26470...@siemens.com/
[2]
https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03...@siemens.com/

--
Siemens AG, Foundational Technologies
Linux Expert Center

[Baurzhan Ismagulov]

Hello Jan,

On 2025-10-23 07:52, Jan Kiszka wrote:
> Patches that carry "Fixes:" tag and address build breakages should be on
> a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
> almost on a daily basis, but those are just two recent examples.
>

> [1] https://patchwork.isar-build.org/project/isar/patch/20251017085344.26470...@siemens.com/
> [2] https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03...@siemens.com/

I acknowledge the desire to have fast path for "Fixes:" and build breakages.
We've merged #2. The full results for #1 will be available tomorrow. There is a
number of failures due to e.g. external networking issues; we'll be sharing
patches for those.

I think introducing fast path for certain changes is the easiest part of the
equation. In #2, you also express regret that there is no testcase for the
feature in question. In general, we should think about adding testcases on use
case introduction and bug fixes -- here I agree with Cedric. I think this has
larger contribution to the quality, benefits all downstreams and saves the
overall fixing effort, as every applied change could break the feature. Once a
fix goes in, a testcase never follows. Currently, we are developing testcases
for certain changes from the list; longer-term, this will not work.

The Linux "tested by many" model helps only to some extent. We also see e.g.
conflicting changes being applied back and forth, even in Linux. Having the use
case view backed by tests at the Isar level would avoid this.

To summarize, we'll evaluate this proposal and also invite for more
collaboration on testcases.

With kind regards,
Baurzhan

[MOESSBAUER, Felix]

On Thu, 2025-10-23 at 12:42 +0200, Baurzhan Ismagulov wrote:
> Hello Jan,
>
> On 2025-10-23 07:52, Jan Kiszka wrote:
> > Patches that carry "Fixes:" tag and address build breakages should be on
> > a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
> > almost on a daily basis, but those are just two recent examples.
> >
> > [1] https://patchwork.isar-build.org/project/isar/patch/20251017085344.26470...@siemens.com/
> > [2] https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03...@siemens.com/
>
> I acknowledge the desire to have fast path for "Fixes:" and build breakages.
> We've merged #2. The full results for #1 will be available tomorrow. There is a
> number of failures due to e.g. external networking issues; we'll be sharing
> patches for those.

That's one of the core problems: The test suite itself is too unstable
and also way too complicated to be executed on each addition (actually
tests should be executed on a per-commit level, not per patch-series
level).

>
> I think introducing fast path for certain changes is the easiest part of the
> equation.
>

That's at least something which can be implemented process-wise without
a major refactoring of the testsuite.

> In #2, you also express regret that there is no testcase for the
> feature in question. In general, we should think about adding testcases on use
> case introduction and bug fixes -- here I agree with Cedric.
>

I fully agree, but guidance is needed regarding how to write the tests
(e.g. to which tag to add them). Also, the test execution takes WAY too
long. We should consider allowing the use of the sstate cache and
download caching in the tests to significantly speedup the test
execution. That's also what Yocto is doing.

Regarding the test-execution itself: What we need is a simply CLI to
execute the tests in a container (this CLI should offer options to list
the testcases without executing them and executing just some tests).

> I think this has
> larger contribution to the quality, benefits all downstreams and saves the
> overall fixing effort, as every applied change could break the feature. Once a
> fix goes in, a testcase never follows. Currently, we are developing testcases
> for certain changes from the list; longer-term, this will not work.

Also agree, but we need to start somewhere. In the long term, I
envision test automation based on the ML, so whenever you send a patch
to the ML a set of possibly related tests is executed and the result is
sent back to the ML. For that, the cost of the test execution needs to
be reduced (e.g. executing a per-patch test run must not take longer
than 10 mins) and hardware needs to be setup.

>
> The Linux "tested by many" model helps only to some extent. We also see e.g.
> conflicting changes being applied back and forth, even in Linux. Having the use
> case view backed by tests at the Isar level would avoid this.

With syzcaller and alike being added to Linux, this model is anyways no
longer the only used one.

Best regards,
Felix

>
> To summarize, we'll evaluate this proposal and also invite for more
> collaboration on testcases.
>
> With kind regards,
> Baurzhan

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[MOESSBAUER, Felix]

On Thu, 2025-10-23 at 11:05 +0000, 'MOESSBAUER, Felix' via isar-users
wrote:

> On Thu, 2025-10-23 at 12:42 +0200, Baurzhan Ismagulov wrote:
> > Hello Jan,
> >
> > On 2025-10-23 07:52, Jan Kiszka wrote:
> > > Patches that carry "Fixes:" tag and address build breakages should be on
> > > a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
> > > almost on a daily basis, but those are just two recent examples.
> > >
> > > [1] https://patchwork.isar-build.org/project/isar/patch/20251017085344.26470...@siemens.com/
> > > [2] https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03...@siemens.com/
> >
> > I acknowledge the desire to have fast path for "Fixes:" and build breakages.
> > We've merged #2. The full results for #1 will be available tomorrow. There is a
> > number of failures due to e.g. external networking issues; we'll be sharing
> > patches for those.
>
> That's one of the core problems: The test suite itself is too unstable
> and also way too complicated to be executed on each addition (actually
> tests should be executed on a per-commit level, not per patch-series
> level).
>
> >
> > I think introducing fast path for certain changes is the easiest part of the
> > equation.
> >
>
> That's at least something which can be implemented process-wise without
> a major refactoring of the testsuite.

I again tried really hard to work with the testsuite, but this beast
has a will of its own...

I tried executing the tests with sstate cache enabled (via the
undocumented -p sstate=1 option), only to find out that my
tests sporadically fail due to bitbake server errors (do I have to run
with --max-parallel-task=1?). Once done that, the CI failed because of
missing disk space (in which mountpoint?) I still have plenty of space
in the build dir.

./kas/kas-container shell kas/isar.yaml --command \
"rm -rf /work/build/conf && /work/scripts/ci_setup.sh"
cd /work/testsuite
avocado run citest.py:CrossTest -p sstate=1

For me as a developer, the testsuite is currently not usable. I'm
willing to add tests again, once the infrastructure is ready for that.
But this requires upfront redesign.

Felix

[cedric.h...@siemens.com]

On Thu, 2025-10-23 at 11:05 +0000, 'MOESSBAUER, Felix' via isar-users
wrote:

Some weeks back, we have added some hints on how to start a container
where you may run the test-suite with all of its dependencies
installed:

kas/kas-container shell \
--command /work/isar/scripts/ci_setup.sh kas.yml

I think we can add a simple wrapper to make it even easier to invoke (I
always have to open the contributing guide to look it up).

I could modify the entry script to give a few hints such as:
- run "avocado list citest.py" to list existing tests
- use "avocado -p sstate=1 run citest.py" to use sstate (thanks Felix
for that hint!)
- use "avocado run citest.py:TestClass.test_method" to run a specific
test

As I am modifying some test cases to catch some of the recent
breakages, I am volunteering to enhance the documentation to get the
ball rolling

--
Cedric Hombourger
Siemens AG
www.siemens.com