Hi Everybody,
On 22.01.25 11:00, 'Michael Adler' via swupdate wrote:
> Hi all,
>
>> Agreed. If I can help in any way, let me know.
>>
>> Could we add a testing phase 14 days before the next regular release?
>
> Just to share my thoughts - considering the severity of the bug and given our
> new GitHub CI setup, I believe it would be useful to have an automated test
> case for this. Perhaps not a unit test, but rather a "real world" test case.
>
This is not unknown and I have already raised the issue. It is
documented in the improvements section, see:
https://github.com/sbabic/swupdate/blob/master/doc/source/improvement_proposals.rst#test-and-continuous-integration
In the past, I took time to test on the projects that I had at the time
of the release. Not everything can be checked. But I was mostly the only
contributor. With increasing number of patches, that is of course a good
thing for the project, this does not match anymore. In last two release
I mostly relied on external testers, and patches were merged on master
with low test from my site, and with the hope that something is reported
before the official release. This does not work, too, because everybody
is working on projects with timelines, etc., and the next SWUpdate
release is not urgent. 2023.12 needed to be patched, 2024.05 twice,
2024.12 is needed, too.
The github integration, recently introduced, is more for users able to
push on their branches / users. As stated before, Github is a
proprietary platform and they provide a max amount of free minutes for
the runners. And Github is of course right, I am using their resources.
Nothing to say against. I am not sure how minutes are counted, as after
just pushing few commits, 84 minutes were already consumed. So this
probably does not scale by increasing tests etc., but there are other
reasons.
What I would like to have is a small factory, with some real hardware
where new release is actively tested. And via gitlab-ci and test
frameworks, everything could be automatized. Bandwidth can be optimiyed
as factory and server resides on the server room, and more tests can be
executed. What I think it is good, is to have enough hardware to test at
least these configurations:
Architectures:
- Intel x86
- ARM
- ARM64
Boot:
- U-Boot (of course)
- Grub
- efiBootGuard
Storage:
- SSD / disks
- Raw NAND + UBI
- Raw NOR Flash
And both single-copy and A/B should be automatically tested. As reported
in my previous answer to Mark, pipelines are currently not accessible
outside. I have no problem to make them public to see results, as I
always run them before pushing to the official repo. But I do not want
to make possible that everyone can push and start runners, as this is
resource hungry.
So this CI with real hardware requires some investition, and I can state
here that I have already tried this year with one of public fund
subsidized by Germany in case of FOSS projects with public utility (I
think SWUpdate is). Answer will be later in second quarter 2025. If this
happens, I will have resources to implement it.
I should maybe raise the priority of this item from medium to high in
the improvements file, so that I make clearer to more users its importance.
Best regards,
Stefano