EL7 Packages and Build Test Failures

[Gordan Bobic]

Quick question, mainly aimed at Jacco - how many packages in the el7 set did you have to build with --nocheck ?

The reason I ask is because I'm seeing quite a lot of things that built successfully but didn't pass the %check stage.

My 1st pass ss7 build is about 50% complete. Annoyingly, most of the process runs single threaded most of the time because various extremely long running parts don't benefit from multiple cores. I'll make 8 separate mock configs for the 2nd pass, to build all of the packages that failed in the 1st pass.

I'm also outright depressed with just how much "throw it over the wall and walk away" attitude there is upstream. Not only do various packages' configure/build scripts insist on auto-detecting CPU capabilities and using those instead of just doing as they are damn well told by the CFLAGS, there is also outright CFLAGS overriding going on, e.g. the rpm passed CFLAGS are set first followed by the package set CFLAGS. That just seems outright broken because the latter set of options should override, and in fact in some packages there are situations where something really messed up happens with things like -march=armv5tel being passed along with -mcpu=cortex-a8 which is clearly incompatible (and gcc emits a warning with words to that effect.

I am debating whether to even bother filing what is likely to be hundreds of bugs upstream about it just to make a point, because I expect they will all be closed as "won't fix, builds on my x86-64 machine and the rest of the world can burn for all I care".

It makes me wonder if anyone upstream takes any pride at all in their work. We can't be the first ones to discover these bugs, the c7 armv7hl guys must have also seen most of them and just ignored them (part of the reason why CentOS 7 doesn't work properly on ARMv7 Tegra).

/rant

[Jacco Ligthart]

On 2019-09-20 23:40, Gordan Bobic wrote:
> Quick question, mainly aimed at Jacco - how many packages in the el7
> set did you have to build with --nocheck ?

All should be documented here:
https://github.com/redsleeve-linux/el7/blob/master/README.md

Yes I know what you mean, I also found dozens of issues.

What is also very annoying is, that sometimes fixes are only implemented
in the x86 tree. So is bolt 0.7 not buildable because the
'copy_file_range' syscall is not implemented for arm. This syscall has
been backported into RHEL7, but only for the x86 tree.


Jacco

[Fred Gleason]

On Sep 20, 2019, at 17:40, Gordan Bobic <gor...@redsleeve.org> wrote:

I'm also outright depressed with just how much "throw it over the wall and walk away" attitude there is upstream. Not only do various packages' configure/build scripts insist on auto-detecting CPU capabilities and using those instead of just doing as they are damn well told by the CFLAGS, there is also outright CFLAGS overriding going on, e.g. the rpm passed CFLAGS are set first followed by the package set CFLAGS. That just seems outright broken because the latter set of options should override, and in fact in some packages there are situations where something really messed up happens with things like -march=armv5tel being passed along with -mcpu=cortex-a8 which is clearly incompatible (and gcc emits a warning with words to that effect.

While there is doubtless some of that attitude out there, I suspect that much more of it is simple ignorance. Many of us on the application development side came to Linux at a time when the market had pretty much flattened out the available hardware choices to two: x86 (cheap, ubiquitous and ‘Windows compatible') and PowerPC (available, but ‘Apple based’ and hence expensive). Thus, the art of crafting a build system to be able to deal gracefully with multiple architectures (not to mention cross-compiling toolchains!) was not deeply bred in the bone. I know that this was the case with myself. The advent of inexpensive ARM boards like the BeagleBone and the Raspberry Pi led to a revelation to me in this regard.

I am debating whether to even bother filing what is likely to be hundreds of bugs upstream about it just to make a point, because I expect they will all be closed as "won't fix, builds on my x86-64 machine and the rest of the world can burn for all I care”.

It might be well to remember that these are Open Source projects; presumably, a great many of the maintainers of which are motivated by goals beyond the simple “it’s just a job”. While filing hundreds of bug reports does indeed sound like a daunting task, would it perhaps be practical to work up a short guide or FAQ delineating the most common problems and “gotchas” found when attempting to cross-compile packages and how to avoid them? I for one would find such a thing quite useful.

Cheers!

|---------------------------------------------------------------------|

| Frederick F. Gleason, Jr. |             Chief Developer             |

|                           |             Paravel Systems             |

|---------------------------------------------------------------------|

|  ... indifference is a militant thing ... when it goes away it      |

|  leaves smoking ruins, where lie citizens bayonetted through the    |

|  throat. It is not a children's pastime like mere highway robbery.  |

|                                                                     |

|                                                  -- Stephen Crane   |

|---------------------------------------------------------------------|

[Gordan Bobic]

On Fri, 20 Sep 2019, 23:56 Fred Gleason, <fr...@paravelsystems.com> wrote:

On Sep 20, 2019, at 17:40, Gordan Bobic <gor...@redsleeve.org> wrote:

I'm also outright depressed with just how much "throw it over the wall and walk away" attitude there is upstream. Not only do various packages' configure/build scripts insist on auto-detecting CPU capabilities and using those instead of just doing as they are damn well told by the CFLAGS, there is also outright CFLAGS overriding going on, e.g. the rpm passed CFLAGS are set first followed by the package set CFLAGS. That just seems outright broken because the latter set of options should override, and in fact in some packages there are situations where something really messed up happens with things like -march=armv5tel being passed along with -mcpu=cortex-a8 which is clearly incompatible (and gcc emits a warning with words to that effect.

While there is doubtless some of that attitude out there, I suspect that much more of it is simple ignorance. Many of us on the application development side came to Linux at a time when the market had pretty much flattened out the available hardware choices to two: x86 (cheap, ubiquitous and ‘Windows compatible') and PowerPC (available, but ‘Apple based’ and hence expensive). Thus, the art of crafting a build system to be able to deal gracefully with multiple architectures (not to mention cross-compiling toolchains!) was not deeply bred in the bone. I know that this was the case with myself. The advent of inexpensive ARM boards like the BeagleBone and the Raspberry Pi led to a revelation to me in this regard.

Well, CFLAGS ordering is arguably a distro packaging issue. Then again maybe the distro package maintainer did it as a bodge because the upstream package maintainer bodged it with certain compiler parameters to work around a bug in the version of GCC they were building with at the time, etc.

It's often a workaround for a bodge, implemented due to bug in a dependency, that spontaneously turns into a landmine that I just happen to be treading on now, years later.

But that doesn't make it any less infuriating.

I am debating whether to even bother filing what is likely to be hundreds of bugs upstream about it just to make a point, because I expect they will all be closed as "won't fix, builds on my x86-64 machine and the rest of the world can burn for all I care”.

It might be well to remember that these are Open Source projects; presumably, a great many of the maintainers of which are motivated by goals beyond the simple “it’s just a job”.

When it comes to RH, that's questionable. See the link to a bugzilla I posted in the other message on this thread. When the distro package maintainer won't accept your bugfixes out of pure lazyness, it is difficult to be an apologist for that kind of behaviour.

It's not the only example either. I filed a LOT of bugs on RHBZ in my time, and a large fraction of then were "won't fix" closed. Eventually I stopped bothering.

While filing hundreds of bug reports does indeed sound like a daunting task, would it perhaps be practical to work up a short guide or FAQ delineating the most common problems and “gotchas” found when attempting to cross-compile packages and how to avoid them? I for one would find such a thing quite useful.

I am not technically cross compiling. I am building using mock, which creates a suitable arch chroot (in this case armv5tel). It just happens to all be running on a fairly obscene spec aarch64 box (Gigabyte MP30-AR0 with 128GB of RAM). A lot of this might not manifest if building on an ARMv5 machine, but my DreamPlugs would take days to do what the aarch64 box achieves in minutes. And it would still be a bug, even if it didn't manifest under certain circumstances.

[Fred Gleason]

On Sep 20, 2019, at 19:32, Gordan Bobic <gor...@redsleeve.org> wrote:

When it comes to RH, that's questionable. See the link to a bugzilla I posted in the other message on this thread. When the distro package maintainer won't accept your bugfixes out of pure lazyness, it is difficult to be an apologist for that kind of behaviour.

It's not the only example either. I filed a LOT of bugs on RHBZ in my time, and a large fraction of then were "won't fix" closed. Eventually I stopped bothering.

Yes, I’ve run into this attitude when reporting issues to RH as well. In that case, it was an upstream maintainer who was also employed by RH to manage the downstream integration. Unfortunately, it can be easy in those situations to allow the narrower interests of the downstream port to prevail over one’s responsibility to the larger project.

I am not technically cross compiling. I am building using mock, which creates a suitable arch chroot (in this case armv5tel). It just happens to all be running on a fairly obscene spec aarch64 box (Gigabyte MP30-AR0 with 128GB of RAM). A lot of this might not manifest if building on an ARMv5 machine, but my DreamPlugs would take days to do what the aarch64 box achieves in minutes. And it would still be a bug, even if it didn't manifest under certain circumstances.

Gotcha. Thanks for the clarification!

As someone whose ARM ‘build farm’ consists of an ASUS TinkerBoard running RS7 (armv5tel), all I can say is that I’m jealous!  :)

Cheers!

|---------------------------------------------------------------------|

| Frederick F. Gleason, Jr. |             Chief Developer             |

|                           |             Paravel Systems             |

|---------------------------------------------------------------------|

|   Take care of the molehills, and the mountains will take care of   |

|   themselves.                                                       |

|                                                                     |

|                             -- Motto of the Federal Civil Service   |

|---------------------------------------------------------------------|