Re: Arch qualification for buster: call for DSA, Security, toolchain concernsj

[Julien Cristau]

[s/debian-ports/debian-arm/]

On 06/29/2018 09:16 AM, Uwe Kleine-König wrote:
> Hello,
>
> On Wed, Jun 27, 2018 at 08:03:00PM +0000, Niels Thykier wrote:
>> armel/armhf:
>> ------------
>>
>> * Undesirable to keep the hardware running beyond 2020. armhf VM
>> support uncertain. (DSA)
>> - Source: [DSA Sprint report]
>>
>> [DSA Sprint report]:
>> https://lists.debian.org/debian-project/2018/02/msg00004.html
>
> In this report Julien Cristau wrote:
>
>> In short, the hardware (development boards) we're currently using to
>> build armel and armhf packages aren't up to our standards, and we
>> really, really want them to go away when stretch goes EOL (expected in
>> 2020). We urge arm porters to find a way to build armhf packages in
>> VMs or chroots on server-class arm64 hardware.
>
> If the concerns are mostly about the hardware not being rackable, there
> is a rackable NAS by Netgear:
>
> https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
>
> with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
> GiB) are good enough. The machine can run mainline Linux[1]. I think
> U-Boot doesn't support this machine in mainline though.
>
Rackable, while good, is only part of it. The main part is remote
management. I'm not seeing any mention of ipmi or anything like that in
the datasheet?

2G is also way too little memory these days for a new buildd.

Cheers,
Julien

[Uwe Kleine-König]

Hello Julien,

On 06/29/2018 11:23 AM, Julien Cristau wrote:
>> If the concerns are mostly about the hardware not being rackable, there
>> is a rackable NAS by Netgear:
>>
>> https://www.netgear.com/business/products/storage/readynas/RN2120.aspx#tab-techspecs
>>
>> with an armhf cpu. Not sure if cpu speed (1.2 GHz) and available RAM (2
>> GiB) are good enough. The machine can run mainline Linux[1]. I think
>> U-Boot doesn't support this machine in mainline though.
>>
> Rackable, while good, is only part of it. The main part is remote
> management. I'm not seeing any mention of ipmi or anything like that in
> the datasheet?

you can access the serial console, but I don't think there is built-in
support for something IPMI-like.

> 2G is also way too little memory these days for a new buildd.

Then the machine is out, the amount of RAM isn't upgradable.

Best regards
Uwe

[Roger Shimizu]

I don't think 2GB is not enough for 32-bit machine.

I see armel is already not a candidate for buster [0].
So it seems we can discuss armhf, but no armel at all.
I don't agree with this idea.
And I think we should treat armel and armhf equally.

Both armel and armhf are working fine are millions of boards and
embedded devices, and have stable quality [1].
They deserve the support from a community driven distro.

[0] https://release.debian.org/buster/arch_qualify.html
[1] https://lists.debian.org/debian-arm/2017/11/msg00061.html

Cheers,
--
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

[Samuel Thibault]

Roger Shimizu, le ven. 29 juin 2018 23:04:26 +0900, a ecrit:

> On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König
> <u...@kleine-koenig.org> wrote:
> > On 06/29/2018 11:23 AM, Julien Cristau wrote:
> >> 2G is also way too little memory these days for a new buildd.
> >
> > Then the machine is out, the amount of RAM isn't upgradable.
>
> I don't think 2GB is not enough for 32-bit machine.

I can say that 2GB is really not enough for a quite-non-small list
of packages. Sure you can add swap, but then e.g. link phases are
agonizingly long.

Samuel

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 3:28 PM, Samuel Thibault <sthi...@debian.org> wrote:

> Roger Shimizu, le ven. 29 juin 2018 23:04:26 +0900, a ecrit:
>> On Fri, Jun 29, 2018 at 10:04 PM, Uwe Kleine-König
>> <u...@kleine-koenig.org> wrote:
>> > On 06/29/2018 11:23 AM, Julien Cristau wrote:
>> >> 2G is also way too little memory these days for a new buildd.
>> >
>> > Then the machine is out, the amount of RAM isn't upgradable.
>>
>> I don't think 2GB is not enough for 32-bit machine.
>
> I can say that 2GB is really not enough for a quite-non-small list
> of packages.

and that list is only going to get inexorably and slowly bigger, to
the point where even on 64-bit systems at some point someone is going
to notice. 7GB (and climbing) of resident RAM for the linker phase on
firefox to keep it out of swap space *should* be ringing alarm bells
even for amd64 build maintainers.

> Sure you can add swap, but then e.g. link phases are
> agonizingly long.

sorry if it was not clear, and (to people who have read the analysis
already) apologies for repeating it for the third time, but this is
precisely and exactly why i said that it would be a good idea to
investigate adding "-Wl,--no-keep-memory" to the linker phase of
32-bit builds.

https://sourceware.org/bugzilla/show_bug.cgi?id=22831

linker phases 100% guaranteed to go into thrashing, due to the massive
amount of cross-referencing needed to be done in object-file linking
has been a known problem for over nine years, which nobody really
seems to understand or acknowledge or tackle.

l.

[Gene Heskett]

hijacking your thread, my apologies:

And I might add, on the likes of a pi-3b, demands that swap somehow be
migrated to spinning rust or an SSD, else from the descriptions on
binutils above this msg, will likely finish off the best of a u-sd card,
which its booting and running from now. Because I intend to build a new
rt-kernel for that pi, on that pi, the 60 GB SSD is already plugged in
but I've not yet moved swap. And it sounds like it will need far more
than 2GB, so I'm thinking at least 10 would be indicated based on the
binutils comments. Comments otherwise?

Thanks everybody.

--
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Jonathan Wiltshire]

On Fri, Jun 29, 2018 at 11:04:26PM +0900, Roger Shimizu wrote:
> I see armel is already not a candidate for buster [0].
> So it seems we can discuss armhf, but no armel at all.
> I don't agree with this idea.
> And I think we should treat armel and armhf equally.

Please review the mail which originated this thread [1] where you will see
that both armel and armhf are affected by DSA's concern. If I understand
correctly, virtualisation of architectures in general is a possible
solution but there are problems in the case of these two.

At the end of the day, if Debian can't reliably run builders for an
architecture we do not do users a service by pretending to be able to
support it in a formal release. A freeze may be for Christmas, but stable
is for at least five+ years.

--
Jonathan Wiltshire j...@debian.org
Debian Developer http://people.debian.org/~jmw

4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51

[Steve McIntyre]

Nod - lots of packages are just too big for that now.

So, here's a summary of the current build/porter machines we have for
the arm ports.

armel/armhf *only* build machines
=================================

We're mostly using dev boards to build 32-bit arm stuff yet.

* 7 Marvell Armada XP machines: dev boards supplied in a box, 4GiB
RAM - see http://photos.einval.com/gallery/2014_marvell_buildd
These are nice powerful little machines, but they're a PITA to
manage for power (won't cold boot without a button push) and serial
(USB serial exposed only). We have 6 of these as build boxes and 1
porter box, and I have a spare ready to go into service if
desired. They *don't* have NEON, so we also still have:

* 1 Freescale imx53 dev board as a porter box - old, slow Cortex A8
machine with only 1GiB of RAM. This works better for serial, but
remote power issues again - needs a button push to cold boot. Will
happily retire this once we have NEON available by default.

Each of these dev boards only has support for 1 disk, so disk failures
are painful.

arm64 build machines
====================

These are all more normal computers, with better support for remote
power and serial, DIMM slots, multiple disks, etc.

* APM Mustang (X-Gene 1): officially EOL, but working fine for
now. Normal server-class machine (although supplied in a small
desktop case!) with some onboard server management stuff. We
currently have one of these. We used to have more loaned/hosted by
Linaro, and I've had an offer of more of these if we're
interested. They'll build and run A32 (32-bit instruction set) as
well as A64.

* Gigabyte MP30-AR0 (X-Gene 1): server systems based on the Mustang
core - see
https://b2b.gigabyte.com/Server-Motherboard/MP30-AR0-rev-11#ov
Capable of building/running A32 and A64.

* AMD Seattle (Opteron A1100): officially EOL too, but working
fine. Same as the Softiron 3000, 2U rackmount case. Capable of
building/running A32 and A64. One of these has just been configured
to build armhf only.

Future options
==============

I understand DSA's reluctance to continue supporting dev boards as
build platforms - I've been the one working on some of these machines
in the machine room at Arm, and it's painful when you can't reliably
reboot or get onto the console of crashed machines. We've also had a
spate of disk failures recently which has caused extended downtime.
I'm just in the middle of switching the arm64 machines here to using
SW RAID to mitigate that in future, and that's just not an option on
the dev boards. We want to move away from dev boards for these
reasons, at the very least.

So, at the moment as far as I can see we're happy with our current
arm64 build machines. They are ageing, so obviously I'm continuing to
look out for new options there as well. *But* my priority is new
options for 32-bit building too. Following standard Debian practice,
we want to build *natively* (i.e. not using cross-building or using
hardware emulation). Building 32-bit code on a 64-bit platform should
not be an issue so long as the platform can also execute that 32-bit
code directly.

I am not aware of any 32-bit Arm *server* platforms shipping
today. Some have existed in the past (e.g. Calxeda), but almost
universally people have moved on to 64-bit now. The awkward thing that
is now becoming more common in the arm64 server world is that quite a
few of the vendors are not seeing any value in A32 support so they're
leaving it out of their CPUs. We'll need to be careful about that.

Options I can see today for new publically available machines are
here. I'm sure I'll have missed something obvious - please feel free
to improve on this list!

* Macchiatobin [1] - based on the Marvell 8040 SoC (4-core Cortex
A72), supports both A32 and A64. Standard format (mini-itx) board
mountable in a rack case. DIMM slot supports up to 16GiB, 3 SATA
ports, multiple onboard NICs. Supported in mainline upstream
kernel. Console on USB :-/. Readily available to buy.

* Synquacer [2] - based on the Socionext SC2A11 SoC (24-core Cortex
A53), supports both A32 and A64. Standard format (ATX) board
mountable in a rack case. DIMM slots supports up to 4x16GiB, 2 SATA
ports, onboard NIC. Supported in mainline upstream kernel. I'm
hoping to get some of these machines donated to us from Linaro.

* Qualcomm Centriq [3] - based on Qualcomm's Falkor CPU. Only
supports A64 - no A32 support. All the big features you'd want in a
big expensive server (management, RAM, I/O). Development machines
available, but difficult to get hold of for the general
public. Supported in mainline upstream kernel, some of it
backported for Stretch (9.5) in #896775.

* ThunderX 2 [4] - Cavium's second generation of AArch64 server
CPU. Only supports A64 - no A32 support. All the big features you'd
want in a big expensive server (management, RAM, I/O). Development machines
available, but difficult to get hold of for the general
public. Supported in mainline upstream kernel.

* HiSilicon D05 [5] - HiSilicon's latest AArch64 server CPU and
board. AFAIK only supports A64 - no A32 support. All the big
features you'd want in a big expensive server (management, RAM,
I/O). Development machines available, but difficult to get hold of
for the general public - I've been trying for a while with
HiSilicon. Not sure about upstream kernel support.

While they're on the lower end of this list, I think the Macchiatobin
and Synquacer are probably our best bets *at the moment*, particularly
when considering A32 support. In suitable rack cases with PDU and
serial console, would those work for DSA's needs?

[1] http://macchiatobin.net/
[2] https://www.cnx-software.com/2017/09/24/gigabyte-synquacer-96boards-enterprise-platform-is-powered-by-socionext-sc2a11-24-core-armv8-soc/)
[3] https://www.qualcomm.com/products/qualcomm-centriq-2400-processor
[4] https://www.cavium.com/product-thunderx2-arm-processors.html
[5] http://open-estuary.org/d05/

--
Steve McIntyre, Cambridge, UK. st...@einval.com
Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...

[John Paul Adrian Glaubitz]

> On Jun 29, 2018, at 9:50 AM, Jonathan Wiltshire <j...@debian.org> wrote:
>
>> On Fri, Jun 29, 2018 at 11:04:26PM +0900, Roger Shimizu wrote:
>> I see armel is already not a candidate for buster [0].
>> So it seems we can discuss armhf, but no armel at all.
>> I don't agree with this idea.
>> And I think we should treat armel and armhf equally.
>
> Please review the mail which originated this thread [1] where you will see
> that both armel and armhf are affected by DSA's concern. If I understand
> correctly, virtualisation of architectures in general is a possible
> solution but there are problems in the case of these two.

I have just talked to a colleague at SUSE about this and apparently Alex Graf from SUSE’s QEMU/KVM team has fixed many bugs regarding ARM32 on ARM64 virtualization. If I understand correctly, SUSE builds ARMv7 on ARM64 without problems.

I have reached out to Alex Graf and asked him for clarification on what possibilities we currently have.

Adrian

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 5:21 PM, Steve McIntyre <st...@einval.com> wrote:

>>2G is also way too little memory these days for a new buildd.
>
> Nod - lots of packages are just too big for that now.

apologies for repeating it again: this is why i'm recommending people
try "-Wl,--no-keep-memory" on the linker phase as if it works as
intended it will almost certainly drastically reduce memory usage to
the point where it will stay, for the majority of packages, well
within the 2GB limit i.e. within resident RAM.

i'm really not sure why the discussions continue not to take this
into account, repeating the status-quo and accepting "packages are too
big" as if there is absolutely no possible way that this problem may
be solved or even attempted to be solved... ever. i am very confused
by this. perhaps it is down to latency in discussions as new people
contribute (but have signficant delay on reading), i don't know.

> Future options
> ==============
>
> I understand DSA's reluctance to continue supporting dev boards as
> build platforms - I've been the one working on some of these machines
> in the machine room at Arm, and it's painful when you can't reliably
> reboot or get onto the console of crashed machines. We've also had a
> spate of disk failures recently which has caused extended downtime.

that is not a surprise to hear: the massive thrashing caused by the
linker phase not being possible to be RAM-resident will be absolutely
hammering the drives beyond reasonable wear-and-tear limits. which is
why i'm recommending people try "-Wl,--no-keep-memory".

... oh, i have an idea which people might like to consider trying.
it's to use "-Wl,--no-keep-memory" on the linker phase of 32-bit
builds. did i mention that already? :) it might save some build
hardware from being destroyed if people try using
"-Wl,--no-keep-memory"!

> I'm just in the middle of switching the arm64 machines here to using
> SW RAID to mitigate that in future, and that's just not an option on
> the dev boards. We want to move away from dev boards for these
> reasons, at the very least.

most of them won't have native SATA: very few 32-bit ARM systems do.
GbE is not that common either (so decent-speed network drives are
challenging, as well). so they'll almost certainly be USB-based
(USB-to-SATA, which is known-unreliable), and putting such vast
amounts of drive-hammering through USB-to-SATA due to thrashing isn't
going to help :)

the allwinner A20 and R40 are the two low-cost ARM systems that i'm
aware of that have native SATA.


there is however a new devboard that is reasonably cheap and should
be available really soon: the Rock64Pro (not to be confused with the
Rock64, which does NOT have PCie), from pine64:
https://www.pine64.org/?page_id=61454

it's one of the first *low-cost* ARM dev-boards that i've seen which
has 4GB of RAM and has a 4x PCIe slot. the team have tested it out
with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to
hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec).

also worth noting, they're working on a 2U rackmount server which
will have i think something insane like 48 Rock64Pro boards in one
full-length case.

the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core
CortexA72 for a total 6-core SMP system, all 64-bit.

if anyone would like me to have a word with TL Lim (the CEO of
pine64) i can see if he is willing and able to donate some Rock64Pro
boards to the debian farm, let me know.

l.

[Jonathan Wiltshire]

On Fri, Jun 29, 2018 at 06:05:55PM +0100, Luke Kenneth Casson Leighton wrote:
> apologies for repeating it again: this is why i'm recommending people
> try "-Wl,--no-keep-memory" on the linker phase as if it works as
> intended it will almost certainly drastically reduce memory usage to
> the point where it will stay, for the majority of packages, well
> within the 2GB limit i.e. within resident RAM.

[...]

> most of them won't have native SATA: very few 32-bit ARM systems do.
> GbE is not that common either (so decent-speed network drives are
> challenging, as well). so they'll almost certainly be USB-based
> (USB-to-SATA, which is known-unreliable), and putting such vast
> amounts of drive-hammering through USB-to-SATA due to thrashing isn't
> going to help :)

[...]

>
> the allwinner A20 and R40 are the two low-cost ARM systems that i'm
> aware of that have native SATA.
>
>
> there is however a new devboard that is reasonably cheap and should
> be available really soon: the Rock64Pro (not to be confused with the
> Rock64, which does NOT have PCie), from pine64:
> https://www.pine64.org/?page_id=61454
>
> it's one of the first *low-cost* ARM dev-boards that i've seen which
> has 4GB of RAM and has a 4x PCIe slot. the team have tested it out
> with an NVMe SSD and also 4x SATA PCIe cards: they easily managed to
> hit 20 Gigabits per second on the NVMe drive (2500 mbytes/sec).
>
> also worth noting, they're working on a 2U rackmount server which
> will have i think something insane like 48 Rock64Pro boards in one
> full-length case.
>
> the Rock64Pro uses the RK3399 which is a 4-core CortexA53 plus 2-core
> CortexA72 for a total 6-core SMP system, all 64-bit.
>
> if anyone would like me to have a word with TL Lim (the CEO of
> pine64) i can see if he is willing and able to donate some Rock64Pro
> boards to the debian farm, let me know.

None of this addresses the basic DSA requirement of remote management.
Troubling local hands to change a disk once in a while is reasonable; being
blocked waiting for a power cycle on a regular basis is not (and I can't
imagine hosting sponsors are wild about their employees' time being used
for that either).

Development boards just don't cut it any longer.

[Luke Kenneth Casson Leighton]

On Fri, Jun 29, 2018 at 6:59 PM, Jonathan Wiltshire <j...@debian.org> wrote:

>> also worth noting, they're working on a 2U rackmount server which
>> will have i think something insane like 48 Rock64Pro boards in one
>> full-length case.

> None of this addresses the basic DSA requirement of remote management.
> Troubling local hands to change a disk once in a while is reasonable; being
> blocked waiting for a power cycle on a regular basis is not (and I can't
> imagine hosting sponsors are wild about their employees' time being used
> for that either).

i know exactly what you mean, i've had to deal with data centres.
i'll make sure that TL Lim is aware of this, and will ask him if
there's a way to include remote power-management / power-cycling of
boards in the planned product or if it's already been thought of.

l.

[Florian Weimer]

* Luke Kenneth Casson Leighton:

> that is not a surprise to hear: the massive thrashing caused by the
> linker phase not being possible to be RAM-resident will be absolutely
> hammering the drives beyond reasonable wear-and-tear limits. which is
> why i'm recommending people try "-Wl,--no-keep-memory".

Note that ld will sometimes stuff everything into a single RWX segment
as a result, which is not desirable.

Unfortunately, without significant investment into historic linker
technologies (with external sorting and that kind of stuff), I don't
think it is viable to build 32-bit software natively in the near
future. Maybe next year only a few packages will need exceptions, but
the number will grow with each month. Building on 64-bit kernels will
delay the inevitable because more address space is available to user
space, but that's probably 12 to 18 month extended life-time for
native building.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Fri, Jun 29, 2018 at 8:31 PM, Florian Weimer <f...@deneb.enyo.de> wrote:
> * Luke Kenneth Casson Leighton:
>
>> that is not a surprise to hear: the massive thrashing caused by the
>> linker phase not being possible to be RAM-resident will be absolutely
>> hammering the drives beyond reasonable wear-and-tear limits. which is
>> why i'm recommending people try "-Wl,--no-keep-memory".
>
> Note that ld will sometimes stuff everything into a single RWX segment
> as a result, which is not desirable.

florian, thank you for responding: i've put a copy of the insights
that you give into the bugreport at
https://sourceware.org/bugzilla/show_bug.cgi?id=22831#c16

> Unfortunately, without significant investment into historic linker
> technologies (with external sorting and that kind of stuff),

yes, ah ha! funnily enough the algorithm that i was asked to create
back in 1988 was an external matrix-multiply, i take it you are
talking about the same thing, where linking is done using explicit
load-process-save cycles rather than relying on swap.

> I don't
> think it is viable to build 32-bit software natively in the near
> future.

i noted an alternative strategy in the bugreport: if binutils *at
the very least* were to look at the available resident RAM and only
malloc'd and used up to that amount, and kept only a few (or even just
one) object file in memory at a time and did all the linking for that,
it would be infinitely better than the current situation which is
*only going to get worse*.

> Maybe next year only a few packages will need exceptions, but
> the number will grow with each month.

well... that ignores the fact that at some point in the next few
years there will be a package that needs 16 GB of resident RAM to
link. and a few years after that it will be 32 GB. and that's on
64-bit systems. the package's name will probably be "firefox", given
the current growth rate.

does debian *really* want to have to upgrade all 64-bit systems in
the build farm first to 16 GB RAM and then to 32 GB and then to 64
GB?? can the powerpc64 systems and all other 64-bit architectures
even *be* upgraded to 16 GB then 32 GB then 64 GB of RAM??

basically the problems faced by 32-bit systems are a warning shot
across the bows about ld not really being kept up-to-date with the
increases in software complexity that's being thrown at it. it's
*NOT* just about 32-bit.

this problem can basically be faced REactively... or it can be faced
PROactively: the historic linker strategies that you mention are i
feel going to be needed in some years' time *even for 64-bit*.

l.

[Luke Kenneth Casson Leighton]

TL informs me that all the power and reset signals for all 48 of the
RockPro64s tucked into the full-length 2U case are brought out to the
back panel. an MCU (or MCUs) or SBC (or SBCs) may therefore be
connected directly to those in order to provide *individual* remote
power / reset management of all 48 RockPro64s. DIY remote power
management, but it is actual remote power management.

l.

[Luke Kenneth Casson Leighton]

spoke again to TL and asked if pine64 would be willing to look at
sponsorship witn rockpro64 boards (the ones that take 4x PCIe): if
someone from debian were to contact him direct he would happily
consider it.

i then asked him if i could cc him into this discussion and he said he
was way *way* too busy to enter into another mailing list discussion,
so if someone from debian emails me privately, off-list, i will then
cc him and/or put them in touch with him on irc. i can also be
reached on freenode and oftc as "lkcl", if that is easier.

l.