Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

How to self-load non-freeware firmware on existing netinst ISO installer

189 views
Skip to first unread message

Robbi Nespu

unread,
Feb 24, 2021, 10:10:04 AM2/24/21
to
I have another laptop which dual boot with Fedora and Windows10, it is not my primary laptop since 9 months ago and I haven't used it since then. I plan to fully install Debian testing on this machine because this laptop has dual graphic cards (optimus) and theoretically, with Debian 11 (or newer), special configuration shouldn't be needed and offloading should be available as soon as you've installed the proprietary drivers[1]. This machine has a faulty LAN port and only 1 of 3 USB ports are usable. So I can only use 1 USB per time and only WIFI for the network connection.

Since Debian provides an unofficial netinst image for i386/amd64/powerpc with the non-free firmware, I go and download the ISO[2] and make a bootable USB installer.

But when I during the installation process (on "detect network hardware" phase),   debian-installer dialog  asked me to load 2 firmware which is iwIwifi-2030-6.ucode and  iwIwifi-2030-5.ucode  .. I feel perplexed about this. So I checked the USB, it already have the firmware but with deb package format:

/firmware/firmware-iwlwifi_20201218-3_all.deb
/pool/non-free/f/firmware-nonfree/firmware-iwlwifi_20201218-3_all.deb

Seems this firmware load on the next stage which is after network setup and after do the partitioning stuff. I wonder why, since this is netinst I need network working from the during installation process, not after that.

Question :
a) Is it possible to do self modification on the installation script so the wifi chip will be available and working during the installation process so I can pull the packages that I want during installation from Debian mirror. If yes, how and if no, please suggest me some solutions that don't need me to fix LAN port, buy cable and router (since I use wifi hotspot from my phone)

The precompile firmware are available here[3] and here[4]

TLDR; I want  firmware-iwlwifi  already loaded and working during Debian installation phase, not after install.

[1] https://wiki.debian.org/NVIDIA%20Optimus#PRIMEOffload
[2] http://cdimage.debian.org/cdimage/unofficial/non-free/images-including-firmware/weekly-builds/amd64/iso-cd/firmware-testing-amd64-netinst.iso
[3] https://wireless.wiki.kernel.org/_media/en/users/drivers/iwlwifi-2030-ucode-18.168.6.1.tgz
[4] https://github.com/OpenELEC/iwlwifi-firmware/tree/master/firmware

Kenneth Parker

unread,
Feb 24, 2021, 10:40:05 AM2/24/21
to


On Wed, Feb 24, 2021, 10:09 AM Robbi Nespu <robbi...@gmail.com> wrote:
I have another laptop which dual boot with Fedora and Windows10, it is not my primary laptop since 9 months ago and I haven't used it since then. I plan to fully install Debian testing on this machine because this laptop has dual graphic cards (optimus) and theoretically, with Debian 11 (or newer), special configuration shouldn't be needed and offloading should be available as soon as you've installed the proprietary drivers[1]. This machine has a faulty LAN port and only 1 of 3 USB ports are usable. So I can only use 1 USB per time and only WIFI for the network connection.

Since Debian provides an unofficial netinst image for i386/amd64/powerpc with the non-free firmware, I go and download the ISO[2] and make a bootable USB installer.

But when I during the installation process (on "detect network hardware" phase),   debian-installer dialog  asked me to load 2 firmware which is iwIwifi-2030-6.ucode and  iwIwifi-2030-5.ucode  .. I feel perplexed about this. So I checked the USB, it already have the firmware but with deb package format:

/firmware/firmware-iwlwifi_20201218-3_all.deb
/pool/non-free/f/firmware-nonfree/firmware-iwlwifi_20201218-3_all.deb

Seems this firmware load on the next stage which is after network setup and after do the partitioning stuff. I wonder why, since this is netinst I need network working from the during installation process, not after that.

Question :
a) Is it possible to do self modification on the installation script so the wifi chip will be available and working during the installation process so I can pull the packages that I want during installation from Debian mirror. If yes, how and if no, please suggest me some solutions that don't need me to fix LAN port, buy cable and router (since I use wifi hotspot from my phone)

When I had a situation like that, my workaround was to install without the Network (but with the netinst CD), stopping at the "minimal system". It actually gives you a bootable system.  And then, use the external media to get WiFi working, and go from there. 
Kenneth Parker 

Brian

unread,
Feb 24, 2021, 11:10:05 AM2/24/21
to
On Wed 24 Feb 2021 at 22:51:05 +0800, Robbi Nespu wrote:

[...]

> TLDR; I want firmware-iwlwifi already loaded and working during Debian
> installation phase, not after install.

This is from memory; I haven't done it for some time.

1. The USB stick you boot from will have empty space or a secomd
partition.

2. Extract the firmware files from the .deb and put them on the
stick.

3. Boot and change to console 2: ALT-F2.

4. Mount the partition holding the firmware on /mnt.

5. Create /lib/firmware: mkdir /lib/firmware and transfer the
firmware there.

6. ALT-F1 to go back to d-i. d-i should now find the firmware.

--
Brian.

John Boxall

unread,
Feb 24, 2021, 11:20:05 AM2/24/21
to
Alternatively, you can extract the firmware files to a different USB
stick and put them in the root of that one, insert both and the
installer will find the files when you boot the original USB stick.
--
Regards,

John Boxall

Brian

unread,
Feb 24, 2021, 11:30:05 AM2/24/21
to
Indeed. However, the problem (unless I am misunderstanding) is that
the firmware is not even being found on the non-free netinstall,
which is the whole point of having that image available.

--
Brian.

Robbi Nespu

unread,
Feb 24, 2021, 9:50:05 PM2/24/21
to
On Wed, 24 Feb 2021 10:39:16 -0500 Kenneth Parker <sea7...@gmail.com>
wrote:
> When I had a situation like that, my workaround was to install without the Network (but with the netinst CD), stopping at the "minimal system". It actually gives you a bootable system. And then, use the external media to get WiFi working, and go from there.

I also thinking to go this way but there could be another better
solution to load the firmware at the beginning of installation (which is
the most prefer solution) since I have ucode precompiled binary from
Intel with me, let use it but I don't know how modify the installer or
how to re-create the installer.

--
Email : Robbi Nespu <robbinespu AT SPAMFREE gmail DOT com>
PGP fingerprint : D311 B5FF EEE6 0BE8 9C91 FA9E 0C81 FA30 3B3A 80BA
PGP key : https://keybase.io/robbinespu/pgp_keys.asc

Robbi Nespu

unread,
Feb 24, 2021, 10:00:04 PM2/24/21
to
On Wed, 24 Feb 2021 11:14:59 -0500, John Boxall <jbox...@gmail.com> wrote:
> Alternatively, you can extract the firmware files to a different USB stick and put them in the root of that one, insert both and the installer will find the files when you boot the original USB stick.

To bad, this machine only have 1 working USB port. Plan to fix the other
2 USB port too but currently I can't cross district travel since
"lock-down 2.0" in Malaysia. The USB port are faulty causes by USB fans..

Robbi Nespu

unread,
Feb 24, 2021, 10:00:04 PM2/24/21
to
On Wed, 24 Feb 2021 16:07:09 +0000, Brian <ad...@cityscape.co.uk> wrote:
> his is from memory; I haven't done it for some time.
>
> 1. The USB stick you boot from will have empty space or a secomd
> partition.
>
There only one partition but there is lot of empty space

> 2. Extract the firmware files from the .deb and put them on the
> stick.
>
It doesn't matter where I put right (on root location)? or need to have
special directory?

> 3. Boot and change to console 2: ALT-F2.
>
> 4. Mount the partition holding the firmware on /mnt.
>
This is tricky part as I mentioned, I only have one partition. Where I
can find my USB device location? try with `dh` command but that command
not working/available

> 5. Create /lib/firmware: mkdir /lib/firmware and transfer the
> firmware there.
>
> 6. ALT-F1 to go back to d-i. d-i should now find the firmware.
You suggestion look like the most closest one, just need to find the
perfect way how to accomplish it

Robbi Nespu

unread,
Feb 24, 2021, 10:40:05 PM2/24/21
to
On Wed, 24 Feb 2021 16:28:33 +0000, Brian <ad...@cityscape.co.uk> wrote:
> Indeed. However, the problem (unless I am misunderstanding) is that
> the firmware is not even being found on the non-free netinstall,
> which is the whole point of having that image available.

Yes, the firmware not even being found on certain stages of installation
process, it will be found later and available after install but it quite
late since I need to "tick" certain package to be install over network.

David Wright

unread,
Feb 24, 2021, 11:50:04 PM2/24/21
to
On Thu 25 Feb 2021 at 10:36:40 (+0800), Robbi Nespu wrote:
> On Wed, 24 Feb 2021 16:07:09 +0000, Brian <ad...@cityscape.co.uk> wrote:
> > [T]his is from memory; I haven't done it for some time.
> >
> > 1. The USB stick you boot from will have empty space or a secomd
> > partition.
> >
> There only one partition but there is lot of empty space

Take care: this stick will have very strange partitioning.
This is how fdisk sees a buster stick on this system:

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 0 774143 774144 378M 0 Empty
/dev/sdb2 4048 9711 5664 2.8M ef EFI (FAT-12/16/32)

Fdisk can create a partition for you to use with the commands:
(defaults in parentheses)

# fdisk /dev/sdX
p
n
(p)
(3)
(0000000) accept the defaults. The actual numbers may vary
(0000000) from week to week with different versions.
t
(3)
c
p
w

Don't just type them blindly: read and take account of the
responses from fdisk.

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 0 774143 774144 378M 0 Empty
/dev/sdb2 4048 9711 5664 2.8M ef EFI (FAT-12/16/32)
/dev/sdb3 786432 1998847 1212416 592M c W95 FAT32 (LBA)

Now you can create the filesystem:

# mkdosfs -v -i 12345670 -F 32 -n mypartition /dev/sdX3

> > 2. Extract the firmware files from the .deb and put them on the
> > stick.

Mount it

# mount /dev/sdX3 /mnt

Copy the firmware from wherever you have it

# cp -ip /lib/firmware/iwlwifi-2030-6.ucode /mnt

# umount /mnt

> It doesn't matter where I put right (on root location)? or need to
> have special directory?

Root is best. IIRC the installer looks widely for partitions,
but not to any depth in each.

> > 3. Boot and change to console 2: ALT-F2.
> >
> > 4. Mount the partition holding the firmware on /mnt.

Type:

# mount -t vfat /dev/

and press TAB twice. You'll get a listing of everything in /dev.
Usually sticks appear as /dev/sd… so type s and press TAB twice.
You should be able to distinguish the stick from any other drives.
So you end up with, say:

# mount -t vfat /dev/sdb3 /mnt

Check by typing:

# ls -l /mnt
-rwxr-xr-x 1 root root 707392 Aug 22 2019 iwlwifi-2030-6.ucode

That would confirm the correct partition.

> > 5. Create /lib/firmware: mkdir /lib/firmware and transfer the
> > firmware there.
> >
> > 6. ALT-F1 to go back to d-i. d-i should now find the firmware.

Note that on most systems, steps 3 through 6 are unnecessary as
the installer will find the firmware itself anyway.

If you can find iwlwifi-2030-5.ucode, by all means copy it onto
the stick as well as iwlwifi-2030-6.ucode. The latter is at least
8 years old, and can probably do all that -5 ever could. Typically
the installer will try loading each version in reverse order
(newest first). (It's IWLWIFI, or course.)

Cheers,
David.

Robbi Nespu

unread,
Feb 25, 2021, 5:20:05 AM2/25/21
to
I found out a nice solution (thanks David Wright for the hint!), simply:

1. download and save the firmware to root of our debian installer USB
2. Boot USB
3. Press Ctrl+Alt+F2 to spawn busybox terminal
4. execute this command

# /bin/mountmedia
# cp -ir /media/YOUR_FIRMWARE_FILE_HERE.ucode /lib/firmware
# /bin/unmount
# exit

5. Press Ctrl+Alt+F5 to continue GUI installation..and surprise! Debian
installer driver will load the firmware, so you can use WIFI and
continue installation process

Done!

P/s : Thus "unmount" script is to release our USB media and let debian
installer mount it later automatically, if you forgot to umount, this
trick won't worked. All of this script is standard, no need to customize
the installer.

Brian

unread,
Feb 25, 2021, 9:40:05 AM2/25/21
to
On Wed 24 Feb 2021 at 22:47:59 -0600, David Wright wrote:

> On Thu 25 Feb 2021 at 10:36:40 (+0800), Robbi Nespu wrote:
>
> > > 5. Create /lib/firmware: mkdir /lib/firmware and transfer the
> > > firmware there.
> > >
> > > 6. ALT-F1 to go back to d-i. d-i should now find the firmware.
>
> Note that on most systems, steps 3 through 6 are unnecessary as
> the installer will find the firmware itself anyway.

That's the advice given in Section 6.4.1 if the Installer Guide and,
if it works, it works. However, I would not like to guarantee that it
does, just as, in the case of the OP, the non-free installer ISO does
not come up with installing the firmware.

Transferring the firmware files directly to where the installer kernel
can find them seems more assured of success. If it doesn't, there is a
big problem for the system both during and after installation.

--
Brian.

Semih Ozlem

unread,
Feb 26, 2021, 12:50:05 AM2/26/21
to
You can get a usb hub to be able to have multiple usbs connected from one port. Not sure if that would solve all the problems, but it may help.

Robbi Nespu <robbi...@gmail.com>, 24 Şub 2021 Çar, 18:09 tarihinde şunu yazdı:

Brian

unread,
Feb 26, 2021, 8:10:09 AM2/26/21
to
Inspired by David Wright's and Robbi Nespu's contributions I adapted
my six-step advice to take advantage of preseeding. The needed firmware
files are assumed to be available in a directory named firmware on a
USB stick partition labelled FIRMWARE.

The following addition is made to the installer's kernel command line.
Press TAB when theinstallation choice is highlighted to make the line
visible.

(The line is broken for readability).

preseed/early_command="modprobe vfat ; sleep 2 ; \
mount /dev/disk/by-label/FIRMWARE /media ; \
cp -a /media/firmware /lib"

Firmware blob location and loading is put with the installer kernel
rather than have the installer, sometimes unreliably, locating and
dealing with the files.

--
Brian.

David Wright

unread,
Feb 28, 2021, 7:40:04 PM2/28/21
to
On Thu 25 Feb 2021 at 14:32:49 (+0000), Brian wrote:
> On Wed 24 Feb 2021 at 22:47:59 -0600, David Wright wrote:
> > On Thu 25 Feb 2021 at 10:36:40 (+0800), Robbi Nespu wrote:
> >
> > > > 5. Create /lib/firmware: mkdir /lib/firmware and transfer the
> > > > firmware there.
> > > >
> > > > 6. ALT-F1 to go back to d-i. d-i should now find the firmware.
> >
> > Note that on most systems, steps 3 through 6 are unnecessary as
> > the installer will find the firmware itself anyway.
>
> That's the advice given in Section 6.4.1 if the Installer Guide and,
> if it works, it works. However, I would not like to guarantee that it
> does, just as, in the case of the OP, the non-free installer ISO does
> not come up with installing the firmware.

Agreed: it's difficult to do too much for the installer.

And AFAICT my advice and the Installation Guide § 6.4.1 and
§ 4.3.1 are all out of date. It appears that nowadays (official
buster 10.8 netinst, amd64 and i386), loose firmware files
don't get picked up, either off a second USB stick¹, or off an
extra partition added to the hybrid installation stick. You
have to offer it the firmware in .deb files.

OTOH my official wheezy 7.1.0 netinst i386 installer does pick up
the loose files, exactly as described in the Installation Guide.
So it appears we have a regression in the debian-installer.

I haven't tested jessie or stretch because my netinst media are
all firmware versions, so I can't pinpoint the change in behaviour.
The Installation Guide's wording in those two sections (above)
became settled by the time of wheezy, but has remained unchanged
since then. Unless the problem has been fixed for bullseye²,
I would suggest that the guide text needs revising.

> Transferring the firmware files directly to where the installer kernel
> can find them seems more assured of success. If it doesn't, there is a
> big problem for the system both during and after installation.

I also haven't tested whether, when you copy loose files to
/lib/firmware, you have to preserve the directory structure.
For example, wheezy automatically copies the file tg3_tso5.bin
from '/media/tigon' to '/lib/firmware/tigon', but would manually
copying it to /lib/firmware/tg3_tso5.bin still work (if the
user wasn't aware of its usual location, …path-to/tigon/tg3_tso5.bin)?
↑↑↑↑↑↑

¹ All my USB sticks are partitioned, ie /dev/sdX1, not /dev/sdX.
² I haven't yet tried bullseye, or its installer, at all.

Cheers,
David.

Charles Curley

unread,
Feb 28, 2021, 10:00:05 PM2/28/21
to
On Sun, 28 Feb 2021 18:36:49 -0600
David Wright <deb...@lionunicorn.co.uk> wrote:

> It appears that nowadays (official buster 10.8 netinst, amd64 and
> i386), loose firmware files don't get picked up, either off a second
> USB stick¹, or off an extra partition added to the hybrid
> installation stick. You have to offer it the firmware in .deb files.

I see recent Bullseye i386 installers pick up firmware .deb files in the
root of an additional USB stick. I have not experimented with loose
files.

I prefer the .deb files, as a means of keeping the firmware consistent
with the version of Debian being installed. But perhaps d-i can offer
both without too much bloat?

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Brian

unread,
Mar 1, 2021, 9:20:04 AM3/1/21
to
On Sun 28 Feb 2021 at 18:36:49 -0600, David Wright wrote:

> On Thu 25 Feb 2021 at 14:32:49 (+0000), Brian wrote:
> > On Wed 24 Feb 2021 at 22:47:59 -0600, David Wright wrote:
> > > On Thu 25 Feb 2021 at 10:36:40 (+0800), Robbi Nespu wrote:
> > >
> > > > > 5. Create /lib/firmware: mkdir /lib/firmware and transfer the
> > > > > firmware there.
> > > > >
> > > > > 6. ALT-F1 to go back to d-i. d-i should now find the firmware.
> > >
> > > Note that on most systems, steps 3 through 6 are unnecessary as
> > > the installer will find the firmware itself anyway.
> >
> > That's the advice given in Section 6.4.1 if the Installer Guide and,
> > if it works, it works. However, I would not like to guarantee that it
> > does, just as, in the case of the OP, the non-free installer ISO does
> > not come up with installing the firmware.
>
> Agreed: it's difficult to do too much for the installer.
>
> And AFAICT my advice and the Installation Guide § 6.4.1 and
> § 4.3.1 are all out of date. It appears that nowadays (official
> buster 10.8 netinst, amd64 and i386), loose firmware files
> don't get picked up, either off a second USB stick¹, or off an
> extra partition added to the hybrid installation stick. You
> have to offer it the firmware in .deb files.

"loose firmware files" is a bit of a woolly term; it needs expanding
on. The .deb files will have a directory structure (please see later).

> OTOH my official wheezy 7.1.0 netinst i386 installer does pick up
> the loose files, exactly as described in the Installation Guide.
> So it appears we have a regression in the debian-installer.

That isn't quite my recollection but it is so long ago that I am not
willing to push it. I am also not motivated (yet) to try it out.

> I haven't tested jessie or stretch because my netinst media are
> all firmware versions, so I can't pinpoint the change in behaviour.
> The Installation Guide's wording in those two sections (above)
> became settled by the time of wheezy, but has remained unchanged
> since then. Unless the problem has been fixed for bullseye²,
> I would suggest that the guide text needs revising.
>
> > Transferring the firmware files directly to where the installer kernel
> > can find them seems more assured of success. If it doesn't, there is a
> > big problem for the system both during and after installation.
>
> I also haven't tested whether, when you copy loose files to
> /lib/firmware, you have to preserve the directory structure.
> For example, wheezy automatically copies the file tg3_tso5.bin
> from '/media/tigon' to '/lib/firmware/tigon', but would manually
> copying it to /lib/firmware/tg3_tso5.bin still work (if the
> user wasn't aware of its usual location, …path-to/tigon/tg3_tso5.bin)?
> ↑↑↑↑↑↑

The directory structure has to remain intact. Execute, for example,

modinfo zd1211rw | less

At the top we have

firmware: zd1211/zd1211_uphr

The kernel would expect the firmware file to be in zd1211/.

--
Brian.

David Wright

unread,
Mar 1, 2021, 12:00:05 PM3/1/21
to
I thought it would be clearer to use the d-i's own nomenclature.
Anyway, here are the relevant lines of syslog, with just tg3_tso5.bin,
the one that comes in a subdirectory:

kernel: [] tg3.c:v3.121 (November 2, 2011)

kernel: [] tg3 0000:02:02.0: eth0: Tigon3 [partno(BCM95705A50) rev 3003] (PCI:33MHz:32-bit) MAC address 00:c0:9f:44:15:b5
kernel: [] tg3 0000:02:02.0: eth0: attached PHY is 5705 (10/100/1000Base-T Ethernet) (WireSpeed[0], EEE[0])
kernel: [] tg3 0000:02:02.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
kernel: [] tg3 0000:02:02.0: eth0: dma_rwctrl[763f0000] dma_mask[64-bit]

net/hw-detect.hotplug: Detected hotpluggable network interface eth0

kernel: [] tg3 0000:02:02.0: firmware: agent aborted loading tigon/tg3_tso5.bin (not found?)
kernel: [] tg3 0000:02:02.0: eth0: TSO capability disabled
kernel: [] ADDRCONF(NETDEV_UP): eth0: link is not ready
kernel: [] tg3 0000:02:02.0: wake-up capability enabled by ACPI
check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
check-missing-firmware: missing firmware files (tigon/tg3_tso5.bin) for tg3
check-missing-firmware: copying loose file tg3_tso5.bin from '/media/tigon' to '/lib/firmware/tigon'
↑↑↑↑↑
kernel: [] tg3.c:v3.121 (November 2, 2011)
net/hw-detect.hotplug: Detected hotpluggable network interface eth0
kernel: [] tg3 0000:02:02.0: eth0: Tigon3 [partno(BCM95705A50) rev 3003] (PCI:33MHz:32-bit) MAC address 00:c0:9f:…
kernel: [] tg3 0000:02:02.0: eth0: attached PHY is 5705 (10/100/1000Base-T Ethernet) (WireSpeed[0], EEE[0])
kernel: [] tg3 0000:02:02.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
kernel: [] tg3 0000:02:02.0: eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
kernel: [] tg3 0000:02:02.0: firmware: agent loaded tigon/tg3_tso5.bin into memory

> > > Transferring the firmware files directly to where the installer kernel
> > > can find them seems more assured of success. If it doesn't, there is a
> > > big problem for the system both during and after installation.
> >
> > I also haven't tested whether, when you copy loose files to
> > /lib/firmware, you have to preserve the directory structure.
> > For example, wheezy automatically copies the file tg3_tso5.bin
> > from '/media/tigon' to '/lib/firmware/tigon', but would manually
> > copying it to /lib/firmware/tg3_tso5.bin still work (if the
> > user wasn't aware of its usual location, …path-to/tigon/tg3_tso5.bin)?
> > ↑↑↑↑↑↑
>
> The directory structure has to remain intact. Execute, for example,
>
> modinfo zd1211rw | less
>
> At the top we have
>
> firmware: zd1211/zd1211_uphr
>
> The kernel would expect the firmware file to be in zd1211/.

I confirmed that. The good news is that the d-i tells you the name of
the subdirectory: I had forgotten that (and these are the screens that
are almost impossible to log). So it's not an issue even for someone
who doesn't know the name of the module, or doesn't know to get a
shell and use modinfo.

But the regression remains, AFAICT.

Cheers,
David.

Brian

unread,
Mar 1, 2021, 3:50:05 PM3/1/21
to
That's good.

> But the regression remains, AFAICT.

I am going to pass on investigating an unsupported wheezy. I have my
installations working well with hd-media and a suitable preseed.cfg.
Naive users can sort themselves out,

--
Brian.
0 new messages