Qubes and Broadcom BCM4360 - A Success Story
Dave C
# Background
I have a MacBookPro which sometimes I boot as a Mac, but also want to be able to boot to Qubes. Rather than install directly to the hard drive, I installed Qubes on a portable SSD (http://amzn.com/B00N0V4JG2), but I don't think that makes any difference in getting broadcom drivers set up.
If you try the portable SSD, I found I had to use standard partitioning as LVM didn't not work. Otherwise its a normal Qubes install with all but the boot sector encrypted. Qubes will boot just fine on the MacBookPro. But it won't recognize the Broadcom wifi hardware.
# Getting a Network Connection
There's a chicken and egg problem here. You'll need to get online in order to install the software needed to make the broadcom work. This is tricky if like mine the macbook has no ethernet port. There's only one USB PCI device, you can't associate it with a netvm (i.e. to tether). But there are a couple thunderbolt ports.
To get online I used a thunderbolt to ethernet with USB adapter (http://amzn.com/B00PY194CK). This adapter should work with ethernet, obviously, and also a USB tether through android device. I was able to get both to work, although the tethering was flakey. The point of this awkward device is to use it just long enough to get the broadcom working, then you shouldn't need it any more.
Since I installed Qubes on the portable SSD, I could have instead taken the portable SSD drive and booted it on some other hardware (i.e. a desktop with more linux-friendly hardware) and downloaded the necessary software there. That's what I'd do if I had to do this all over again, but the first time through it was handy to be on the machine with the broadcom, as there was some trial and error.
# Some Assembly Required
Once online, all the steps needed to get broadcom working can be found. It's a matter of sorting through the weeds to get to what works. What follows should help.
## Net VM Setup
I decided not to modify sys-net directly. I created a new net vm called net-powerbook. I even cloned the fedora-23 template, so my net-powerbook uses a template called f23-broadcom. I don't think the additional template is necessary. At the time, I thought I'd simply `sudo yum install broadcom-wl akmod-wl` and presto I'd have the drivers. With Qubes it is not that simple.
Attach the right PCI device to net-powerbook. In my case it is:
03:00.0 Network controller: Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter (rev 03)
Note, while using the thunderbolt adapter, I also had another PCI device attached. It's not plugged in now, so Qubes doesn't even list the PCI device, otherwise I'd paste it here.
Fire up a net-powerbook terminal.
## Install Broadcom Driver (on net-powerbook)
As I mentioned, a simple `sudo yum install broadcom-wl akmod-wl` didn't work for me.
I followed these instructions to build and install the wl module:
https://onpub.com/install-broadcom-linux-wi-fi-driver-on-fedora-23-s7-a192
Next, I followed Marek's method, and moved all modules into the /rw directory of net-powerbook vm:
https://groups.google.com/d/msg/qubes-users/Wt9Nm7posho/msTN_v2oa_oJ
I ended up with these lines in /rw/config/rc.local:
```
mount --bind /rw/modules /lib/modules
systemctl --no-block restart systemd-udevd
modprobe wl
```
## Give the PCI Device Permissive Passthrough (on dom0)
Although the driver is installed now when net-powerbook starts up, it won't work, I find, unless the device has permissive mode. Instructions to follow are here:
https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues
My /etc/systemd/system/qubes-pre-netvm.service ended up like this:
```
[Unit]
Description=permission pci netvm fixup
Before=qubes-netvm.servce
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo "0000:03:00.0\n0000:0a:00.0" > /sys/bus/pci/drivers/pciback/permissive'
[Install]
WantedBy=multi-user.target
```
Do note, my ExecStart line gives two devices permissive mode. 03:00.0 is the broadcom wireless. 0a:00.0 is the thunderbolt port that I was using. I suspected that permissive mode might help with the flakey USB tethering, so I am giving both permissive mode in this example. You'll need to edit your ExecStart line to refer to your broadcom device.
# Conclusion
Hopfully I didn't forget much. If the details aren't all there, some trial and error should get you through. If you find yourself rebooting the net vm you may also need to reboot the machine, if it is not detecting PCI hardware.
Of course, this post would not be complete without an HCL!
---
layout:
'hcl'
type:
'notebook'
hvm:
'yes'
iommu:
'yes'
tpm:
'unknown'
brand: |
Apple Inc.
model: |
MacBookPro11,1
bios: |
MBP111.88Z.0138.B16.1509081438
cpu: |
Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
cpu-short: |
FIXME
chipset: |
Intel Corporation Haswell-ULT DRAM Controller [8086:0a04] (rev 09)
chipset-short: |
FIXME
gpu: |
Intel Corporation Haswell-ULT Integrated Graphics Controller [8086:0a2e] (rev 09) (prog-if 00 [VGA controller])
gpu-short: |
FIXME
network: |
Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter (rev 03)
memory: |
16292
scsi: |
APPLE SSD SM0256 Rev: JA1Q
OTG Rev: 0
SD Card Reader Rev: 3.00
versions:
- works:
'FIXME:yes|no|partial'
qubes: |
R3.1
xen: |
4.6.0
kernel: |
4.1.13-9
remark: |
FIXME
credit: |
FIXAUTHOR
link: |
FIXLINK
---
Marek Marczykowski-Górecki
Hash: SHA256
On Fri, Mar 11, 2016 at 10:30:35PM -0800, Dave C wrote:
> With earlier Qubes I tried to get a broadcom BCM4360 wireless card working,
> but had no luck. Recently I tried again with Qubes 3.1 and guess what, its
> working. I'm using it to post this message now.
See below for a little comment.
After completing steps to enable it[1], just `sudo yum install
broadcom-wl akmod-wl` should be enough. At least in theory...
[1] https://www.qubes-os.org/doc/managing-vm-kernel/#tocAnchor-1-3
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJW4+bJAAoJENuP0xzK19csGbcH/jvD1Z1HAGr9TjeyYURRtG89
7tubCKIXdRnQhes8Sl+ZAmBhYPrkpnfN198p2oSWuOG7N/5AWF+5eQBmE4szsUN1
RkBmQ3omwyvVGXGyjgWUVo1LpDQ7g0tJgl1a+5reJ41mljHZnYAx+/y2fL/V2KNy
D++d1RXMrfcyUnITAtA387EJyYKHcbMGt5jAK2JQwoQZLIC5F6OelOROlPfNji+6
4uLFqRH7XtPPrMF3A6ncpd/rODkhavrudsgzOc/fupe6RTofO/1GVCQ5VUMnhmvx
ri2mQKoPIahkISiNJgqXqo/dOJCtv/ngWBAZH8SS9P2Q991n8+oaAncMGGbPYJE=
=M8oa
-----END PGP SIGNATURE-----
Dave C
On Saturday, March 12, 2016 at 1:52:19 AM UTC-8, Marek Marczykowski-Górecki wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Fri, Mar 11, 2016 at 10:30:35PM -0800, Dave C wrote:
> With earlier Qubes I tried to get a broadcom BCM4360 wireless card working,
> but had no luck. Recently I tried again with Qubes 3.1 and guess what, its
> working. I'm using it to post this message now.
Thanks!
I'm happy to contribute a little something back.
This sounds like a more future-proof approach. No risk of future kernel upgrades breaking the net vm. Will explore this as time permits.
BTW, I think `sudo yum install broadcom-wl akmod-wl` did not work as intended in Fedora 23 - nothing to do with Qubes. While the command had no errors, it doesn't seem to produce the wl module. Running `sudo modprobe wl` afterwards fails. Regardless of how you build the module, the pvgrub2 method sounds pretty clean.
-Dave
kent.f...@gmail.com
Generally, the steps required are:
in dom0:
echo 0000:03:00.0 | sudo tee /sys/bus/pci/drivers/pciback/permissive
in sys-net:
sudo dnf install gcc kernel-devel -y
cp -a /lib/modules /rw/modules
mount --bind /rw/modules /lib/modules
systemctl restart systemd-udevd
mkdir hybrid_wl_f23
cd hybrid_wl_f23
FILE='hybrid-v35_64-nodebug-pcoem-6_30_223_271.tar.gz'
wget http://www.broadcom.com/docs/linux_sta/$FILE
tar zxvf $FILE
make clean && make
sudo make install
sudo depmod -a
sudo rmmod b43 ssb bcma
/etc/modprobe.d/wl.conf
modprobe wl
I'm still working on getting the changes to survive reboot with rc.local / qubes-pre-netvm.service / modules-load / modprobe
tfor...@gmail.com
kent.f...@gmail.com
Dave C
Anyway, I thought of that as an excuse to upgrade my net vm for broadcom to the latest version of fedora. Unfortunately things are not working 100%. So while I've made progress, this is actually a call for help to get things working on fedora 24.
# Give the PCI Device Permissive Passthrough (on dom0)
https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues
My /etc/systemd/system/qubes-pre-netvm.service ended up like this:
```
[Unit]
Description=permission pci netvm fixup
Before=qubes-netvm.servce
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo "0000:03:00.0" > /sys/bus/pci/drivers/pciback/permissive'
[Install]
WantedBy=multi-user.target
```
# Preparation to compile modules
Start a VM running same template and kernel version that your netvm
will use. This can be a dispvm.
From https://www.broadcom.com/support/download-search/?pf=Wireless+LAN+Infrastructure
Move tarball to that vm (if not there already) and extract with
```
tar xvzf hybrid-v25_64-nodebug-nopcoem-6_30_223_271.tar.gz
```
Conveniently the README is not included. You might find it at
https://docs.broadcom.com/docs/1211168561592, a link which will soon
be just as broken as all its predecessors scattered all over the
internet.
## Compile the modules
```
# Install kernel-devel that matches running kernel!
sudo yum install "kernel-devel-uname-r == $(uname -r)"
# I think necessary (?)
sudo modprobe cfg80211
# possibly necessary (?)
sudo yum install gcc
# Finally
make
```
## Install
I compiled in a dvm, so my install procedure was...
First, on the dvm where I compiled:
```
# Copy the built module
qvm-copy-to-vm net-powerbook24 wl.ko
# Find out what the install command will be. Note -n will not actually install.
make -n install
```
Now we can use the command shown by `make -n install` earlier, in the new net vm (net-powerbook24 in my case).
```
install -D -m 755 QubesIncoming/disp2/wl.ko /lib/modules/`uname -r`/kernel/drivers/net/wireless
# This is documented in the readme, but not done by the install command.
depmod -a
```
But we want the module to be loaded automatically when the netvm starts. This is the part I'm having trouble with! If anyone can suggest a better way, please let me know.
Start by using Marek's technique for custom modules. (https://groups.google.com/forum/#!msg/qubes-users/Wt9Nm7posho/msTN_v2oa_oJ)
I've been experimenting with what to put in /rw/config/rc.local. What I have at the moment is shown below. But here's the problem...
When first starting the netvm, it does not successfully use the wifi. However, if I manually open a terminal and run `sudo /rw/config/rc.local` then it will be able to connect to wifi. So, something about this script is working, but not working on initial vm startup!
I have the following in /rw/config/rc.local:
```
# Unload conflicting modules.
rmmod ssb
rmmod bcma
rmmod b43
rmmod brcmsmac
rmmod wl
# blacklist modules that may interfere with wl (broadcom)
# (Not convinced these actually prevent the modules from loading!)
echo "blacklist ssb" >> /etc/modprobe.d/blacklist.conf
echo "blacklist bcma" >> /etc/modprobe.d/blacklist.conf
echo "blacklist b43" >> /etc/modprobe.d/blacklist.conf
echo "blacklist brcmsmac" >> /etc/modprobe.d/blacklist.conf
mount --bind /rw/modules /lib/modules
systemctl restart systemd-udevd
modprobe wl
```
...and make that rc.local executable.
I've tried various experiments. I've changed the order in that script of when I remove conflicting modules. Nothing I've tried makes that script work successfully on startup. I suspect that conflicting modules are loaded despite my attempt to blacklist them. But when I run the script manually in a terminal, it works as hoped, and I can connect to wifi. As I said earlier, I'd appreciate any help!
Kent Davis
all the modules in sys-net:/rw/config/rc.local.
Is your rc.local executable?
> You received this message because you are subscribed to a topic in the Google Groups "qubes-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-users/VVwWqvz5dX4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to qubes-users...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/f191cec7-3514-427f-9e78-3ec066b9cec8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Dave C
> My solution involves directly modifying the sys-net VM and I handle
> all the modules in sys-net:/rw/config/rc.local.
>
> Is your rc.local executable?
>
It is:
[user@net-powerbook24 ~]$ ls -l /rw/config/rc.local
-rwxr-xr-x 1 root root 523 Dec 18 15:57 /rw/config/rc.local
And I had a simple rc.local working for earlier version of fedora. But with 24, there are conflicting modules and I haven't managed to get it right.
I'm wondering when rc.local is executed during startup, and whether I need to explicitly wait for other startup to complete. As I mentioned, my script works only after startup, when I manually run in a terminal.
-Dave
Kent Davis
towards the end of the boot process.
> You received this message because you are subscribed to a topic in the Google Groups "qubes-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-users/VVwWqvz5dX4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to qubes-users...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
Dave Cohen
On Wed, Dec 21, 2016, at 08:53 AM, Kent Davis wrote:
> I'm not certain, but based on when wireless comes up, I think it runs
> towards the end of the boot process.
>
Kent Davis
rc.local? kinda hacky though...
friend...@gmail.com
I figured it out! The problem is that you ordered your rmmods wrong. I figured this out because if I disabled the rc.local script and just ran it on my own after boot, it would require to be run twice because two of the other modules were being used by b43 and therefore could not be disabled with it still running. So, you need to disable it first. This rc.local works perfectly on Fedora 24:
```
# Unload conflicting modules.
rmmod b43
rmmod ssb
rmmod bcma
rmmod brcmsmac
rmmod wl
# blacklist modules that may interfere with wl (broadcom)
# Not sure if this matters, haven't tried without
Gray Olson
Okay so I've found another issue. For some reason, when the system is hibernated (e.g. lid is closed), the module seems to unload itself and the whole netvm freezes, requiring a "kill vm" and restart for it to work again. Any ideas on the cause of this?