Dividing Qubes Into Separate Networks (FAILED)

[Zsolt Bicskey]

Network setup: pfsense router + Unif Switch

I have two NICs on the server runninq Qubes. On one I want the nework to conect to the main LAN via DHCP and get out that way (that's done and working like charm).

On the other NIC I want a separate gateway (sys-net) and separate firewall going through a VLAN out to the internet. pfsense and switch is setup properly. If I connect a Windows laptop to that dedicated port it works. It does not work on Qubes:

I cloned the main firewall named it to pentest-firewall. I cloned the main gateway name it to pentest-gw. If I point the pentest-firewall to the main-gw everything works but then I am reaching the internet from the wrong NIC. But if I point the pentest-firewall at the pentest-gw there is no internet. I assigned the NIC to the pentest-gw. I see the mac address but I am not getting IP via DHCP. If I set the IP manually then I see on the switch the dedicated port cycles every 2 seconds between off / on  / blocked. Either way I cannot access the internet.

What am I missing?

I also tried not clonging the main-gw but creating a VM from a template, check "provides network" and assign the NIC and still didn't work.

[unman]

On Fri, May 01, 2020 at 07:53:53PM +0000, 'Zsolt Bicskey' via qubes-users wrote:
> Network setup: pfsense router + Unif Switch
>
> I have two NICs on the server runninq Qubes. On one I want the nework to conect to the main LAN via DHCP and get out that way (that's done and working like charm).
>
> On the other NIC I want a separate gateway (sys-net) and separate firewall going through a VLAN out to the internet. pfsense and switch is setup properly. If I connect a Windows laptop to that dedicated port it works. It does not work on Qubes:
>

> I cloned the main firewall named it to pentest-firewall. I cloned the main gateway name it to pentest-gw. If I point the pentest-firewall to the main-gw everything works but then I am reaching the internet from the wrong NIC. But if I point the pentest-firewall at the pentest-gw there is no internet. I assigned the NIC to the pentest-gw. I see the mac address but I am not getting IP via DHCP. If I set the IP manually then I see on the switch the dedicated port cycles every 2 seconds between off / on?? / blocked. Either way I cannot access the internet.
>
> What am I missing?
>

Without a lot more information, it's difficult to say.
Have you checked that the new qube has necessary firmware?
Is the NIC on the pentest-gw working correctly?
Does it work when connected to the port currently used by main-gw?
Set the VLAN correctly?
Set all parameters necessary to satisfy any port security on the switch?

[Zsolt Bicskey]

> Without a lot more information, it's difficult to say.
> Have you checked that the new qube has necessary firmware?

what firmware?

> Is the NIC on the pentest-gw working correctly?

yes

> Does it work when connected to the port currently used by main-gw?

yes
> Set the VLAN correctly?
yes, as I said if I connect a Windows latptop it works right away

> Set all parameters necessary to satisfy any port security on the switch?

Yes, same answer as above

[dhorf-hfre...@hashmail.org]

On Sat, May 02, 2020 at 03:14:42AM +0000, 'Zsolt Bicskey' via qubes-users wrote:

> > Set the VLAN correctly?
> yes, as I said if I connect a Windows latptop it works right away
> > Set all parameters necessary to satisfy any port security on the switch?
> Yes, same answer as above

actualy, those are not "answers" at all.
there is nothing in this description confirming you know how to
configure a network interface under linux.

since you confirmed the second port is working in general, this
is unlikely to be a qubes problem.
may be a whatever-your-netvm-distro-is problem.
or more likely, a configuration problem.

try booting whatever the distro in your netvm is off a USB stick
or dvd (== without xen involved), and get the right network
interface to work on the right port with that.
then copy over the interface configuration to your netvm.

[unman]

This.

[Zsolt Bicskey]

> > > > Set the VLAN correctly?
> > > > yes, as I said if I connect a Windows latptop it works right away
> > > > Set all parameters necessary to satisfy any port security on the switch?
> > > > Yes, same answer as above
> >

> > actualy, those are not "answers" at all.
> > there is nothing in this description confirming you know how to
> > configure a network interface under linux.

My apologies unman that I am not a Linux poweruser. I have only been using it casually for the past 20 years. I have yet to run into a situation where I was not able to configure my network on any SUSE/Slackware, Debian or RHEL based systems. The reason I came here is to get help, not to be reminded what I do not understand or know about networking.

> > since you confirmed the second port is working in general, this
> > is unlikely to be a qubes problem.
> > may be a whatever-your-netvm-distro-is problem.
> > or more likely, a configuration problem.
> > try booting whatever the distro in your netvm is off a USB stick
> > or dvd (== without xen involved), and get the right network
> > interface to work on the right port with that.
> > then copy over the interface configuration to your netvm.
>

So my understanding is that it would not solve the DHCP settings but if I were to try manually setting it then Fedora stores the settings in /etc/sysconfig/network-scripts/ifcfg-Wired_connection_# which is not a permanent location in Qubes so I'd lose it with every reboot.


And just to be really sure everything is working on the machine I tested all interfaces with manual and DHCP settings with the Live boot Fedora and everything was working.


Can someone please try to help me solve this issue? Everything in line after Qubes is 100% working as designed. There is someting with the Qubes gateway that I am goofing up and I cannot find what is wrong.

[Jarrah]

> My apologies unman that I am not a Linux poweruser. I have only been using it casually for the past 20 years. I have yet to run into a situation where I was not able to configure my network on any SUSE/Slackware, Debian or RHEL based systems. The reason I came here is to get help, not to be reminded what I do not understand or know about networking.

You got that response because of the terse replys to Unmans ask for more
information. Had you been a little more forthcoming, we might have a
solution.

> So my understanding is that it would not solve the DHCP settings but
> if I were to try manually setting it then Fedora stores the settings
> in /etc/sysconfig/network-scripts/ifcfg-Wired_connection_# which is
> not a permanent location in Qubes so I'd lose it with every reboot.

These settings are re-written at each reboot. However, network manager
is not (see /rw/config/NM-system-connections). Try configuring it from
network manager (nmcli or nm-connection-editor). Also ensure that things
such as link speed negotiation, MTU and MAC address (if you are using
port security) are correct. All of the above are in pentest-gw, which I
assume is a clone of sys-net.

What you are doing is absolutely possible in Qubes.

[unman]

You seem to have taken offence at a comment not made by me.
I *do* endorse the statement that your replies were not answers.

The question isn't whether you have correctly configured VLAN on the
port, but whether you have configured the interface correctly.
Saying that a Windows laptop can connect is irrelevant to that point.

Now we know that Fedora live connects to the VLAN port - what
configuration was required? (I leave open the possibility that Fedora
live implements some autoconfiguration not included in the basic Fedora
template)
Have you made the same configuration in your pentest-gw?
What do the logs say? (In Qubes and on switch.)

[Zsolt Bicskey]

> You seem to have taken offence at a comment not made by me.

My bad. I lost track who said what.

> Now we know that Fedora live connects to the VLAN port - what
> configuration was required? (I leave open the possibility that > Fedora
> live implements some autoconfiguration not included in the basic Fedora template)

No configuration was required. On DHCP it just automatically worked. I switched over to manual config, assigned a different IP and once again it just worked. I copied the config file out from th network scripts and tried implementing the same on the pentest-gw

> Have you made the same configuration in your pentest-gw?

I tried the same settings in the pentests-gw that I gaterhered from the liveboot and made no difference.

> What do the logs say? (In Qubes and on switch.)

SWITCH LOGS:
May 4 14:56:25 UBNT daemon.notice switch: TRAPMGR: Link Up: 0/6
May 4 14:56:25 UBNT daemon.info switch: DOT1S: Port (6) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED

PENTEST-GW LOGS:

May 04 14:55:46 pentest-gw systemd[559]: Started Mark boot as successful after the user session has run 2 minutes.
May 04 14:55:46 pentest-gw systemd[559]: Reached target Paths.
May 04 14:55:46 pentest-gw systemd[559]: Reached target Timers.
May 04 14:55:46 pentest-gw systemd[559]: Starting D-Bus User Message Bus Socket.
May 04 14:55:46 pentest-gw systemd[559]: Listening on Multimedia System.
May 04 14:55:46 pentest-gw systemd[559]: Condition check resulted in Sound System being skipped.
May 04 14:55:46 pentest-gw systemd[559]: Listening on D-Bus User Message Bus Socket.
May 04 14:55:46 pentest-gw systemd[559]: Reached target Sockets.
May 04 14:55:46 pentest-gw systemd[559]: Reached target Basic System.
May 04 14:55:46 pentest-gw systemd[559]: Reached target Main User Target.
May 04 14:55:46 pentest-gw systemd[559]: Startup finished in 175ms.
May 04 14:55:48 pentest-gw systemd[559]: Starting D-Bus User Message Bus...
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Service file '/usr/share/dbus-1/services/org.gnome.evolution.dataserver.Calendar.service' is not named after the D-Bus name 'org.gnome.evolution.dataserver.Calendar8'.
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Service file '/usr/share/dbus-1/services/org.gnome.evolution.dataserver.AddressBook.service' is not named after the D-Bus name 'org.gnome.evolution.dataserver.AddressBook10'.
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Service file '/usr/share/dbus-1/services/org.gnome.evolution.dataserver.UserPrompter.service' is not named after the D-Bus name 'org.gnome.evolution.dataserver.UserPrompter0'.
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Service file '/usr/share/dbus-1/services/org.freedesktop.mate.Notifications.service' is not named after the D-Bus name 'org.freedesktop.Notifications'.
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Service file '/usr/share/dbus-1/services/org.gnome.evolution.dataserver.Sources.service' is not named after the D-Bus name 'org.gnome.evolution.dataserver.Sources5'.
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
May 04 14:55:48 pentest-gw dbus-broker-launch[711]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
May 04 14:55:48 pentest-gw systemd[559]: Started D-Bus User Message Bus.
May 04 14:55:48 pentest-gw dbus-broker-lau[711]: Ready
May 04 14:55:48 pentest-gw systemd[559]: Starting Tracker metadata database store and lookup manager...
May 04 14:55:48 pentest-gw systemd[559]: Starting Virtual filesystem service...
May 04 14:55:48 pentest-gw systemd[559]: Started Virtual filesystem service.
May 04 14:55:48 pentest-gw systemd[559]: Started Tracker metadata database store and lookup manager.
May 04 14:55:48 pentest-gw systemd[559]: Created slice dbus\x2d:1.2\x2dca.desrt.dconf.slice.
May 04 14:55:48 pentest-gw systemd[559]: Started dbus-:1.2-ca.de...@0.service.
May 04 14:55:48 pentest-gw pulseaudio[729]: vchan module loading
May 04 14:55:48 pentest-gw pulseaudio[729]: using domid: 0
May 04 14:55:48 pentest-gw pulseaudio[729]: play libvchan_fd_for_select=19, ctrl=0x57995005aa40
May 04 14:55:48 pentest-gw pulseaudio[729]: rec libvchan_fd_for_select=23, ctrl=0x57995005abe0
May 04 14:55:48 pentest-gw pulseaudio[729]: sink cork req state =1, now state=-2
May 04 14:55:48 pentest-gw pulseaudio[729]: source cork req state =1, now state=-2
May 04 14:55:48 pentest-gw pulseaudio[729]: module-rescue-stream is obsolete and should no longer be loaded. Please remove it from your configuration.
May 04 14:55:49 pentest-gw systemd[559]: Starting Accessibility services bus...
May 04 14:55:49 pentest-gw systemd[559]: Started Accessibility services bus.
May 04 14:55:49 pentest-gw at-spi-bus-launcher[816]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +15: Eavesdropping is deprecated and ignored
May 04 14:55:49 pentest-gw at-spi-bus-launcher[816]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +17: Eavesdropping is deprecated and ignored
May 04 14:55:49 pentest-gw dbus-broker-lau[821]: Ready
May 04 14:55:49 pentest-gw systemd[559]: Created slice dbus\x2d:1.13\x2dorg.a11y.atspi.Registry.slice.
May 04 14:55:49 pentest-gw systemd[559]: Started dbus-:1.13-org.a11y....@0.service.
May 04 14:55:49 pentest-gw at-spi2-registryd[830]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
May 04 14:55:53 pentest-gw pulseaudio[729]: source cork req state =2, now state=1
May 04 14:55:53 pentest-gw pulseaudio[729]: sink cork req state =2, now state=1
May 04 14:56:07 pentest-gw systemd[559]: Starting Portal service...
May 04 14:56:07 pentest-gw systemd[559]: Starting flatpak document portal service...
May 04 14:56:07 pentest-gw systemd[559]: Starting sandboxed app permission store...
May 04 14:56:07 pentest-gw systemd[559]: Started sandboxed app permission store.
May 04 14:56:07 pentest-gw systemd[559]: Started flatpak document portal service.
May 04 14:56:07 pentest-gw systemd[559]: Starting Portal service (GTK+/GNOME implementation)...
May 04 14:56:07 pentest-gw systemd[559]: Started Portal service (GTK+/GNOME implementation).
May 04 14:56:07 pentest-gw systemd[559]: Started Multimedia Service.
May 04 14:56:07 pentest-gw xdg-desktop-por[838]: Failed to get application states: GDBus.Error:org.freedesktop.portal.Error.Failed: Could not get window list: Cannot invoke method; proxy is for the well-known name org.gnome.Shell without>
May 04 14:56:07 pentest-gw pipewire[864]: [E][module-protocol-native.c:336 client_new()] no peersec: Protocol not available
May 04 14:56:07 pentest-gw systemd[559]: Started Portal service.
May 04 14:56:19 pentest-gw tracker-store[738]: OK
May 04 14:56:19 pentest-gw systemd[559]: tracker-store.service: Succeeded.
May 04 14:57:07 pentest-gw xdg-desktop-por[838]: Failed to get application states: GDBus.Error:org.freedesktop.portal.Error.Failed: Could not get window list: Cannot invoke method; proxy is for the well-known name org.gnome.Shell without>
May 04 14:58:07 pentest-gw xdg-desktop-por[838]: Failed to get application states: GDBus.Error:org.freedesktop.portal.Error.Failed: Could not get window list: Cannot invoke method; proxy is for the well-known name org.gnome.Shell without>
May 04 14:58:33 pentest-gw systemd[559]: Starting Mark boot as successful...
May 04 14:58:33 pentest-gw systemd[559]: grub-boot-success.service: Succeeded.
May 04 14:58:33 pentest-gw systemd[559]: Started Mark boot as successful.
May 04 14:59:07 pentest-gw xdg-desktop-por[838]: Failed to get application states: GDBus.Error:org.freedesktop.portal.Error.Failed: Could not get window list: Cannot invoke method; proxy is for the well-known name org.gnome.Shell without>
May 04 14:59:54 pentest-gw qubes.StartApp+org.gnome.Terminal-dom0[962]: # _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’
May 04 14:59:54 pentest-gw systemd[559]: Starting GNOME Terminal Server...
May 04 14:59:54 pentest-gw systemd[559]: Started GNOME Terminal Server.
May 04 14:59:54 pentest-gw qubes.StartApp+org.gnome.Terminal-dom0[962]: # _g_io_module_get_default: Found default implementation dconf (DConfSettingsBackend) for ‘gsettings-backend’
May 04 14:59:54 pentest-gw qubes.StartApp+org.gnome.Terminal-dom0[962]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)
May 04 14:59:54 pentest-gw qubes.StartApp+org.gnome.Terminal-dom0[962]: # unwatch_fast: "/org/gnome/terminal/legacy/" (active: 0, establishing: 1)
May 04 14:59:54 pentest-gw qubes.StartApp+org.gnome.Terminal-dom0[962]: # watch_established: "/org/gnome/terminal/legacy/" (establishing: 0)
May 04 15:00:07 pentest-gw xdg-desktop-por[838]: Failed to get application states: GDBus.Error:org.freedesktop.portal.Error.Failed: Could not get window list: Cannot invoke method; proxy is for the well-known name org.gnome.Shell without>

[Zsolt Bicskey]

Here is full summary of where I am at. Could someone please provide guidance with this? Thank you very much.


Qubes OS version
Qubes OS R4.0

Affected component(s) or functionality
Networking

Brief summary
I tried to separate everything into to two subnets meanings 2 NICs, 2 gateways (sys-net), 2 firewalls. Everything works on the network before the new gw and after it. All qubes can communicate to the firewall. After the gateway everything works properly on the physical network as designed and can get out to the internet if I connect any client other to it but the new gateway.

The main gateway remains functional but the new one can't get on the network, hence the whole chain doesn't work.

To Reproduce
Steps to reproduce the behavior (I tried 3 different way, same results):
First Version:
Simply clone the main gateway from Qubes Manager.

Second Version:
From dom0 (as root) under /srv/formulas/base/virtual/machines/formula duplicate and edit the following two files: sys-net.top and sys-net.sls and run qubesctl state.apply qvm/sys-net2 to create a new sys-net from scratch.

Third version:
Create new stanadlone VM, mark "provides networking"

Expected behavior
My hope was that once I have a new sys-net I can just assign the other NIC to it and connect to the network just like the main gateway

Actual behavior
If I leave the advanced network manager on DHCP then the gw is not getting and IP from the server. (If I connect any other non-Qubes clients they get an IP right away). If I set the IP manually then it "takes it" but I still cannot get on the network, and can't get online.

Additional context
The physical setup is this: modem <--> pfsense firewall <--> Unifi Switch <--> Server Running Qubes

The server has two built in NICs, one PCI and one WiFi. It might be important that if I assign all 3 (not in use) NICs to the 2nd gw then only 1 has a mac address. The other 2 show up as ens[0-9] but I don't see a mac

The network is setup so that the main gw on Qubes is on the main LAN segment on the network. The 2nd gw has a designated VLAN setup

Solutions you've tried
1) To make sure everything works on the server running Qubes and the network itself I used a live boot Linux and tried all NICs. Every NIC was able to connect to both the main LAN and the separate VLAN using both DHCP and manual IP settings.

2) As I listed above I tried cloning the 2nd gw from the main one and I tried creating from scratch

3) I tried editing the gw network settings though nmcli and the GUI

4) I booted the server with a Fedora 31 live USB, set network setting manually, copied out the /etc/sysconfig/network-scripts/ifcfg-interface-name and manually entered all those through nmcli

Just to reiterate once more, the network setup outside of Qubes is 100% functional. If I connect any machines to any segment of network to any port on the switch they always work as intended.

[dono...@unseen.is]

> --
Hello. I have a similar setup but without a VLAN - never been a fan. I have a 4-port pfsense router (community edition on a Protectli appliance), a couple of small unmanaged switches and a couple of ubiquiti APs. I cloned sys-net & sys-firewall to, say, sys-net-play & sys-firewall-play.

My Qubes box has 2 wired NICs - one is assigned the default network, the other play. I added a new DHCP scope to the pfsense for play (typical consumer class c), tossed a couple of firewall rules on the pfsense box for both subnets to prevent traffic between them. Each LAN has its own switch and AP.

From my Qubes box, I can assign either network to any VM. In fact, I do just that to remote control some hobby gear I have on the play net.

I am wondering it you might need to use two wired NICs.

DG

[Matt Drez]

>

> Hello. I have a similar setup but without a VLAN - never been a fan. I have a 4-port pfsense router (community edition on a Protectli appliance), a couple of small unmanaged switches and a couple of ubiquiti APs. I cloned sys-net & sys-firewall to, say, sys-net-play & sys-firewall-play.
>

> My Qubes box has 2 wired NICs - one is assigned the default network, the other play. I added a new DHCP scope to the pfsense for play (typical consumer class c), tossed a couple of firewall rules on the pfsense box for both subnets to prevent traffic between them. Each LAN has its own switch and AP.
>

> From my Qubes box, I can assign either network to any VM. In fact, I do just that to remote control some hobby gear I have on the play net.
>

> I am wondering it you might need to use two wired NICs.
>

> DG
>

I also have an almost identical setup. I wanted to do what you were attempting (Zsolt) but had the same outcome so I quit trying . I thought it's not possible. I tried following this old article but the commands did no longer work the same way https://blog.invisiblethings.org/2011/09/28/playing-with-qubes-networking-for-fun.html

I am not sure if your goal is feasible at all. It didn't work for me but I am fairly new to Linux so actually don't listen to me lol :)

I have the quad port commercial pfsense netgate appliance but I only use an unmanaged switch unlike your unifi. I could not make the VLAN work. I ended up just having 1 sys-net and separate everything with two firewalls and can chose on each VM which route to take similar to what DG was saying.

[Jarrah]

>
> I also have an almost identical setup. I wanted to do what you were attempting (Zsolt) but had the same outcome so I quit trying . I thought it's not possible. I tried following this old article but the commands did no longer work the same way https://blog.invisiblethings.org/2011/09/28/playing-with-qubes-networking-for-fun.html

This document is quite old, but conceptually not bad. The commands in it
are likely to fail now, but the theory still applies.

>
> I am not sure if your goal is feasible at all. It didn't work for me but I am fairly new to Linux so actually don't listen to me lol :)

I can confirm that splitting Qubes networking into two zones with two
different NICs is feasible. I have exactly this configuration using a
dual-nic motherboard and handling VLANs on a managed switch.

>
> I have the quad port commercial pfsense netgate appliance but I only use an unmanaged switch unlike your unifi. I could not make the VLAN work. I ended up just having 1 sys-net and separate everything with two firewalls and can chose on each VM which route to take similar to what DG was saying.

Having an unmanaged switch defeats the purpose of this. You will receive
two IP addresses on the same network. You may be able to assign the VLAN
on the NIC and have the PFSense device recognize it, but this is not
guaranteed to work.

[unman]

On Mon, May 11, 2020 at 03:16:03PM +0000, 'Zsolt Bicskey' via qubes-users wrote:
> Here is full summary of where I am at. Could someone please provide guidance with this? Thank you very much.
>
>
> Qubes OS version
> Qubes OS R4.0
>
> Affected component(s) or functionality
> Networking
>
> Brief summary
> I tried to separate everything into to two subnets meanings 2 NICs, 2 gateways (sys-net), 2 firewalls. Everything works on the network before the new gw and after it. All qubes can communicate to the firewall. After the gateway everything works properly on the physical network as designed and can get out to the internet if I connect any client other to it but the new gateway.
>
> The main gateway remains functional but the new one can't get on the network, hence the whole chain doesn't work.
>

<snip detail>

> Solutions you've tried
> 1) To make sure everything works on the server running Qubes and the network itself I used a live boot Linux and tried all NICs. Every NIC was able to connect to both the main LAN and the separate VLAN using both DHCP and manual IP settings.
>
> 2) As I listed above I tried cloning the 2nd gw from the main one and I tried creating from scratch
>
> 3) I tried editing the gw network settings though nmcli and the GUI
>
> 4) I booted the server with a Fedora 31 live USB, set network setting manually, copied out the /etc/sysconfig/network-scripts/ifcfg-interface-name and manually entered all those through nmcli
>
> Just to reiterate once more, the network setup outside of Qubes is 100% functional. If I connect any machines to any segment of network to any port on the switch they always work as intended.
>

The thing is, you simply refuse to give us any of the detail that might be
useful here - you don't identify the NICs (may be relevant), you don't
tell us *what* configuration you have set on the VLAN, and (crucially)
what configuration you have set on the NIC.
When you used the live Fedora, did you connect both NICs to the ports?

I accept some reluctance to give out identifying data, but you
cant expect help without this.

The key facts seem to be:
1. The NIC attached to pentest-gw has MAC address assigned, and works
when attached to non-VLAN port on switch.
2. That NIC can be configured without error with static IP address.
3. That NIC does not automatically connect to a VLAN port, and the
switch shows an error (off/on/blocked cycle)

Th obvious conclusion is that there's something wrong with your
VLAN configuration of the NIC. (Since the NIC connected to the port
under Fedora Live we can rule out problems with the NIC itself.)
The fact that Fedora Live autoconfigures, but the Fedora template based
qube does not, may indicate that there's some crucial package missing.
Test this by creating an HVM, assigning the NIC, and booting from Fedora
Live.

[unman]

Just for fun, can you run `lsmod|grep 8021q` in pentest-gw?
Post the name and content of your config file.

[Zsolt Bicskey]

> The thing is, you simply refuse to give us any of the detail that might be

There is nothing I am refusing. Please, let me know what you need and I'll provide it.

> useful here - you don't identify the NICs (may be relevant), you don't

> tell us what configuration you have set on the VLAN, and (crucially)

> what configuration you have set on the NIC.

I am not sure what you mean. Tell me where to look or what command you run and I'll give you the output.

The NIC is an Intel Gigabit Onboard NIC
VLAN config on the pfsense is simply tagged with the VLAN ID and passed on the LAN interface. The switch has a dedicated port that only allows that tag through. Pfsense also has a DHCP server running on the VLAN's subnet.

> When you used the live Fedora, did you connect both NICs to the ports?

YES

> I accept some reluctance to give out identifying data, but you
> cant expect help without this.

There is absolutely no reluctance. Ask specifically for something and I'll provide it

> under Fedora Live we can rule out problems with the NIC itself.)
> The fact that Fedora Live autoconfigures, but the Fedora template based
> qube does not, may indicate that there's some crucial package missing.
> Test this by creating an HVM, assigning the NIC, and booting from Fedora
> Live.

I supposed those would be two separate tasks. What do I do once the HVM boots up with the NIC assigned?

> Just for fun, can you run `lsmod|grep 8021q` in pentest-gw?

Did not give any output

> Post the name and content of your config file.

Which config file are you referring to?


Thanks for trying to help.

[unman]

And, THERE is your problem.
Dot1q is the networking standard that defines use of VLANs - formally,
IEEE 802.1Q.
To make use of it in Linux there is a kernel module that has to be loaded.
It's called 8021q. In many Fedora systems it is automatically loaded - in
your Fedora Live, for example. In the Fedora qube it isn't - you need to
load it manually.
You do this using the command, `modprobe 8021q`

Then you should be able to configure the VLAN, and connect.

[Zsolt Bicskey]

> You do this using the command, `modprobe 8021q`

> Then you should be able to configure the VLAN, and connect.

That's it then. I ran that command.

We are getting very close. I appreciate your help.


Now how should I configure it?


All Fedora guides I found were talking about adding a new file to the /etc/sysconfig/network-scripts/ but that would be lost with every reboot.


So, I tried nmcli but that didn't work either:
nmcli con add type vlan con-name VLAN21 ifname VLAN21 dev ens6 id 21
nmcli connection modify VLAN21 ipv4.addresses 172.10.10.11/24 \
ipv4.method manual ipv4.gateway 172.10.10.1 \
ipv4.dns 172.10.10.1

[unman]

On Thu, May 14, 2020 at 02:13:42AM +0000, Zsolt Bicskey wrote:
> > You do this using the command, `modprobe 8021q`
>
> > Then you should be able to configure the VLAN, and connect.
>
> That's it then. I ran that command.
>
> We are getting very close. I appreciate your help.
>
>
> Now how should I configure it?
>
>

Hmm, I had the impression that you had *already* configured the VLAN on
the NIC.

> All Fedora guides I found were talking about adding a new file to the /etc/sysconfig/network-scripts/ but that would be lost with every reboot.
>

You could investigate the use of `bind-dirs` in Qubes -
https://www.qubes-os.org/doc/bind-dirs`

unman

[Zsolt Bicskey]

> > Now how should I configure it?
>

> Hmm, I had the impression that you hadalready configured the VLAN on

I had it configured on the ens6 interface and I tried connecting to the network like that. Since that is how the fedora liveboot connected. I never tried to configure VLAN on Linux before with nmcli or any other way. I never had to. In the past anytime I built a network I was able to connect to the VLAN with DHCP or static addressing the interface. That's why I didn't look for the the modprobe 8021q becuase I never heard of it. That is why I cannot wrap my mind around why this is not working on the Qubes sys-net. I am not a Linux expert for sure but never had such issue ever before.

[unman]

OK.
You say that you have read configuration guides - please follow the
instructions in a recent Fedora guide, (which includes the modprobe
instruction), and see if that solves your issue.
If you still have a problem, then report back with details of what you
have done, and what errors you encountered.
If you fix it, just drop a line, saying "Followed *this* guide - now
working". Then it's useful for someone in the future.

[Zsolt Bicskey]

> If you still have a problem, then report back with details of what you

I have tried the following ways. All failed. (Ignore the discrepancies between interface names and commands. I copied them from the article I used.)

METHOD #1
# ip link add link eth0 name eth0.5 type vlan id 5
# ip link
# ip -d link show eth0.5
# ip addr add 192.168.1.200/24 brd 192.168.1.255 dev eth0.5
# ip link set dev eth0.5 up

METHOD #2
# vconfig add eth0 5
# ifconfig eth0.5 192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255 up

I can see that there are frames moving but it is not connecting to the network
sudo cat /proc/net/vlan/ens6.5

ens6.5 VID: 5 REORDER_HDR: 1 dev->priv_flags: 1001
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0

total frames transmitted 23
total bytes transmitted 1986
Device: ens6
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings:


METHOD #3

Add VLAN through the GUI Advanced Network Manager


This is what I see on the switch:
May 15 19:15:33 UBNT daemon.notice switch: TRAPMGR: Link Down: 0/6
May 15 19:15:33 UBNT daemon.info switch: DOT1S: Port (6) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
May 15 19:15:36 UBNT daemon.notice switch: TRAPMGR: Link Up: 0/6
May 15 19:15:36 UBNT daemon.info switch: DOT1S: Port (6) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED

I'm attaching the last 200 lines from the journalctl