sr-iov issues, reset_hw() failed with error -100

[Ultima]

Decided to do some testing with iovctl to see how sr-iov is coming along.
Currently when adding the vf's there are a couple errors, and the network
no longer function after iovctl is started. My guess is the reset_hw() call
that is failing. Any ideas why this call would fail? I tested this on both
ports, ix1 is detached and unused for this test, however inserting a cable
results in an unusable port. iovctl -Dd ix1 removes the vf's, however
functionality is still not restored without a system restart.

FreeBSD S1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r295736: Wed Feb 17
21:17:28 EST 2016 root@S1:/usr/obj/usr/src/sys/MYKERNEL amd64

/boot/loader.conf
hw.ix.num_queues="4"

/etc/iovctl.conf
PF {
device : ix1;
num_vfs : 31;
}

DEFAULT {
passthrough : true;
}
VF-0 {
passthrough : false;
}
VF-1 {
passthrough : false;
}

# iovctl -C -f /etc/iovctl.conf

dmesg
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version -
1.4.6-k> at device 0.129 on pci12
ixv0: Using MSIX interrupts with 2 vectors
ixv0: ixgbe_reset_hw() failed with error -100
device_attach: ixv0 attach returned 5
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version -
1.4.6-k> at device 0.131 on pci12
ixv0: Using MSIX interrupts with 2 vectors
ixv0: ixgbe_reset_hw() failed with error -100
device_attach: ixv0 attach returned 5
pci12: <network, ethernet> at device 0.133 (no driver attached)
pci12: <network, ethernet> at device 0.135 (no driver attached)
pci12: <network, ethernet> at device 0.137 (no driver attached)
pci12: <network, ethernet> at device 0.139 (no driver attached)
pci12: <network, ethernet> at device 0.141 (no driver attached)
pci12: <network, ethernet> at device 0.143 (no driver attached)
pci12: <network, ethernet> at device 0.145 (no driver attached)
pci12: <network, ethernet> at device 0.147 (no driver attached)
pci12: <network, ethernet> at device 0.149 (no driver attached)
pci12: <network, ethernet> at device 0.151 (no driver attached)
pci12: <network, ethernet> at device 0.153 (no driver attached)
pci12: <network, ethernet> at device 0.155 (no driver attached)
pci12: <network, ethernet> at device 0.157 (no driver attached)
pci12: <network, ethernet> at device 0.159 (no driver attached)
pci12: <network, ethernet> at device 0.161 (no driver attached)
pci12: <network, ethernet> at device 0.163 (no driver attached)
pci12: <network, ethernet> at device 0.165 (no driver attached)
pci12: <network, ethernet> at device 0.167 (no driver attached)
pci12: <network, ethernet> at device 0.169 (no driver attached)
pci12: <network, ethernet> at device 0.171 (no driver attached)
pci12: <network, ethernet> at device 0.173 (no driver attached)
pci12: <network, ethernet> at device 0.175 (no driver attached)
pci12: <network, ethernet> at device 0.177 (no driver attached)
pci12: <network, ethernet> at device 0.179 (no driver attached)
pci12: <network, ethernet> at device 0.181 (no driver attached)
pci12: <network, ethernet> at device 0.183 (no driver attached)
pci12: <network, ethernet> at device 0.185 (no driver attached)
pci12: <network, ethernet> at device 0.187 (no driver attached)
pci12: <network, ethernet> at device 0.189 (no driver attached)

pciconf -lv
ix1@pci0:129:0:1: class=0x020000 card=0x00001458 chip=0x15288086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller 10-Gigabit X540-AT2'
class = network
subclass = ethernet
none155@pci0:129:0:129: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
none156@pci0:129:0:131: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt0@pci0:129:0:133: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt1@pci0:129:0:135: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt2@pci0:129:0:137: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt3@pci0:129:0:139: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt4@pci0:129:0:141: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt5@pci0:129:0:143: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt6@pci0:129:0:145: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt7@pci0:129:0:147: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt8@pci0:129:0:149: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt9@pci0:129:0:151: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt10@pci0:129:0:153: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt11@pci0:129:0:155: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt12@pci0:129:0:157: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt13@pci0:129:0:159: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt14@pci0:129:0:161: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt15@pci0:129:0:163: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt16@pci0:129:0:165: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt17@pci0:129:0:167: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt18@pci0:129:0:169: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt19@pci0:129:0:171: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt20@pci0:129:0:173: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt21@pci0:129:0:175: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt22@pci0:129:0:177: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt23@pci0:129:0:179: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt24@pci0:129:0:181: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt25@pci0:129:0:183: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt26@pci0:129:0:185: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt27@pci0:129:0:187: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet
ppt28@pci0:129:0:189: class=0x020000 card=0x00001458 chip=0x15158086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'X540 Ethernet Controller Virtual Function'
class = network
subclass = ethernet

Ultima
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Ultima]

After reboot...

ifconfig ix1 up

dhclient ix1
DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4
DHCPOFFER from 192.168.1.1
DHCPREQUEST on ix1 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.145 -- renewal in 21600 seconds.

ix0 down
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms

iovctl -Cf /etc/iovctl.conf

ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
^C
--- 192.168.1.1 ping statistics ---
29 packets transmitted, 0 packets received, 100.0% packet loss
ifconfig ix1 up
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
^C
--- 192.168.1.1 ping statistics ---
12 packets transmitted, 0 packets received, 100.0% packet loss

ix1 is no longer usable until a restart...

iovctl -Dd ix1
ifconfig ix1 up
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
^C
--- 192.168.1.1 ping statistics ---
9 packets transmitted, 0 packets received, 100.0% packet loss



Is there anything else that maybe useful?

here is my ifconfig at the end (after ifconfig ix0 up)


ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=e400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether -Hidden-
inet 192.168.1.8 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether -Hidden-
inet 192.168.1.145 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
ether -Hidden-
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=8<VLAN_MTU>
ether -Hidden-
inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner <e...@freebsd.org> wrote:

> Did you do an ifconfig up on ix1 before loading the VF driver?

[Eric Joyner]

Did you do an ifconfig up on ix1 before loading the VF driver?

On Sat, Feb 20, 2016 at 11:57 AM Ultima <ultim...@gmail.com> wrote:

[Pieper, Jeffrey E]

Just out of curiosity, how many cores does your system have?

Jeff

[Ultima]

This system has 24 cores (e5-2670v3)x2

Ultima

[Steven Hartland]

isn't that 48 cores (12 real 12 virtual) per CPU?

[Ultima]

Yeah, dmesg does show 48. 12 Cores and 24 threads each. 48 total

On Mon, Feb 22, 2016 at 4:35 PM, Steven Hartland <kil...@multiplay.co.uk>
wrote:

[Eric Joyner]

I don't really have any ideas on the error -100. Error -100 means there was
a mailbox error, so something failed in the initial communications setup
between the PF and VF, but I don't know what exactly went wrong.

I'm grasping at straws, but try using a smaller number of VFs initially,
like 2? And check to see if VT-d is enabled in your BIOS? (Though I
would've expected iovctl to fail).

- Eric

[Ultima]

I forgot to mention my kernel conf.... I'm not sure if it would cause this
issue, but I'll test again with GENERIC.

--- /usr/src/sys/amd64/conf/GENERIC 2016-02-22 21:05:37.152953000 -0500
+++ /root/MYKERNEL-11-CURRENT-AMD64 2015-12-28 19:18:22.893391452 -0500
@@ -91,6 +91,12 @@
options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed
options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones

+### VIMAGE ###
+options VIMAGE
+
+### ROUTE TABLES ###
+options ROUTETABLES=2
+
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel

The interface is just dead after creating the vf's earlier. Recreating
with only 2, its still dead.

Running out of ideas, decided to try a tcpdump...

# dhclient ix1

DHCPREQUEST on ix1 to 255.255.255.255 port 67
DHCPREQUEST on ix1 to 255.255.255.255 port 67
DHCPREQUEST on ix1 to 255.255.255.255 port 67

DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 13
DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 14

# tcpdump -i ix1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:41:13.234688 IP 192.168.1.145.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300
21:41:16.236671 IP 192.168.1.145.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300
21:41:23.243242 IP 192.168.1.145.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from -Hidden- (oui Unknown), length 300
21:41:38.261015 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from -Hidden- (oui Unknown), length 300
21:41:45.284752 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from -Hidden- (oui Unknown), length 300
21:41:58.292223 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from -Hidden- (oui Unknown), length 300

I don't think this is that helpful tho. =/
tcpdump on the other end, the packets are never received.

I'v already double checked VT-d, but I'll make it tripple! =] I can't reset
the system right now, but ill send an update when possible with only 2 vf's
and unmodified generic.

Ultima

[Ultima]

Upgraded to r295920 and used generic kernel. Then rebooted and checked
bios thoroughly. I found 3, yes 3 different areas in the bios for enabling
sr-iov (some screen shots below). 2 are for vt-d(Forgot to take a
screenshot of the 2nt, all were enabled) and one for sr-iov(was disabled).
Unfortunately with generic kernel, and all these options enabled, adding
only 2vf's resulted in the same behavior as above.

Little bit of good news, when removing the vf's and ifconfig ix1 down/up
the interface's functionality is restored. I also tested this on MYKERNEL
r295920, sr-iov option in bios likely played a role in this.

If you have any ideas, I'm willing to test them.

Thanks! =]

VT-d screen: https://puu.sh/niyiq/4fee92e4a3.jpgn
PCI advanced options: https://puu.sh/niyjg/88e71e48d9.jpgn

Ultima

[Ultima]

Decided to do some more tests, I actually have a second board with sr-iov
capabilities that I used for awhile with vmware esxi. I decided to test
this out and unfortunately it won't activate, it is giving the no space
left on device error message. I double checked bios and all VT-d related
options are enabled and have hw.ix.num_queues="4" in /boot/loader.conf. Is
there anything else that may need to be set? .(It did work on vmware)

For my second test, I moved the X540-AT1 to the board with the X540-AT2.
It functioned with the same issues as the AT2 tho.


I don't think I listed the motherboards in question yet so ill list them
now.

S1200BTLRM -
http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM
MD80-TM0 - http://b2b.gigabyte.com/products/product-page.aspx?pid=5146#ov

I'm not sure if it will be of any help tho.

[Eric Joyner]

Are you still getting the -100 errors when trying to load the VF driver?

I've tried SR-IOV on a system here, and I can confirm that traffic stops
passing on the PF interface when you create a VF interface. That didn't
used to happen, so I'm investigating why that is right now.

On Wed, Feb 24, 2016 at 8:09 AM Ultima <ultim...@gmail.com> wrote:

> Decided to do some more tests, I actually have a second board with sr-iov
> capabilities that I used for awhile with vmware esxi. I decided to test
> this out and unfortunately it won't activate, it is giving the no space
> left on device error message. I double checked bios and all VT-d related
> options are enabled and have hw.ix.num_queues="4" in /boot/loader.conf. Is
> there anything else that may need to be set? .(It did work on vmware)
>
> For my second test, I moved the X540-AT1 to the board with the X540-AT2.
> It functioned with the same issues as the AT2 tho.
>
>
> I don't think I listed the motherboards in question yet so ill list them
> now.
>
> S1200BTLRM -
>
> http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRM
> MD80-TM0

> <http://ark.intel.com/products/69633/Intel-Server-Board-S1200BTLRM?q=S1200BTLRMMD80-TM0>

[Ultima]

Yeah, still getting the -100 error, I do have sendmail disabled. I just
tested with sendmail up and running then add the VF's and it still shows
the error message.

[Eric Joyner]

I don't know if you're still interested in this, but did you do "kenv
hw.vmm.force_iommu=1" before loading the vmm module? I think that might be
necessary.

[Ultima]

I have been interested in this when I first read about it in 2012. =]

Can this be done in loader.conf? I have vmm_load="YES"

I'm not sure if the vf's are usable, I have not actually tested the vf's.
The parent ix1 still shows no response.

kldunload vmm

kenv hw.vmm.force_iommu=1

kldload vmm

iovctl -Cf /etc/iovctl.conf

The same error messages appear, I currently on hmm i'm not sure, I upgraded
with git and it doesn't show rev(last time I use git for
source?) 8b372d1(master)

[Ultima]

Recently upgraded to r297351, once running iovctl -C -f /etc/iovctl.conf
everything appears to work as it should. The main network interface ramains
connected and pinging works fine.

iovctl.conf
PF {
device : ix1;
num_vfs : 31;
}

DEFAULT {
passthrough : true;
}
VF-0 {
passthrough : false;
}
VF-1 {
passthrough : false;
}

Once vf's have been initialized, I have found none-passthrough vf's are
unusable.

# devctl attach pci0:129:0:129
devctl: Failed to attach pci0:129:0:129: Input/output error

The dmesg spits out the same error as before, so it appears that the errors
I was mentioning before is actually the none-passthrough vf's attempting to
attach, but fails.

/var/log/messages:

ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version -
1.4.6-k> at device 0.131 on pci12
ixv0: Using MSIX interrupts with 2 vectors
ixv0: ixgbe_reset_hw() failed with error -100
device_attach: ixv0 attach returned 5
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version -
1.4.6-k> at device 0.129 on pci12
ixv0: Using MSIX interrupts with 2 vectors
ixv0: ixgbe_reset_hw() failed with error -100
device_attach: ixv0 attach returned 5