SR-IOV on bare metal

[jw...@g.clemson.edu]

Hello all,

Is it possible to enable SR-IOV on bare metal machines with a NIC that supports it?  I tried adding creating /etc/modprobe.d/ixgbe.conf with the line "options ixgbe max_vfs=2" in an Ubuntu 14.04 LTS on Dell C6220, but after reboot there is no apparent change in lspci.  I did find a document describing how to enable SR-IOV in the BIOS on Dell servers, which suggests to me that it may be disabled (or I am doing it wrong).  Any thoughts on this?

Thanks in advance,

Jason Anderson

[Mike Hibler]

Hmm...apparently we didn't enable it in the BIOS. Our bad.
We'll fix that!

Incidentally, the Clemson part of cloudlab is chock full of Dell C8220
machines with Intel 10Gb interfaces. But alas, SR-IOV is disabled there
as well.

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/326c41dd-a1e3-4988-b311-f3665ad8ecb6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nicholas Bastin]

On Mon, May 11, 2015 at 11:19 PM, <jw...@g.clemson.edu> wrote:

Is it possible to enable SR-IOV on bare metal machines with a NIC that supports it?  I tried adding creating /etc/modprobe.d/ixgbe.conf with the line "options ixgbe max_vfs=2" in an Ubuntu 14.04 LTS on Dell C6220, but after reboot there is no apparent change in lspci.  I did find a document describing how to enable SR-IOV in the BIOS on Dell servers, which suggests to me that it may be disabled (or I am doing it wrong).  Any thoughts on this?

I suspect that this is disabled in the BIOS of cloudlab machines (as the response so far has indicated), however, when it is enabled there is no need to reboot a machine to create the VFs and test whether this enabled.  In any modern linux you can tweak the VFs under sys:

/sys/class/net/$IFNAME/device/sriov_numvfs

As long as the $IFNAME interface is down (you can't change the number of VFs of an active interface), you can write to this number to change it from 0 to 64, or any other valid number (it depends on the NIC on what is valid, of course).  It may take a few seconds for the VFs to get created, but once the command returns you'll be able to use the VFs without rebooting the system.

--

Nick 

[jw...@g.clemson.edu]

Oh good, I'm glad I wasn't just being dumb.  Thank you for the quick reply.  Do you happen to know if there are any machines that definitely do have SR-IOV enabled right now?  I'm working on an experiment with a relatively close deadline, but I could possibly repurpose an older machine in our lab as a last resort.

[jw...@g.clemson.edu]

Awesome, thank you for the tip.

[Mike Hibler]

Well you can swap in on either Apt or Clemson and I can manually fix up
the BIOS on the machines you get (assuming it isn't very many machines!)

> > an email to cloudlab-user...@googlegroups.com <javascript:>.

> > > To post to this group, send email to cloudla...@googlegroups.com

> > <javascript:>.

> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/cloudlab-users/326c41dd-a1e3-4988-b311-f3665ad8ecb6%40googlegroups.com.
> >
> > > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/c7201619-f022-4084-a4c3-f02e7905ea57%40googlegroups.com.

[Brian Kroth]

For what it's worth, I've just recently (as in the last couple of days)
been trying to enable support for SR-IOV on the nodes in the Wisconsin
part of CloudLab as well.

The short summary is, it's not working for the experimental network
interfaces yet. From my quick read of the driver source code so far, it
appears that the Cisco enic driver for the 10G VIC vNic interfaces
should have some support for it, but it's not showing up in sysfs,
ethtool, lspci, virsh, etc.

The BIOS has support for it enabled and I can get it to work for the 1GB
Intel control network interfaces (igb driver), but no go for the ones
you'd really want to use yet. I'm planning on bringing it up on our
next call with Cisco, but I don't imagine it will be available soon
enough for you to do your work.

Cheers,
Brian

Mike Hibler <mi...@flux.utah.edu> 2015-05-11 21:43:

>To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/20150512034311.GJ64264%40flux.utah.edu.

[Brian Kroth]

Nicholas Bastin <nick....@gmail.com> 2015-05-12 02:19:

^^^ That is good advice. However ...

I've only skimmed the driver source code, so I might be wrong, but I
think that interaction through sysfs like you describe needs to be
enabled in the individual drivers. At least in my own research, I see
references to sysfs in the code for the igb driver, but none in the enic
driver, and sure enough I see entries in the sysfs for interfaces using
the former, but not the latter.

Also, in the past at least, you often times had to boot the kernel with
an extra parameter like intel_iommu=on. That may be getting
auto-detected in more recent kernels though.

Here's a few more links that might be handy for you:
http://lxr.free-electrons.com/source/Documentation/PCI/pci-iov-howto.txt
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
https://www.kernel.org/doc/Documentation/kernel-parameters.txt

Cheers,
Brian

[jw...@g.clemson.edu]

So, your first link (also available at https://www.kernel.org/doc/Documentation/PCI/pci-iov-howto.txt) showed that you can echo the number into /sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_numvfs, which I assume is similar to the method Nicholas mentioned.  I tried this on an 8220 at Clemson and it seems to have worked, after echoing '2' into that file and a 1 second wait, lspci shows this:

82:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

82:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

82:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

82:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

Now, I haven't actually tried using it yet, but so far so good, right?