moving to 10Gbps, part 2

[Jeff Nucciarone]

A few weeks ago I asked about what would be involved on moving my sensor from a 1Gbps network to a 10G one. based on feedback I picked up an Intel NIC and today plugged it in to my 10G switch.

However I am not seeing any packets via tcpdump or otherwise, and I'm a bit stumped. My machine does detect the NIC and the driver is loaded, so I'm not sure if there's any additional SO configuration I need to make or not.

lspci shows the network adapter is present. The first one listed is my eth2 copper 1G interface:

04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

lshw shows the driver (ixgbe) is loaded and the device is in promiscuous mode:

*-network:0
description: Ethernet interface
product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:05:00.0
logical name: eth3
version: 01
serial: 90:e2:ba:a1:aa:b0
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical fibre
configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=3.6.7-k firmware=0x61c10001 latency=0 link=no multicast=yes promiscuous=yes
resources: irq:41 memory:d9900000-d997ffff ioport:dcc0(size=32) memory:d98f8000-d98fbfff memory:d0300000-d03fffff memory:d0200000-d02fffff
*-network:1 UNCLAIMED
description: Ethernet controller
product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:05:00.1
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress cap_list
configuration: latency=0
resources: memory:d9980000-d99fffff ioport:dce0(size=32) memory:d98fc000-d98fffff memory:d0100000-d01fffff memory:d0000000-d00fffff

For good measure hwew is the output of lsmod:

lsmod |grep -E 'ixgbe|mdio|igb|ioat|dca'
nf_conntrack_broadcast 12589 1 nf_conntrack_netbios_ns
nf_conntrack 81926 8 nf_conntrack_ipv6,xt_state,nf_conntrack_netbios_
ns,nf_conntrack_broadcast,nf_nat_ftp,nf_nat,nf_conntrack_ipv4,nf_conntrack_ftp
ixgbe 179594 0
dca 15232 1 ixgbe
mdio 13807 1 ixgbe

ifconfig shows no packets:

eth3 Link encap:Ethernet HWaddr 90:e2:ba:a1:aa:b0
UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

I did verify that I have data flowing on the switch side (plus I switched from copper to fiber on the port)

Any suggestions before I move to an Ubuntu forum?

Thanks.

[Jeff Nucciarone]

I forgot to point out that I replaced an existing, configured (but unused) 1 Gbps eth3 device. I removed that NIC from the PCI slot and replaced it woth the 10 Gbps device. There was an existing eth3 configuration in /etc/network/interfaces.

[Doug Burks]

On Mon, Sep 14, 2015 at 4:46 PM, Jeff Nucciarone
<jeff.nu...@gmail.com> wrote:
> ifconfig shows no packets:
>
> eth3 Link encap:Ethernet HWaddr 90:e2:ba:a1:aa:b0
> UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Hi Jeff,

According to the ifconfig output, eth3 has the correct MAC address, is
UP, and is configured properly. I would double-check the cabling and
the switch configuration.


--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Jeff Nucciarone]

> According to the ifconfig output, eth3 has the correct MAC address, is
> UP, and is configured properly. I would double-check the cabling and
> the switch configuration.
>

Issue fixed, but not what I originally thought!

Here's the post-mortem for those following along in case this issue happens to them.

It turns out it was not configured correctly. I used ethtool to blink the light on what I thought was eth3. The light was blinking on the port I was not using. I moved the sfp over to that port and the ixgbe module promptly unloaded itself.

dmesg indicated the driver unloaded due to use of a 'non appoved' sfp (Finisar sfp, Intel NIC).

After a bit of research I found the set of commands:

rmmod ixgbe
modprobe ixgbe allow_unsupported_sfp=1

and all was well with the world.

I need to figure out how to have this happen at boot so I don't have to manually run this set of commands every time the machine reboots.

In the end it was not related to any prior configuration.

[Andrea De Pasquale]

On Tuesday, September 15, 2015 at 9:22:48 PM UTC+2, Jeff Nucciarone wrote:
> I need to figure out how to have this happen at boot so I don't have to manually run this set of commands every time the machine reboots.

Hi Jeff,
should be as simple as creating the file:

/etc/modprobe.d/ixgbe.conf

and writing this inside it:

options ixgbe allow_unsupported_sfp=1

Have a look at /etc/modprobe.d/pf_ring.conf if what I wrote above is not clear enough.

Regards,
-- Andrea De Pasquale