Ethtool Driver

[Pavan Outlaw]

NOTESome adapters based on the Intel(R) Ethernet Controller 700 Series onlysupport Intel Ethernet Optics modules. On these adapters, other modules are notsupported and will not function. In all cases Intel recommends using IntelEthernet Optics; other modules may function but are not validated by Intel.Contact Intel for supported media types.

Allows you to send all packets with a specific VLAN tag to a particular SR-IOVvirtual function (VF). Further, this feature allows you to designate aparticular VF as trusted, and allows that trusted VF to request selectivepromiscuous mode on the Physical Function (PF).

The vf-true-promisc-support priv-flag does not enable promiscuous mode; rather,it designates which type of promiscuous mode (limited or true) you will getwhen you enable promiscuous mode using the ip link commands above. Note thatthis is a global setting that affects the entire device. However,thevf-true-promisc-support priv-flag is only exposed to the first PF of thedevice. The PF remains in limited promiscuous mode (unless it is in MFP mode)regardless of the vf-true-promisc-support setting.

Note that the order in which you set the VF to promiscuous mode and add theVLAN interface does not matter (you can do either first). The end result inthis example is that the VF will get all traffic that is tagged with VLAN 100.

NOTE: The Linux i40e driver supports the following flow types: IPv4, TCPv4, andUDPv4. For a given flow type, it supports valid combinations of IP addresses(source or destination) and UDP/TCP ports (source and destination). Forexample, you can supply only a source IP address, a source IP address and adestination port, or any combination of one or more of these four parameters.

NOTE: The Linux i40e driver allows you to filter traffic based on auser-defined flexible two-byte pattern and offset by using the ethtool user-defand mask fields. Only L3 and L4 flow types are supported for user-definedflexible filters. For a given flow type, you must clear all Intel Ethernet FlowDirector filters before changing the input set (for that flow type).

ATR is enabled by default when the kernel is in multiple transmit queue mode.An ATR Intel Ethernet Flow Director filter rule is added when a TCP-IP flowstarts and is deleted when the flow ends. When a TCP-IP Intel Ethernet FlowDirector rule is added from ethtool (Sideband filter), ATR is turned off by thedriver. To re-enable ATR, the sideband can be disabled with the ethtool -Koption. For example:

The second command will fail with an error. You may program multiple filterswith the same fields, using different values, but, on one device, you may notprogram two tcp4 filters with different matching fields.

Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload.Thus to match the first byte of the payload, you must actually add 4 bytes tothe offset. Also note that ip4 filters match both ICMP frames as well as raw(unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame.

The maximum offset is 64. The hardware will only read up to 64 bytes of datafrom the payload. The offset must be even because the flexible data is 2 byteslong and must be aligned to byte 0 of the packet payload.

The user-defined flexible offset is also considered part of the input set andcannot be programmed separately for multiple filters of the same type. However,the flexible data is not part of the input set and multiple filters may use thesame offset but match against different data.

Link messages will not be displayed to the console if the distribution isrestricting system messages. In order to see network driver link messages onyour console, set dmesg to eight by entering the following:

The driver utilizes the ethtool interface for driver configuration anddiagnostics, as well as displaying statistical information. The latest ethtoolversion is required for this functionality. Download it at:

In the default mode, an Intel(R) Ethernet Network Adapter using copperconnections will attempt to auto-negotiate with its link partner to determinethe best setting. If the adapter cannot establish link with the link partnerusing auto-negotiation, you may need to manually configure the adapter and linkpartner to identical settings to establish link and pass packets. This shouldonly be needed when attempting to link with an older switch that does notsupport auto-negotiation or one that has been forced to a specific speed orduplex mode. Your link partner must match the setting you choose. 1 Gbps speedsand higher cannot be forced. Use the autonegotiation advertising setting tomanually set devices for 1 Gbps and higher.

Caution: Only experienced network administrators should force speed and duplexor change autonegotiation advertising manually. The settings at the switch mustalways match the adapter settings. Adapter performance may suffer or youradapter may not operate if you configure the adapter differently from yourswitch.

An Intel(R) Ethernet Network Adapter using fiber-based connections, however,will not attempt to auto-negotiate with its link partner since those adaptersoperate only in full duplex and only at their native speed.

Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enablereceiving and transmitting pause frames for i40e. When transmit is enabled,pause frames are generated when the receive packet buffer crosses a predefinedthreshold. When receive is enabled, the transmit unit will halt for the timedelay specified when a pause frame is received.

Note: This command only enables or disables Flow Control if auto-negotiation isdisabled. If auto-negotiation is enabled, this command changes the parametersused for auto-negotiation with the link partner.

Virtual Extensible LAN (VXLAN) allows you to extend an L2 network over an L3network, which may be useful in a virtualized or cloud environment. SomeIntel(R) Ethernet Network devices perform VXLAN processing, offloading it fromthe operating system. This reduces CPU utilization.

VXLAN offloading is controlled by the Tx and Rx checksum offload optionsprovided by ethtool. That is, if Tx checksum offload is enabled, and theadapter has the capability, VXLAN offloading is also enabled.

Minimum TX Bandwidth is the guaranteed minimum data transmission bandwidth, asa percentage of the full physical port link speed, that the partition willreceive. The bandwidth the partition is awarded will never fall below the levelyou specify.

DCB is a configuration Quality of Service implementation in hardware. It usesthe VLAN priority tag (802.1p) to filter traffic. That means that there are 8different priorities that traffic can be filtered into. It also enablespriority flow control (802.1Qbb) which can limit or eliminate the number ofdropped packets during network stress. Bandwidth can be allocated to each ofthese priorities, which is enforced at the hardware level (802.1Qaz).

Adapter firmware implements LLDP and DCBX protocol agents as per 802.1AB and802.1Qaz respectively. The firmware based DCBX agent runs in willing mode onlyand can accept settings from a DCBX capable peer. Software configuration ofDCBX parameters via dcbtool/lldptool are not supported.

The range of 0-235 microseconds provides an effective range of 4,310 to 250,000interrupts per second. The value of rx-usecs-high can be set independently ofrx-usecs and tx-usecs in the same ethtool command, and is also independent ofthe adaptive interrupt moderation algorithm. The underlying hardware supportsgranularity in 4-microsecond intervals, so adjacent values may result in thesame interrupt rate.

The above command would disable adaptive interrupt moderation, and allow amaximum of 5 microseconds before indicating a receive or transmit was complete.However, instead of resulting in as many as 200,000 interrupts per second, itlimits total interrupts per second to 50,000 via the rx-usecs-high parameter.

NOTE: For better performance when processing small (64B) frame sizes, tryenabling Hyper threading in the BIOS in order to increase the number of logicalcores in the system and subsequently increase the number of queues available tothe adapter.

Application Device Queues (ADq) allows you to dedicate one or more queues to aspecific application. This can reduce latency for the specified application,and allow Tx traffic to be rate limited per application. Follow the steps belowto set ADq.

If either aws ec2 describe-instances --instance-ids instance_id --query "Reservations[].Instances[].EnaSupport" or aws ec2 describe-images --image-id ami_id --query "Images[].EnaSupport" (where instance_id is your (i-e04566365f208b6584) instance id and ami_id is your AMI's ID (ami-6365f208)) doesn't return "True" something is not marked as supporting ENA.

If it's your instance. Stop it, and run aws ec2 modify-instance-attribute --instance-id instance_id --ena-support to enable support and restart it. If it's your AMI, you'll either have to select an AMI that already has ENA support, or make your own from an instance that's been tagged (per above) as supporting ENA.

If your AMI already supported ENA, after running the above command, start it back up and ethtool -i eth0 should show support. If your AMI was the problem, you'll need to launch a fresh instance from an AMI that has ENA support.

Connecting via the local console and doing some testing with tcpdump I have concluded that the NIC is receiving traffic, but not sending traffic. I've proved this beyond doubt. So this is the problem.

If I boot the server from a Knoppix live CD I can configure the interfaces and they all work perfectly. So the hardware is fine. Something has gone screwy with the GAiA TCP stack on the server, receiving but not sending.

If the system is a member of a cluster, that interface may not be transmitting anything because it thinks it is the standby/backup member for that interface. I think this is the most likely scenario. Does the problematic interface appear in cphaprob -a if or show vrrp interfaces? Is the cluster showing a healthy condition via cphaprob state?

3a8082e126