Anyone using Solarflare on FreeBSD 10-STABLE ?

Mark Saad

unread,

Oct 17, 2017, 12:33:07 PM10/17/17

to

Hello
I was wondering if anyone else out there was using the solarflare
cards supported via the sfxge driver on 10.3-STABLE. Speificly I
wanted to know if anyone knew what this sysctl refers to

dev.sfxge.NN.stats.rxdp_di_dropped_pkts

--
mark saad | none...@longcount.org
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

Allan Jude

unread,

Oct 17, 2017, 1:36:41 PM10/17/17

to

On 10/17/2017 12:32, Mark Saad wrote:
> Hello
> I was wondering if anyone else out there was using the solarflare
> cards supported via the sfxge driver on 10.3-STABLE. Speificly I
> wanted to know if anyone knew what this sysctl refers to
>
> dev.sfxge.NN.stats.rxdp_di_dropped_pkts
>
>

I have two 11.1 boxes with a dual ported sfxge each, but I have no idea
what that counter is trying to describe.

The man page says this is Philip's fault.

--
Allan Jude

Philip Paeps

unread,

Oct 17, 2017, 1:54:40 PM10/17/17

to

On 2017-10-17 19:36:20 (+0200), Allan Jude wrote:
> On 10/17/2017 12:32, Mark Saad wrote:
>> I was wondering if anyone else out there was using the solarflare
>> cards supported via the sfxge driver on 10.3-STABLE.

Yes.

>> Speificly I wanted to know if anyone knew what this sysctl refers to
>>
>> dev.sfxge.NN.stats.rxdp_di_dropped_pkts
>
> I have two 11.1 boxes with a dual ported sfxge each, but I have no
> idea
> what that counter is trying to describe.
>
> The man page says this is Philip's fault.

This code is not my fault! ;-) I believe arybchik added it.

Looking at the code, it's packets that have been dropped in the data
path by the dispatcher cpu. Probably related to virtual functions?

arybchik (Cc:ed) likely has a more recent datasheet than I do!

Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information

Mark Saad

unread,

Oct 17, 2017, 3:15:59 PM10/17/17

to

On Tue, Oct 17, 2017 at 1:54 PM, Philip Paeps <phi...@freebsd.org> wrote:
> On 2017-10-17 19:36:20 (+0200), Allan Jude wrote:
>>
>> On 10/17/2017 12:32, Mark Saad wrote:
>>>
>>> I was wondering if anyone else out there was using the solarflare
>>> cards supported via the sfxge driver on 10.3-STABLE.
>
>
> Yes.
>
>>> Speificly I wanted to know if anyone knew what this sysctl refers to
>>>
>>> dev.sfxge.NN.stats.rxdp_di_dropped_pkts
>>
>>
>> I have two 11.1 boxes with a dual ported sfxge each, but I have no idea
>> what that counter is trying to describe.
>>
>> The man page says this is Philip's fault.
>
>
> This code is not my fault! ;-) I believe arybchik added it.
>
> Looking at the code, it's packets that have been dropped in the data path by
> the dispatcher cpu. Probably related to virtual functions?
>
> arybchik (Cc:ed) likely has a more recent datasheet than I do!
>
> Philip
>

Solarflare replied to my support request I raised with them saying ...

"This counter is of packets discarded in hardware due to not being
address to this host, or to any multicast group it is subscribed to.

Seeing a very high number here usually indicates the switch is not
performing IGMP snooping, and is forwarding all multicast; or that
there is some routing issue. However it will also usually tick up
over time, due to route discovery protocols."

I have no idea how you get to that; as a result of reading that oid,
but ok if they say so.

I was just being cautious I disabled the Event Merge Timeout on the
card and I was just watching the card counters , among other things,
to see if there was any drops as a result.

> --
> Philip Paeps
> Senior Reality Engineer
> Ministry of Information
>
> _______________________________________________
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

--
mark saad | none...@longcount.org

Andrew Rybchenko

unread,

Oct 17, 2017, 3:50:41 PM10/17/17

to

On 10/17/2017 08:54 PM, Philip Paeps wrote:
> On 2017-10-17 19:36:20 (+0200), Allan Jude wrote:
>> On 10/17/2017 12:32, Mark Saad wrote:
>>> Speificly I wanted to know if anyone knew what this sysctl refers to
>>>
>>> dev.sfxge.NN.stats.rxdp_di_dropped_pkts
>>
>> I have two 11.1 boxes with a dual ported sfxge each, but I have no idea
>> what that counter is trying to describe.
>>
>> The man page says this is Philip's fault.
>
> This code is not my fault! ;-) I believe arybchik added it.
>
> Looking at the code, it's packets that have been dropped in the data
> path by the dispatcher cpu. Probably related to virtual functions?

Philip, thanks. I don't think in this particular case it is related to
virtual functions.

Basically the counter means that ingress packet does not match any
installed filter. E.g. promiscuous mode is disabled and:
- destination MAC is unicast and not the interface MAC address, OR
- destination MAC is multicast and there is no matching multicast
address added.

There is a race condition as well on interface bring up when packet is
received but default filters are not installed yet.

SFN8522 and SFN8542 have other cases for encapsulated traffic depending
on driver version (right now I don't remember the state in 10-STABLE).

Mark, let me know if I can help more.

Andrew.

Mark Saad

unread,

Oct 24, 2017, 5:04:57 PM10/24/17

to

Could you clarify what was meant by Virtual Functions ? Vlans , laggs
etc or srv-io ?
Also I do have IGMP snoop enabled and routing is done locally on each
router -, the freebsd box with the sfxge cards.

--
mark saad | none...@longcount.org

Andrew Rybchenko

unread,

Oct 27, 2017, 3:06:00 AM10/27/17

to

I meant SR-IOV. Which SF NIC do you use?

Andrew.

Mark Saad

unread,

Oct 27, 2017, 2:03:23 PM10/27/17

to

Andrew
I am using IIRC 7140's This is what I get from the various utils

Solarflare Flareon Ultra 7000 Series 10G Adapter
Firmware version: v6.4.3
Controller type: Solarflare SFC9140
Boot ROM version: v5.0.4.1000

--
mark saad | none...@longcount.org

Andrew Rybchenko

unread,

Oct 30, 2017, 6:12:24 AM10/30/17

to

Do I understand correctly that rxdp_di_dropped_pktsgrows when interface
is running?
I.e. it is not just few packets at start up.

If it is possible, I'd try to enable promisc mode and check if stops the
counter grow.
If so and traffic is not huge, I'd try to sniff on the interface and
filter out traffic to local
MAC, broadcast and locally added multicast addresses to check if
unexpected traffic
comes to the interface.

Andrew.

Mark Saad

unread,

Oct 30, 2017, 8:48:02 AM10/30/17

to

> Do I understand correctly that rxdp_di_dropped_pkts grows when interface is running?

> I.e. it is not just few packets at start up.

Correct the counter is a 6 -7 digit number, depending on which server I am looking at. It increases but I am not exactly sure why of the rate .

>
> If it is possible, I'd try to enable promisc mode and check if stops the counter grow.
> If so and traffic is not huge, I'd try to sniff on the interface and filter out traffic to local
> MAC, broadcast and locally added multicast addresses to check if unexpected traffic
> comes to the interface.
>
> Andrew.

I will try this on my lab and see what happens . Thanks again for the ideas .

---
Mark Saad | none...@longcount.org