What does VMA_TX_MC_LOOPBACK do?

[Shawn Bohrer]

Perhaps this is a stupid question, but is transmit multicast loopback actually supported with libvma?  My testing has shown that if my sending application is using libvma that none of the multicast packets get looped back and thus receivers on the same host do not receive the packets.  This makes sense to me since we are bypassing the kernel and sending the packets straight through the NIC.

Interestingly I see that there is a VMA_TX_MC_LOOPBACK option that defaults to enabled and is additionally twiddled by the IP_MULTICAST_LOOP socket option.  As far as I can tell the only thing this really does is discard received loopback packets if disabled (see sockinfo_udp::rx_input_cb).  If this all that the VMA_TX_MC_LOOPBACK config option does it seems it should be called RX not TX.  I see some comments about IPoIB and loopback in the README but I'm running all of this over Ethernet not IB.

Thanks,
Shawn

[orkme...@gmail.com]

In linux kernel, multicast loopback is done in the TX part of the stack. As you wrote, since we are bypassing the kernel, and we have a separated stack for each process, we can't implement loopback properly in the software layer.

In latest MLNX_OFED versions there was a change to allow loopback by the NIC. If you upgrade to latest MLNX_OFED/VMA MC loopback should work.

(for IB it was already working, and now it is working for ETH as well).

Regarding VMA_TX_MC_LOOPBACK, you are right.

In linux, the IP_MULTICAST_LOOP socket option allow to control the MC loopback behavior.

The control in the kernel is done in the TX flow.

VMA_TX_MC_LOOPBACK control the default value for this socket option.

But, since we can't currently control it in TX flow, we filter the loopback messages on RX flow.

In the future, I hope we will be able to control the loopback from the TX flow, by instructing the NIC to do loopback per packet. 

For now, the proper name is probably VMA_RX_MC_LOOPBACK as you mentioned.

I will consider changing it, or leaving it as is for current users consistency.

[Shawn Bohrer]

Thanks Or,

I probably wouldn't change the name to VMA_RX_MC_LOOPBACK but it may be worth expanding the description in the README.

Have the loopback driver changes in MLNX_OFED been sent upstream yet?  I searched the net, net-next, infiniband, and Linus' trees and didn't see anything relevant.  I prefer to run an upstream kernel.