NS3 Buffer condition failed

[Jubran Altaweel]

Hey guys,

I am trying to simulate 3 different docker containers talking to each other through NS3. But when I try to run the simulator I get this error log:

TapBridge::ReadCallback(): Scheduling handler
TapBridge:ForwardToBridgedDevice(0x57f55463dbe0, 33, 110)
Received packet from tap device
TapBridge:Filter(0x57f55463dbe0, 0x57f554624b50)
Pkt source is 02:50:5d:c7:98:46
Pkt destination is 33:33:00:00:00:16
Pkt LengthType is 34525
Pkt source is 02-06-02:50:5d:c7:98:46
Pkt destination is 02-06-33:33:00:00:00:16
Pkt LengthType is 34525
Forwarding packet to ns-3 device via Send()
TapBridge:ForwardToBridgedDevice(0x57f55462d7f0, , 98)
Received packet from tap device
TapBridge:Filter(0x57f55462d7f0, 0x57f5545ca660)
Pkt source is 02:42:ac:11:00:01
Pkt destination is 00:00:00:00:00:02
Pkt LengthType is 2048
Pkt source is 02-06-02:42:ac:11:00:01
Pkt destination is 02-06-00:00:00:00:00:02
Pkt LengthType is 2048
Forwarding packet to ns-3 device via Send()
Ipv4L3Protocol:Receive(0x57f5545cfae0, 0x57f5545d6d10, 0x57f554666c40, 2048, 02-06-42:9b:44:0a:ff:0a, 02-06-00:00:00:00:00:02, 0)
Packet from 02-06-42:9b:44:0a:ff:0a received on node 1
Forwarding state: true
Forwarding state: true
Forwarding state: true
Ipv4L3Protocol:IpForward(0x57f5545cfae0, 0x763398010bc0, 0x57f554666da0, tos 0x0 DSCP Default ECN Not-ECT ttl 64 id 895 protocol 1 offset (bytes) 0 flags [DF] length: 84 10.0.1.10 > 10.0.2.10)
Forwarding logic for node: 1
Ipv4L3Protocol:SendRealOut(0x57f5545cfae0, 0x763398010bc0, 0x57f554666e10, 0x7ffe4f150fe8)
Send via NetDevice ifIndex 2 ipv4InterfaceIndex 2
Send to gateway 10.1.1.2
Ipv4L3Protocol:Receive(0x57f5545d88b0, 0x57f5545c2aa0, 0x57f5545d6380, 2048, 02-06-00:00:00:00:00:07, 02-06-00:00:00:00:00:08, 0)
Packet from 02-06-00:00:00:00:00:07 received on node 3
Forwarding state: true
Forwarding state: true
Ipv4L3Protocol:SourceAddressSelection(0x57f5545d88b0, 1, " ", 10.0.2.0)
Ipv4L3Protocol:IpForward(0x57f5545d88b0, 0x57f5545b27e0, 0x57f554666c40, tos 0x0 DSCP Default ECN Not-ECT ttl 63 id 895 protocol 1 offset (bytes) 0 flags [DF] length: 84 10.0.1.10 > 10.0.2.10)
Forwarding logic for node: 3
Ipv4L3Protocol:SendRealOut(0x57f5545d88b0, 0x57f5545b27e0, 0x57f554666da0, 0x7ffe4f151058)
Send via NetDevice ifIndex 1 ipv4InterfaceIndex 1
Send to destination 10.0.2.10
ArpL3Protocol:Lookup(0x57f5545dc0a0, 0x57f554666da0, 10.0.2.10, 0x57f5545bfd10, 0x57f5545caa70, 0x7ffe4f150e72)
node=3, no entry for 10.0.2.10 -- send arp request
ArpL3Protocol:SendArpRequest(0x57f5545dc0a0, 0x57f5545caa70, 10.0.2.10)
Ipv4L3Protocol:SelectSourceAddress(0x57f5545d88b0, 0x57f5545bfd10, 10.0.2.10, 2)
ARP: sending request from node 3 || src: 02-06-00:00:00:00:00:04 / 10.0.2.2 || dst: 02-06-ff:ff:ff:ff:ff:ff / 10.0.2.10
TapBridge:ReceiveFromBridgedDevice(0x57f55463dbe0, 0x57f5545d7ac0, 0x57f554624b50, 2054, 02-06-00:00:00:00:00:04, 02-06-ff:ff:ff:ff:ff:ff, 2)
Packet UID is 6
Writing packet to Linux host
Pkt source is 00:00:00:00:00:04
Pkt destination is ff:ff:ff:ff:ff:ff
Pkt LengthType is 2054
Pkt size is 60
End of receive packet handling on node 2
TapBridge:DoRead(0x57f554666560)
Calling read on tap device fd 8
TapBridge:ReadCallback(0x57f55463dbe0, , 42TapBridge:DiscardFromBridgedDevice(0x57f55463dbe0, 0x57f5545d7ac0, 0x57f554624b50, 2054, 02-06-)
00:00:00:TapBridge::ReadCallback(): Received packet on node 2
TapBridge::ReadCallback(): Scheduling handler
00:00:04)
Discarding packet stolen from bridged device 0x57f5545d7ac0
TapBridge:ForwardToBridgedDevice(0x57f55463dbe0, , 42)
Received packet from tap device
TapBridge:Filter(0x57f55463dbe0, 0x57f554624b50)
Pkt source is 02:42:ac:11:00:02
Pkt destination is 00:00:00:00:00:04
Pkt LengthType is 2054
Pkt source is 02-06-02:42:ac:11:00:02
Pkt destination is 02-06-00:00:00:00:00:04
Pkt LengthType is 2054
Forwarding packet to ns-3 device via Send()
ArpL3Protocol:Receive(0x57f5545dc0a0, 0x57f5545bfd10, 46, 2054, 02-06-02:50:5d:c7:98:46, 02-06-00:00:00:00:00:04, 0)
ARP: received packet of size 46
ArpL3Protocol:FindCache(0x57f5545dc0a0, 0x57f5545bfd10)
ARP: received reply node=3, got reply from 10.0.2.10 for address 10.0.2.2; we have addresses:
10.0.2.2,
node=3, got reply from 10.0.2.10 for waiting entry -- flush
ArpL3Protocol:Lookup(0x57f5545dc0a0, 0x57f554666da0, 10.0.2.10, 0x57f5545bfd10, 0x57f5545caa70, 0x7ffe4f1512d2)
node=3, alive entry for 10.0.2.10 valid -- send
NS_ASSERT failed, cond="m_current >= delta", +0.770387411s 3 file=/home/zero00/ns-allinone-3.42/ns-3.42/src/network/model/buffer.h, line=876
NS_FATAL, terminating
terminate called without an active exception
Command 'build/src/tap-bridge/examples/ns3.42-tap-csma-virtual-machine-default' died with <Signals.SIGABRT: 6>.


The weird thing is that I am not using any buffer functionality. So, any help would be very appreciated, I have been stuck on this for weeks
This is my implementation so far (not very good, but still a work in progress):

#include "ns3/core-module.h"

#include "ns3/network-module.h"

#include "ns3/tap-bridge-module.h"

#include "ns3/internet-module.h"

#include "ns3/point-to-point-module.h"

#include "ns3/ipv4-static-routing-helper.h"

#include "ns3/arp-cache.h"

#include "ns3/ipv4-l3-protocol.h"

#include "ns3/csma-module.h"

#include <fstream>

#include <iostream>

int main(int argc, char* argv[])

{

CommandLine cmd(__FILE__);

cmd.Parse(argc, argv);

// Enable logging for debugging

LogComponentEnable("ArpL3Protocol", LOG_LEVEL_ALL);

LogComponentEnable("Ipv4L3Protocol", LOG_LEVEL_ALL);

LogComponentEnable("TapBridge", LOG_LEVEL_ALL);

// LogComponentEnableAll(LOG_PREFIX_FUNC);

// LogComponentEnableAll(LOG_PREFIX_TIME);

// LogComponentEnableAll(LOG_PREFIX_NODE);

// LogComponentEnableAll(LOG_LEVEL_INFO);

// Bind real-time simulator and enable checksum calculations

GlobalValue::Bind("SimulatorImplementationType", StringValue("ns3::RealtimeSimulatorImpl"));

GlobalValue::Bind("ChecksumEnabled", BooleanValue(true));

// Create nodes: 3 hosts and 3 routers

NodeContainer nodes;

nodes.Create(6);

// Define node containers for connections

NodeContainer n1r1 = NodeContainer(nodes.Get(0), nodes.Get(1)); // Host 1 <-> Router 1

NodeContainer n2r2 = NodeContainer(nodes.Get(2), nodes.Get(3)); // Host 2 <-> Router 2

NodeContainer n3r3 = NodeContainer(nodes.Get(4), nodes.Get(5)); // Host 3 <-> Router 3

NodeContainer r1r2 = NodeContainer(nodes.Get(1), nodes.Get(3)); // Router 1 <-> Router 2

NodeContainer r1r3 = NodeContainer(nodes.Get(1), nodes.Get(5)); // Router 1 <-> Router 3

NodeContainer r2r3 = NodeContainer(nodes.Get(3), nodes.Get(5)); // Router 2 <-> Router 3

// Install Internet stack

InternetStackHelper stack;

stack.Install(nodes);

CsmaHelper csma;

csma.SetChannelAttribute("DataRate", StringValue("10Gbps"));

csma.SetChannelAttribute("Delay", StringValue("10ms"));

// Install Point-to-Point links

NetDeviceContainer dn1dr1 = csma.Install(n1r1);

NetDeviceContainer dn2dr2 = csma.Install(n2r2);

NetDeviceContainer dn3dr3 = csma.Install(n3r3);

// Configure Point-to-Point links

PointToPointHelper pointToPoint;

pointToPoint.SetDeviceAttribute("DataRate", StringValue("1Gbps"));

pointToPoint.SetChannelAttribute("Delay", StringValue("10ms"));

NetDeviceContainer dr1dr2 = pointToPoint.Install(r1r2);

NetDeviceContainer dr1dr3 = pointToPoint.Install(r1r3);

NetDeviceContainer dr2dr3 = pointToPoint.Install(r2r3);

// Assign IP addresses

Ipv4AddressHelper ipv4;

ipv4.SetBase("10.0.1.0", "255.255.255.0");

Ipv4InterfaceContainer ifHost1 = ipv4.Assign(dn1dr1);

ipv4.SetBase("10.0.2.0", "255.255.255.0");

Ipv4InterfaceContainer ifHost2 = ipv4.Assign(dn2dr2);

ipv4.SetBase("192.168.1.0", "255.255.255.0");

Ipv4InterfaceContainer ifHost3 = ipv4.Assign(dn3dr3);

ipv4.SetBase("10.1.1.0", "255.255.255.0");

Ipv4InterfaceContainer ifR1R2 = ipv4.Assign(dr1dr2);

ipv4.SetBase("10.1.2.0", "255.255.255.0");

Ipv4InterfaceContainer ifR1R3 = ipv4.Assign(dr1dr3);

ipv4.SetBase("10.1.3.0", "255.255.255.0");

Ipv4InterfaceContainer ifR2R3 = ipv4.Assign(dr2dr3);

Ipv4GlobalRoutingHelper::PopulateRoutingTables();

// This successfull

// Ptr<ArpCache> arp = CreateObject<ArpCache>();

// arp->SetAliveTimeout(Seconds(3600 * 24 * 365));

// ArpCache::Entry * entry = arp->Add(Ipv4Address ("10.0.2.10"));

// entry->SetMacAddress(Mac48Address("02:42:ac:11:00:02"));

// entry->MarkPermanent();

// std::pair<Ptr<Ipv4>, uint32_t> returnValue = ifHost2.Get(1);

// Ptr<Ipv4> myipv4 = returnValue.first;

// uint32_t index = returnValue.second;

// Ptr<Ipv4Interface> iface = myipv4->GetObject<Ipv4L3Protocol> ()->GetInterface (index);

// iface->SetArpCache(arp);

// Configure Tap Bridges

TapBridgeHelper tapBridge;

tapBridge.SetAttribute("Mode", StringValue("UseLocal"));

tapBridge.SetAttribute("DeviceName", StringValue("tap1"));

NetDeviceContainer tapbridge1 = tapBridge.Install(nodes.Get(0), dn1dr1.Get(0));

tapBridge.SetAttribute("DeviceName", StringValue("tap2"));

NetDeviceContainer tapbridge2 = tapBridge.Install(nodes.Get(2), dn2dr2.Get(0));

tapBridge.SetAttribute("DeviceName", StringValue("tap_rvs1"));

NetDeviceContainer tapbridge3 = tapBridge.Install(nodes.Get(4), dn3dr3.Get(0));

AsciiTraceHelper ascii;

csma.EnableAsciiAll (ascii.CreateFileStream ("test-docker.tr"));

// Run simulation

Simulator::Stop(Seconds(600.0));

Simulator::Run();

Simulator::Destroy();

return 0;

}

[Tommaso Pecorella]

This is interesting. It looks like a malformed ARP packet or a packet that we fail to decode correctly.

I'd love to see the packets being exchanged by the containers, can you dump them with tcpdump?

Thanks.

[Jubran Altaweel]

Thank you for the response. This is an attached pcap file
Just to add some clarification, for the sender docker container I explicitly specify the MAC address of the node 1 (00:00:00:00:00:02), but that seems to work just fine.

[Jubran Altaweel]

Sorry, forgot to include the sender pcap file.

[Tommaso Pecorella]

This is weird (it's an understatement): all the messages in the traces you posted should have been correctly decoded, and I can't find any call to "Prev", which is the function triggering the assert.

Hence, it must be a problem with ns-3 decoding a "real" packet, which is... basically quite difficult to debug. Alas, we need to.

I don't want to bore you to death, so please send here (or on zulip - https://ns-3.zulipchat.com) all the steps needed to reproduce your setup, we'll need to reproduce it.

Alternatively - or even better, please open an issue on our tracker, with the script and the steps to reproduce: https://gitlab.com/nsnam/ns-3-dev/-/issues

Thanks.

PS: we'll fix it in ns-3-dev.

[Jubran Altaweel]

An issue regarding this matter is created on gitlab https://gitlab.com/nsnam/ns-3-dev/-/issues/1171.

Thanks.