TCP aggregate throughput more than specified bandwidth on the link

[Mohsen Ansari]

Hi ns-3 community,

I've been working on a simulation for quite a while where I have 3 nodes and 2 links to connect them. My first link is 100Mbps and the second one is 10Mbps.

When I run the simulation and make multiple apps (tcp flows) to send data and use flow monitor to capture the throughput, sometimes the aggregate throughput is more than 10Mbps on the receiving end. 

I don't know why it happens and I double and triple checked my throughput calculation and nothing seems to be wrong. Here are some parts of my code:

Network Topology

//
//           100Mb/s, 20ms       10Mb/s, 5ms
//       n0-----------------------n1-------------------------n2


PointToPointHelper leftlink;
leftlink.SetDeviceAttribute ("DataRate", StringValue ("100Mbps"));
leftlink.SetChannelAttribute ("Delay", StringValue ("20ms"));


PointToPointHelper rightlink;
rightlink.SetDeviceAttribute ("DataRate", StringValue ("10Mbps"));
rightlink.SetChannelAttribute ("Delay", StringValue ("5ms"));

creating multiple sink and source apps

for (int port=10; port < 10+tcpflows; port++){
 //sink application
 PacketSinkHelper sink ("ns3::TcpSocketFactory", InetSocketAddress (Ipv4Address::GetAny (), port));


 ApplicationContainer sinkApps = sink.Install (n1n2.Get (1));
   sinkApps.Start (Seconds (1.0));
   sinkApps.Stop (Seconds (20.0));


 //source(s) application
 ApplicationContainer sourceApps;
 
 //loop for creating multiple applications (flows) from source
 BulkSendHelper sources("ns3::TcpSocketFactory", InetSocketAddress (interface12.GetAddress (1), port));  
 //Set the amount of data to send in bytes.  Zero is unlimited.
 sources.SetAttribute ("SendSize", UintegerValue (tcp_app_data_unit_size));
 sources.SetAttribute ("MaxBytes", UintegerValue (maxBytes));
 sourceApps = sources.Install (n0n1.Get (0));
 int start = 0.0+(port - 9);
 sourceApps.Start (Seconds (start));
 sourceApps.Stop (Seconds (20.0));
}

Throughput calculations

total_throughput = 0;
for (std::map<FlowId, FlowMonitor::FlowStats>::const_iterator i = stats.begin (); i != stats.end (); ++i){
 
    Ipv4FlowClassifier::FiveTuple t = classifier->FindFlow (i->first);
        if ((t.sourceAddress=="10.1.1.1" && t.destinationAddress == "10.1.2.2")){
            double flow_throughput = (i->second.rxBytes * 8.0) / (i->second.timeLastRxPacket.GetSeconds() - i->second.timeFirstTxPacket.GetSeconds());
            flow_throughput /= 1024;
            flow_throughput /= 1024;
 
            total_throughput += flow_throughput;
 
            std::cout << "Flow " << i->first  << " (" << t.sourceAddress << " -> " << t.destinationAddress << ")\n";
            std::cout << "  Tx Bytes:   " << i->second.txBytes << "\n";
            std::cout << "  Rx Bytes:   " << i->second.rxBytes << "\n";
            std::cout << "  Throughput: " << flow_throughput  << " Mbps\n";
        }
}

For example, when I set 8 multiple apps and run the simulation, I get 10.5915 Mbps as total throughput while I set the link rate to 10 Mbps.
Any ideas why is this happening? any help would be truly appreciated.

Thanks in advance.

[Tommaso Pecorella]

Hi,

well, I can spot two errors in your calc, but they make things better than they should (i.e., fixing the problem will make your measured throughput even higher).

One: you're calculating the throughput in Mibps instead of Mbps (i.e., dividing by 1024 instead of 1000).
Two: the time delta the throughput is calculated on should be from the first received packet to the last received packet (not from the first transmitted packet).

However, if you redo the calc right, the numbers will be higher.

About what's the problem... no idea. Without the full code it's impossible to say.

T.

[Mohsen Ansari]

//
// Network topology

//
//           100Mb/s, 20ms       10Mb/s, 5ms

//       n0-----------------n1-----------------n2
//
//
// - Tracing of queues and packet receptions to file
//   flow monitor used for the links between n0 and n2
// - pcap traces also generated in the following files
//   "bottleneckrouter-$n-$i.pcap" where n and i represent node and interface
// numbers respectively
//  Usage (e.g.): ./waf --run path-to-containing-folder/bottleneckrouter


#include <iostream>
#include <fstream>
#include <string>
#include <iomanip>      // std::setw


#include "ns3/netanim-module.h"
#include "ns3/core-module.h"
#include "ns3/applications-module.h"
#include "ns3/network-module.h"
#include "ns3/internet-module.h"
#include "ns3/point-to-point-module.h"
#include "ns3/ipv4-global-routing-helper.h"
#include "ns3/flow-monitor-helper.h"
#include "ns3/error-model.h"
#include "ns3/netanim-module.h"
#include "ns3/queue.h"
#include "ns3/rng-seed-manager.h"
#include "ns3/ipv4-flow-classifier.h"


using namespace ns3;


NS_LOG_COMPONENT_DEFINE ("bottleneckrouter");


///////////// variables /////////////////


 int tcpflows = 1; //number of tcp flows default = 1
 int maxBytes = 0; //setting up max number of bytes to send default = 0, 0 is unlimited
 double lossp = 0.01;  //loss probability default = 0.01  -  1%
 int queue_size = 1000; // Queue size at node 1 (router) default 1000 packets
 int seed = 1; // Seed number for random number generator default = 1
 int tcp_app_data_unit_size = 1460;


int main(int argc, char *argv[]){


////////// logging and command line arguments //////////


 //enable logging module
 LogComponentEnable("bottleneckrouter", LOG_LEVEL_LOGIC);  


 //enable commandline argument passing
 CommandLine cmd;
 cmd.AddValue ("tcpflows", "number of tcp flows", tcpflows);
 cmd.AddValue ("lossp", "loss probability between 0.0 and 1.0", lossp);
 cmd.AddValue ("seed", "seed number for random number generator", seed);
 cmd.AddValue ("queue", "queue size of the node in between", queue_size);
 cmd.Parse (argc, argv);


////////////// global configurations /////////////////////


 //setting up seed number for random number generator
 Config::SetGlobal ("RngSeed", IntegerValue(seed));
 //setting up tcp configurations
 Config::SetDefault ("ns3::TcpL4Protocol::SocketType", TypeIdValue (TcpNewReno::GetTypeId ()));
 Config::SetDefault ("ns3::TcpSocket::SegmentSize", UintegerValue (1500));


////////////// creating topology ///////////////


 //creating 2 node containers one for left side one for right side
 NodeContainer n0n1;
 n0n1.Create(2);


 NodeContainer n1n2;
 n1n2.Add(n0n1.Get(1));
 n1n2.Create(1);


 //setting up error model for simulating loss rate
 Ptr<UniformRandomVariable> uv = CreateObject<UniformRandomVariable> ();
 uv->SetStream (100);
 RateErrorModel error_model;
 error_model.SetRandomVariable (uv);
 error_model.SetUnit (RateErrorModel::ERROR_UNIT_PACKET);
 error_model.SetRate (lossp); //error rate


 //defining the 2 point to point links between the nodes

 PointToPointHelper leftlink;
 leftlink.SetDeviceAttribute ("DataRate", StringValue ("100Mbps"));
 leftlink.SetChannelAttribute ("Delay", StringValue ("20ms"));
 


 PointToPointHelper rightlink;
 rightlink.SetDeviceAttribute ("DataRate", StringValue ("10Mbps"));
 rightlink.SetChannelAttribute ("Delay", StringValue ("5ms"));

 rightlink.SetDeviceAttribute ("ReceiveErrorModel", PointerValue(&error_model));  //assigning the error model to the link


 //installing devices for nodes 0, 1
 NetDeviceContainer devices01;
 devices01 = leftlink.Install(n0n1);


 //installing devices for nodes 1, 2
 NetDeviceContainer devices12;
 devices12 = rightlink.Install(n1n2);


 //setting up queue sizes for nodes
 //std::cout<<queue_size<<"\n";
 Ptr<DropTailQueue> queue = DynamicCast<DropTailQueue> (DynamicCast<PointToPointNetDevice> (devices01.Get (1))->GetQueue ()); //middle node
 queue -> SetAttribute("MaxPackets", UintegerValue(1000));


 DynamicCast<PointToPointNetDevice> (devices01.Get (1))->SetQueue (queue);


  Ptr<DropTailQueue> endqueue = DynamicCast<DropTailQueue> (DynamicCast<PointToPointNetDevice> (devices12.Get (0))->GetQueue ()); //middle right side node
 endqueue -> SetAttribute("MaxPackets", UintegerValue(queue_size));


 DynamicCast<PointToPointNetDevice> (devices12.Get (0))->SetQueue (endqueue);


 // installing IP stack
 InternetStackHelper stack;
 stack.InstallAll();


 //defining addresses
 Ipv4AddressHelper address;
 
 //defining address for nodes 0, 1
 address.SetBase ("10.1.1.0", "255.255.255.0");
 Ipv4InterfaceContainer interface01 = address.Assign(devices01);


 //defining addresses for nodes 1, 2
 address.SetBase ("10.1.2.0", "255.255.255.0");
 Ipv4InterfaceContainer interface12 = address.Assign(devices12);


 //applying global routing
 Ipv4GlobalRoutingHelper::PopulateRoutingTables ();


////////////////// creating application //////////////////////

 for (int port=10; port < 10+tcpflows; port++){
 //sink application
 PacketSinkHelper sink ("ns3::TcpSocketFactory", InetSocketAddress (Ipv4Address::GetAny (), port));


 ApplicationContainer sinkApps = sink.Install (n1n2.Get (1));
   sinkApps.Start (Seconds (1.0));
   sinkApps.Stop (Seconds (20.0));


 //source(s) application
 ApplicationContainer sourceApps;
 
 //loop for creating multiple applications (flows) from source
 BulkSendHelper sources("ns3::TcpSocketFactory", InetSocketAddress (interface12.GetAddress (1), port));  
 //Set the amount of data to send in bytes.  Zero is unlimited.
 sources.SetAttribute ("SendSize", UintegerValue (tcp_app_data_unit_size));
 sources.SetAttribute ("MaxBytes", UintegerValue (maxBytes));
 sourceApps = sources.Install (n0n1.Get (0));
 int start = 0.0+(port - 9);
 sourceApps.Start (Seconds (start));
 sourceApps.Stop (Seconds (20.0));
 }

////////////////////// utilities //////////////////////////


 //using flow monitor
 Ptr<FlowMonitor> flowMonitor;
 FlowMonitorHelper flowHelper;
 flowMonitor = flowHelper.InstallAll();


 //using pcap tracing
 rightlink.EnablePcapAll("rightlink");
        leftlink.EnablePcapAll("leftlink");
 std::ostringstream losspstr;
        std::ostringstream tcpflowsstr;
 std::ostringstream qsizestr;
        losspstr << lossp;
        tcpflowsstr << tcpflows;
 qsizestr << queue_size;
        leftlink.EnablePcapAll ("bottleneckrouter-p: "+losspstr.str()+"-n: "+tcpflowsstr.str(), false);


 //run the simlulation
 Simulator::Stop(Seconds(20.0));
 Simulator::Run();




/////////////////////////making results file//////////////////
 std::ofstream file;
 file.open("results.txt", std::ios_base::app);
 file <<std::left<<std::setw(18)<<queue_size <<","<< std::setw(18)<< lossp <<","<< std::setw(18) << tcpflows <<","<< std::setw(18) << seed <<","<< std::setw(18);
 
            //calculating the throughput
 Ptr<Ipv4FlowClassifier> classifier = DynamicCast<Ipv4FlowClassifier> (flowHelper.GetClassifier ());
        std::map<FlowId, FlowMonitor::FlowStats> stats = flowMonitor->GetFlowStats ();


 double total_throughput = 0;

    for (std::map<FlowId, FlowMonitor::FlowStats>::const_iterator i = stats.begin (); i != stats.end (); ++i){
 
    Ipv4FlowClassifier::FiveTuple t = classifier->FindFlow (i->first);
        if ((t.sourceAddress=="10.1.1.1" && t.destinationAddress == "10.1.2.2")){
    double flow_throughput = (i->second.rxBytes * 8.0) / (i->second.timeLastRxPacket.GetSeconds() - i->second.timeFirstTxPacket.GetSeconds());
    flow_throughput /= 1024;
    flow_throughput /= 1024;
            total_throughput += flow_throughput;
 
            std::cout << "Flow " << i->first  << " (" << t.sourceAddress << " -> " << t.destinationAddress << ")\n";
            std::cout << "  Tx Bytes:   " << i->second.txBytes << "\n";
            std::cout << "  Rx Bytes:   " << i->second.rxBytes << "\n";
         std::cout << "  Throughput: " << flow_throughput  << " Mbps\n";
        }
     }

 
 file << total_throughput << ",\n";
 file.close();


//////////////////////////// end test code//////////////////////////////////////


 //saving flow monitor results before destroying simulator
 flowMonitor->SerializeToXmlFile("flowmonitor-p: "+losspstr.str()+"-n: "+tcpflowsstr.str()+" q: "+qsizestr.str()+".xml", true, true);


 Simulator::Destroy();
 




 return 0;
}

Hi Tom,

Thanks for reply. I am aware of those errors you mentioned but when I got the throughput more than  specified bandwidth I thought I'm calculating throughput incorrectly so I changed to the way they are now.

I put my code as you can see above. When I run the code for arguments I give queue_size=100, loss=0, tcp_flows=8, seed=1

seed number isn't important here since my loss rate is 0 and I don't need any random numbers to generate.

[Tommaso Pecorella]

Hi,

beside the fact that you had to attach the script, not copy-paste it (but at least it was formatted), your problem made me think a lot.

Let me just say that you helped find a bug (a very minor one, it doesn't affect the results) and that the problem solution is all but obvious. Alas, your script seems right, but it isn't (obviously).

Let me go through some minor stuff first. The TCP segment size. You did set it to 1500, but the actual TCP and IP headers are to be considered. Your packet size (at NetDevice) was going up to 1552 bytes, so the real optimum segment size is 1448. Of course this optimization will increase the throughput... but we have to face the problems, not hide them.

And now your mistake. A silly one, once you know it, but we're all wise after.

Throughput. Define it. Number of bytes per seconds (or bits, or megabits, or whatever).

Fine. Now, suppose to have a 10Mbps link, and two apps, one starting at second 10 and stopping at second 20, the other starting at second 30 and stopping at second 40. Each one is using the link at 8Mbps - below its capacity.

What does it happens if you sum the two throughputs ? -> 16 Mbps - not right.

Of course, they was using the link at different moments.

And in your case ?

It's the same thing, you can not sum two (or more) throughputs, you have to measure the sum of the flows.

If you sum all the received bytes and you divide by the time used by *all* the flows you'll end up with more reasonable numbers - 8.4198 Mbps - but even this is not totally right. You should consider blank periods (where no packet is exchanged) if you are really interested in the right number. Fortunately, in your case you have a continuous transmission, but if you have long silence periods it could be necessary. Otherwise you'll end up underestimating the link occupancy.

Cheers,

T.

[Mohsen Ansari]

Hi Tom,

Thank you so much for taking the time and replying. I fixed the code and now the numbers make sense and I feel so silly not to notice the problem.

As you said my script seemed right and it made sense to me when I was writing it.

On a side note, the very first reason I considered starting my source apps on different times is that when I was starting them at the same time or starting them with a small time difference (like at 0.0, 0.1, 0.2 seconds), I was getting a throughput close to zero.

Anyway thanks a lot for your help I'll mark this thread as completed.

[Tommaso Pecorella]

Hi Mohsen,

as I said the problem wasn't that obvious. For what I can remember, the fact that throughput isn't something you can add isn't something you're warned a lot of times in your networking courses or books. On the contrary, we do that a lot of times without worrying too much, and perhaps we should.

About starting apps at different times, you're right in doing that. If you start them all together they'll do an ARP storm, some ARP will be lost and connections will fail.

Moreover, if you generate too many packets while ARP is finding an address, you'll loose some of them, again leading to failing connections.

What you could do is to add static ARP entries (not really realistic, but you can) or increase the ARP queue (more or less realistic, it has been discussed in the group).

Moreover, you can put less delay between the apps start (slightly more than two times the network delay to the next hop, that's enough to fill the ARP cache).

Last but not least, you can increase the simulation precision by increasing the simulation time. However, in that case I'd suggest to use the very latest ns-3-dev, as it has a patch to provide more realistic ARP entries invalidation timers.

Cheers,

T.