BBR abnormal behavior in LTE network

[Yuhan Zhou]

I'm new to ns3 so this could be my fault.
I'm using version 3.35 and trying to create an LTE network in a high-speed railway scenario. A single UE tries to upload a giant file to a remote server via eNodeBs. I have tried different congestion control algorithms such as NewReno and CUBIC and they work well.
The question is that BBR behaves abnormally in this scenario: `cwnd` drops to nearly zero, RTT increases quickly and stays at a high value. My simulation code is as below, most of it is copied from `src/lte/examples/lena-x2-handover.cc`

#include "ns3/core-module.h"
#include "ns3/network-module.h"
#include "ns3/internet-module.h"
#include "ns3/mobility-module.h"
#include "ns3/lte-module.h"
#include "ns3/applications-module.h"
#include "ns3/point-to-point-module.h"
#include "ns3/config-store-module.h"
#include "ns3/flow-monitor-module.h"
#include "ns3/traffic-control-module.h"
#include "ns3/mobility-building-info.h"
#include "ns3/buildings-propagation-loss-model.h"
#include "ns3/building.h"
#include "ns3/buildings-module.h"
#include <iostream>
#include <iomanip>

using namespace ns3;

NS_LOG_COMPONENT_DEFINE ("LenaX2HandoverMeasures");

// global configurations
uint16_t numberOfUes = 1;
uint16_t numberOfEnbs = 10;
uint16_t numBearersPerUe = 1;
double distance = 1000.0; // m
double yForEnb = 5; // m
double speed = 50; // m/s
double simTime = 20; // (double) numberOfEnbs * distance / speed + 5;
double enbTxPowerDbm = 36.0; // 46
double ueTxPowerDbm = 23.0;
uint32_t ftpSize = 1e9; // 1GB
uint16_t port = 4000; // port number

Time positionTraceInterval = Seconds (2);
Time ueReportInterval = Seconds (2);
Time reportProgressInterval = Seconds (5);

// experiment result streams
std::string expPrefix = "experiment-results";
std::ofstream ueMeasurements;
std::ofstream packetSinkRx;
std::ofstream cqiTrace;
std::ofstream tcpCongStateTrace;
std::ofstream positionTrace;
Ptr<OutputStreamWrapper> cWndStream;
Ptr<OutputStreamWrapper> ssThreshStream;
Ptr<OutputStreamWrapper> rttStream;
Ptr<OutputStreamWrapper> rtoStream;

// Parse context strings of the form "/NodeList/3/DeviceList/1/Mac/Assoc"
// to extract the NodeId
uint32_t
ContextToNodeId (std::string context)
{
  std::string sub = context.substr (10); // skip "/NodeList/"
  uint32_t pos = sub.find ("/Device");
  return atoi (sub.substr (0, pos).c_str ());
}

void
NotifyConnectionEstablishedUe (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " UE IMSI " << imsi
            << ": connected to CellId " << cellid << " with RNTI " << rnti << std::endl;
}

void
NotifyHandoverStartUe (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti,
                       uint16_t targetCellId)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " UE IMSI " << imsi
            << ": previously connected to CellId " << cellid << " with RNTI " << rnti
            << ", doing handover to CellId " << targetCellId << std::endl;
}

void
NotifyHandoverEndOkUe (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " UE IMSI " << imsi
            << ": successful handover to CellId " << cellid << " with RNTI " << rnti << std::endl;
}

void
NotifyConnectionEstablishedEnb (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " eNB CellId " << cellid
            << ": successful connection of UE with IMSI " << imsi << " RNTI " << rnti << std::endl;
}

void
NotifyHandoverStartEnb (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti,
                        uint16_t targetCellId)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " eNB CellId " << cellid
            << ": start handover of UE with IMSI " << imsi << " RNTI " << rnti << " to CellId "
            << targetCellId << std::endl;
}

void
NotifyHandoverEndOkEnb (std::string context, uint64_t imsi, uint16_t cellid, uint16_t rnti)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " eNB CellId " << cellid
            << ": completed handover of UE with IMSI " << imsi << " RNTI " << rnti << std::endl;
}

void
NotifyConnectionReleaseEnb (std::string context, const uint64_t imsi, const uint16_t cellId,
                            const uint16_t rnti)
{
  std::cout << Simulator::Now ().GetSeconds () << context << " eNB CellId " << cellId
            << ": release UE with IMSI " << imsi << " RNTI " << rnti << std::endl;
}

void
NotifyUeMeasurements (std::string context, uint16_t rnti, uint16_t cellId, double rsrp, double rsrq,
                      bool isServingCell, uint8_t componentCarrierId)

{
  ueMeasurements << std::setw (7) << std::setprecision (3) << std::fixed
                 << Simulator::Now ().GetSeconds () << " " << std::setw (3) << cellId << " "
                 << std::setw (3) << (isServingCell ? "1" : "0") << " " << std::setw (8) << rsrp
                 << " " << std::setw (8) << rsrq << std::endl;
}

void
NotifyPacketSinkRx (std::string context, Ptr<const Packet> packet, const Address &address)
{
  packetSinkRx << std::setw (7) << std::setprecision (3) << std::fixed
               << Simulator::Now ().GetSeconds () << " " << std::setw (5) << packet->GetSize ()
               << std::endl;
}

void
NotifyCqiReport (std::string context, uint16_t rnti, uint8_t cqi)
{
  cqiTrace << std::setw (7) << std::setprecision (3) << std::fixed
           << Simulator::Now ().GetSeconds () << " " << std::setw (4) << ContextToNodeId (context)
           << " " << std::setw (4) << rnti << " " << std::setw (3) << static_cast<uint16_t> (cqi)
           << std::endl;
}

void
NotifyCongState (const TcpSocketState::TcpCongState_t oldValue,
                 const TcpSocketState::TcpCongState_t newValue)
{
  tcpCongStateTrace << std::setw (7) << std::setprecision (3) << std::fixed
                    << Simulator::Now ().GetSeconds () << " " << std::setw (4)
                    << TcpSocketState::TcpCongStateName[newValue] << std::endl;
}

void
NotifyCwnd (uint32_t oldVal, uint32_t newVal)
{
  static bool firstCwnd = true;
  if (firstCwnd) {
    *cWndStream->GetStream () << "0.0 " << oldVal << std::endl;
    firstCwnd = false;
  } else {
    *cWndStream->GetStream () << Simulator::Now ().GetSeconds () << "   " << newVal << std::endl;
  }
}

void
NotifySsThresh (uint32_t oldVal, uint32_t newVal)
{
  static bool firstSsThresh = true;
  if (firstSsThresh) {
    *ssThreshStream->GetStream () << "0.0 " << oldVal << std::endl;
    firstSsThresh = false;
  } else {
    *ssThreshStream->GetStream () << Simulator::Now ().GetSeconds () << "   " << newVal << std::endl;
  }
}

void
NotifyRtt (Time oldVal, Time newVal)
{
  static bool firstRtt = true;
  if (firstRtt)
    {
      *rttStream->GetStream () << "0.0 " << oldVal.GetMilliSeconds () << std::endl;
      firstRtt = false;
    }
  *rttStream->GetStream () << Simulator::Now ().GetSeconds () << "    " << newVal.GetMilliSeconds () << std::endl;
}

void
NotifyRto (Time oldVal, Time newVal)
{
  static bool firstRto = true;
  if (firstRto)
    {
      *rtoStream->GetStream () << "0.0 " << oldVal.GetMilliSeconds () << std::endl;
      firstRto = false;
    }
  *rtoStream->GetStream () << Simulator::Now ().GetSeconds () << "    " << newVal.GetMilliSeconds () << std::endl;
}

void
ConnectTcpCongCallback ()
{
  Config::ConnectWithoutContext ("/NodeList/*/$ns3::TcpL4Protocol/SocketList/0/CongState",
                                 MakeCallback (&NotifyCongState));
}

void
ConnectCwndCallback()
{
  Config::ConnectWithoutContext("/NodeList/*/$ns3::TcpL4Protocol/SocketList/0/CongestionWindow", MakeCallback(&NotifyCwnd));
}

void
ConnectSsThreshCallback()
{
  Config::ConnectWithoutContext("/NodeList/*/$ns3::TcpL4Protocol/SocketList/0/SlowStartThreshold", MakeCallback(&NotifySsThresh));
}

void
ConnectRttCallback()
{
  Config::ConnectWithoutContext("/NodeList/*/$ns3::TcpL4Protocol/SocketList/0/RTT", MakeCallback(&NotifyRtt));
}

void
ConnectRtoCallback()
{
  Config::ConnectWithoutContext("/NodeList/*/$ns3::TcpL4Protocol/SocketList/0/RTO", MakeCallback(&NotifyRto));
}

// function to get the position scheduled every 2 seconds and print it in the output
void
TracePosition (NodeContainer nodes, Time interval)
{

  Ptr<MobilityModel> VM;

  // Vector pos;
  Ptr<Node> node;
  node = nodes.Get (0);

  VM = nodes.Get (0)->GetObject<ConstantVelocityMobilityModel> ();
  positionTrace << Simulator::Now ().GetSeconds () << " " << VM->GetPosition ().x << ","
                << VM->GetPosition ().y << "," << VM->GetPosition ().z << std::endl;

  Simulator::Schedule (interval, &TracePosition, nodes, interval);
}

void
Progress (Time interval)
{
  std::cout << "*** " << Simulator::Now ().GetSeconds () << "/" << simTime << " ***" << std::endl;
  Simulator::Schedule (interval, &Progress, interval);
}

/**
 * Sample simulation script for an automatic X2-based handover based on the RSRQ measures.
 * It instantiates two eNodeB, attaches one UE to the 'source' eNB.
 * The UE moves between eNBs, it reports measures to the serving eNB and
 * the 'source' (serving) eNB triggers the handover of the UE towards
 * the 'target' eNB when it considers it is a better eNB.
 */
int
main (int argc, char *argv[])
{
  // Command line arguments
  CommandLine cmd (__FILE__);
  cmd.AddValue ("simTime", "Total duration of the simulation (in seconds)", simTime);
  cmd.AddValue ("speed", "Speed of the UE (default = 50 m/s)", speed);
  cmd.AddValue ("enbTxPowerDbm", "TX power [dBm] used by HeNBs (default = 36.0)", enbTxPowerDbm);

  cmd.Parse (argc, argv);

  // open trace files
  ueMeasurements.open ((expPrefix + "/ue-measurements.dat").c_str (), std::ofstream::out);
  ueMeasurements << "# time   cellId   isServingCell?  RSRP(dBm)  RSRQ(dB)" << std::endl;
  packetSinkRx.open ((expPrefix + "/tcp-receive.dat").c_str (), std::ofstream::out);
  packetSinkRx << "# time   bytesRx" << std::endl;
  cqiTrace.open ((expPrefix + "/cqi.dat").c_str (), std::ofstream::out);
  cqiTrace << "# time   nodeId   rnti  cqi" << std::endl;
  tcpCongStateTrace.open ((expPrefix + "/tcp-state.dat").c_str (), std::ofstream::out);
  tcpCongStateTrace << "# time   congState" << std::endl;
  positionTrace.open ((expPrefix + "/position.dat").c_str (), std::ofstream::out);
  positionTrace << "# time   position" << std::endl;
  AsciiTraceHelper ascii;
  cWndStream = ascii.CreateFileStream((expPrefix + "/cwnd.dat").c_str());
  ssThreshStream = ascii.CreateFileStream((expPrefix + "/ssthresh.dat").c_str());
  rttStream = ascii.CreateFileStream((expPrefix + "/rtt.dat").c_str());
  rtoStream = ascii.CreateFileStream((expPrefix + "/rto.dat").c_str());
 
  // set TCP CC algorithm, TcpCubic works well
  Config::SetDefault ("ns3::TcpL4Protocol::SocketType", TypeIdValue (TcpBbr::GetTypeId ()));

  Ptr<LteHelper> lteHelper = CreateObject<LteHelper> ();
  Ptr<PointToPointEpcHelper> epcHelper = CreateObject<PointToPointEpcHelper> ();
  lteHelper->SetEpcHelper (epcHelper);
  lteHelper->SetSchedulerType ("ns3::RrFfMacScheduler");

  // This controls the UE measured RSRP to be -90dbm ~ -110dbm
  lteHelper->SetAttribute ("PathlossModel", StringValue ("ns3::LogDistancePropagationLossModel"));
  Config::SetDefault ("ns3::LogDistancePropagationLossModel::ReferenceLoss", DoubleValue (97));
  Config::SetDefault ("ns3::LogDistancePropagationLossModel::ReferenceDistance", DoubleValue (5));
  Config::SetDefault ("ns3::LogDistancePropagationLossModel::Exponent", DoubleValue (1.5));

  lteHelper->SetHandoverAlgorithmType ("ns3::A3RsrpHandoverAlgorithm");
  lteHelper->SetHandoverAlgorithmAttribute ("Hysteresis", DoubleValue (3.0));
  lteHelper->SetHandoverAlgorithmAttribute ("TimeToTrigger", TimeValue (MilliSeconds (256)));

  Ptr<Node> pgw = epcHelper->GetPgwNode ();

  // Create a single RemoteHost
  NodeContainer remoteHostContainer;
  remoteHostContainer.Create (1);
  Ptr<Node> remoteHost = remoteHostContainer.Get (0);
  InternetStackHelper internet;
  internet.Install (remoteHostContainer);

  // Create the Internet
  PointToPointHelper p2ph;
  p2ph.SetDeviceAttribute ("DataRate", DataRateValue (DataRate ("20Mb/s")));
  p2ph.SetDeviceAttribute ("Mtu", UintegerValue (1500));
  p2ph.SetChannelAttribute ("Delay", TimeValue (Seconds (0.010)));
  p2ph.SetQueue ("ns3::DropTailQueue", "MaxSize", StringValue ("200p")); // about 100kb buffer
  NetDeviceContainer internetDevices = p2ph.Install (pgw, remoteHost);

  // configure IP
  Ipv4AddressHelper ipv4h;
  ipv4h.SetBase ("1.0.0.0", "255.0.0.0");
  Ipv4InterfaceContainer internetIpIfaces = ipv4h.Assign (internetDevices);

  // Routing of the Internet Host (towards the LTE network)
  Ipv4StaticRoutingHelper ipv4RoutingHelper;
  Ptr<Ipv4StaticRouting> remoteHostStaticRouting =
      ipv4RoutingHelper.GetStaticRouting (remoteHost->GetObject<Ipv4> ());
  // interface 0 is localhost, 1 is the p2p device
  remoteHostStaticRouting->AddNetworkRouteTo (Ipv4Address ("7.0.0.0"), Ipv4Mask ("255.0.0.0"), 1);

  // log remote host IP
  Ipv4Address remoteAddr = remoteHost->GetObject<Ipv4> ()->GetAddress (1, 0).GetLocal ();
  std::cout << "remoteHost Ipv4 Address: " << remoteAddr << std::endl;

  NodeContainer ueNodes;
  NodeContainer enbNodes;
  enbNodes.Create (numberOfEnbs);
  ueNodes.Create (numberOfUes);

  // Install Mobility Model in eNB
  Ptr<ListPositionAllocator> enbPositionAlloc = CreateObject<ListPositionAllocator> ();
  for (uint16_t i = 0; i < numberOfEnbs; i++)
    {
      Vector enbPosition (distance * (i + 1), yForEnb, 0);
      enbPositionAlloc->Add (enbPosition);
      std::cout << "eNodeB " << i << ": " << enbPosition.x << ", " << enbPosition.y << ", "
                << enbPosition.z << std::endl;
    }
  MobilityHelper enbMobility;
  enbMobility.SetMobilityModel ("ns3::ConstantPositionMobilityModel");
  enbMobility.SetPositionAllocator (enbPositionAlloc);
  enbMobility.Install (enbNodes);

  // Install Mobility Model in UE
  MobilityHelper ueMobility;
  // ueMobility.SetMobilityModel("ns3::ConstantPositionMobilityModel");
  ueMobility.SetMobilityModel ("ns3::ConstantVelocityMobilityModel");
  ueMobility.Install (ueNodes);
  ueNodes.Get (0)->GetObject<MobilityModel> ()->SetPosition (Vector (distance / 2, 0, 0));
  ueNodes.Get (0)->GetObject<ConstantVelocityMobilityModel> ()->SetVelocity (Vector (speed, 0, 0));
 
  // Install LTE Devices in eNB and UEs
  Config::SetDefault ("ns3::LteEnbPhy::TxPower", DoubleValue (enbTxPowerDbm));
  Config::SetDefault ("ns3::LteUePhy::TxPower", DoubleValue (ueTxPowerDbm));
  NetDeviceContainer enbLteDevs = lteHelper->InstallEnbDevice (enbNodes);
  NetDeviceContainer ueLteDevs = lteHelper->InstallUeDevice (ueNodes);

  // Install the IP stack on the UEs
  internet.Install (ueNodes);
  Ipv4InterfaceContainer ueIpIfaces;
  ueIpIfaces = epcHelper->AssignUeIpv4Address (NetDeviceContainer (ueLteDevs));
  lteHelper->Attach (ueLteDevs);

  NS_LOG_LOGIC ("setting up applications");

  Ptr<Ipv4StaticRouting> ueStaticRouting =
      ipv4RoutingHelper.GetStaticRouting (ueNodes.Get (0)->GetObject<Ipv4> ());
  ueStaticRouting->SetDefaultRoute (epcHelper->GetUeDefaultGatewayAddress (), 1);
  BulkSendHelper ftpServer ("ns3::TcpSocketFactory", Address ());

  AddressValue sinkAddress (InetSocketAddress (remoteAddr, port));
  ftpServer.SetAttribute ("Remote", sinkAddress);
  ftpServer.SetAttribute ("MaxBytes", UintegerValue (ftpSize));
  NS_LOG_LOGIC ("setting up TCP flow from UE to remote host");
  ApplicationContainer sourceApp = ftpServer.Install (ueNodes.Get(0));
  sourceApp.Start (Seconds (1));
  sourceApp.Stop (Seconds (simTime));

  Address sinkLocalAddress (InetSocketAddress (Ipv4Address::GetAny (), port));
  PacketSinkHelper sinkHelper ("ns3::TcpSocketFactory", sinkLocalAddress);
  ApplicationContainer sinkApp = sinkHelper.Install (remoteHost);
  sinkApp.Start (Seconds (1));
  sinkApp.Stop (Seconds (simTime));

  Ptr<EpcTft> tft = Create<EpcTft> ();
  EpcTft::PacketFilter dlpf;
  dlpf.localPortStart = port;
  dlpf.localPortEnd = port;
  tft->Add (dlpf);
  EpsBearer bearer (EpsBearer::NGBR_VIDEO_TCP_DEFAULT);
  lteHelper->ActivateDedicatedEpsBearer (ueLteDevs.Get (0), bearer, tft);


  // Add X2 interface
  lteHelper->AddX2Interface (enbNodes);

  lteHelper->EnablePhyTraces ();
  lteHelper->EnableMacTraces ();
  lteHelper->EnableRlcTraces ();
  lteHelper->EnablePdcpTraces ();

  // for calculating throughput
  Ptr<RadioBearerStatsCalculator> rlcStats = lteHelper->GetRlcStats ();
  rlcStats->SetAttribute ("EpochDuration", TimeValue (Seconds (1.0)));
  Ptr<RadioBearerStatsCalculator> pdcpStats = lteHelper->GetPdcpStats ();
  pdcpStats->SetAttribute ("EpochDuration", TimeValue (Seconds (1.0)));

  // connect custom trace sinks for RRC connection establishment and handover notification
  Config::Connect ("/NodeList/*/DeviceList/*/LteEnbRrc/ConnectionEstablished",
                   MakeCallback (&NotifyConnectionEstablishedEnb));
  Config::Connect ("/NodeList/*/DeviceList/*/LteUeRrc/ConnectionEstablished",
                   MakeCallback (&NotifyConnectionEstablishedUe));
  Config::Connect ("/NodeList/*/DeviceList/*/LteEnbRrc/HandoverStart",
                   MakeCallback (&NotifyHandoverStartEnb));
  Config::Connect ("/NodeList/*/DeviceList/*/LteUeRrc/HandoverStart",
                   MakeCallback (&NotifyHandoverStartUe));
  Config::Connect ("/NodeList/*/DeviceList/*/LteEnbRrc/HandoverEndOk",
                   MakeCallback (&NotifyHandoverEndOkEnb));
  Config::Connect ("/NodeList/*/DeviceList/*/LteUeRrc/HandoverEndOk",
                   MakeCallback (&NotifyHandoverEndOkUe));
  Config::Connect ("/NodeList/*/DeviceList/*/LteEnbRrc/NotifyConnectionRelease",
                   MakeCallback (&NotifyConnectionReleaseEnb));
  Config::Connect ("/NodeList/*/DeviceList/*/$ns3::LteUeNetDevice/ComponentCarrierMapUe/*/LteUePhy/"
                   "ReportUeMeasurements",
                   MakeCallback (&NotifyUeMeasurements));
  Config::Connect ("/NodeList/*/ApplicationList/*/$ns3::PacketSink/Rx",
                   MakeCallback (&NotifyPacketSinkRx));

  // trace simulation progress
  Simulator::Schedule (Seconds (0), &TracePosition, ueNodes, positionTraceInterval);
  Simulator::Schedule (Seconds (0), &Progress, reportProgressInterval);
  // Delay trace connection until TCP socket comes into existence
  Simulator::Schedule (Seconds (1.001), &ConnectTcpCongCallback);
  Simulator::Schedule (Seconds (1.001), &ConnectCwndCallback);
  Simulator::Schedule (Seconds (1.001), &ConnectSsThreshCallback);
  Simulator::Schedule (Seconds (1.001), &ConnectRttCallback);
  Simulator::Schedule (Seconds (1.001), &ConnectRtoCallback);

  Simulator::Stop (Seconds (simTime));
  // trace server and client packets
  p2ph.EnablePcap("server-trace", remoteHostContainer);
  p2ph.EnablePcap("client-trace", enbNodes);
  Simulator::Run ();

  Simulator::Destroy ();
  ueMeasurements.close ();
  packetSinkRx.close ();
  positionTrace.close ();
  tcpCongStateTrace.close ();
  cqiTrace.close ();

  return 0;
}

So my question is: what is the cause of this BBR behavior? Probably my setup has some defects but I wonder if there are any problems with the LTE model and BBR implementation in ns3. Besides here are some figure results from the simulation 

[苏琪琛]

Have you solved this problem yet? I also encountered the same problem, maybe I can leave a contact information to facilitate communication. 

qichen...@gmail.com

[Tom Henderson]

ns-3 TCP BBR has a number of open issues (reported bugs) in the tracker
and the maintainer has been unable to devote the time to fix them so
far. I hope that we can fix them sometime this summer.

- Tom

> --
> Posting to this group should follow these guidelines
> https://www.nsnam.org/wiki/Ns-3-users-guidelines-for-posting
> <https://www.nsnam.org/wiki/Ns-3-users-guidelines-for-posting>
> ---
> You received this message because you are subscribed to the Google
> Groups "ns-3-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ns-3-users+...@googlegroups.com
> <mailto:ns-3-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ns-3-users/deab8e4c-08f2-43e3-baef-5fc857e20bcan%40googlegroups.com <https://groups.google.com/d/msgid/ns-3-users/deab8e4c-08f2-43e3-baef-5fc857e20bcan%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Mihai Mazilu]

i am seeing the same behaviour in non LTE simulations, the high RTT is dude to the dup ack count being 1 and the dup ack timeout being 200ms combined with the slow sending rare which inflate the RTTs. As for the low unexpected sending rate, i have observed it happening with low BDP queues (< 2BDP).