[LENA] UL traffic slows down DL in AM mode

[Xing Xu]

Dear all,

I found a strange observation may potentially due to a bug in AM mode of LENA: AFAIK, UL would not interfere with DL (UL and DL should be independent), but I found that UL traffic (even really light, like 3 UL packets per second) can slow down DL rate in AM mode.

Experiment configuration:

I have only 1 BS (no interference, no fading), collocated 30 UEs (1000m to the BS) paired with 30 RH, and on DL those 30 RHs are sending enough UDP packets (1 per ms) to congest the base station.

Now I add different level of UL traffic to these UEs (I tried no UL packet, 1 UL packet per 100ms, 1 UL packet per 300ms and 1 UL packet per 500ms).

Result:

When there is no UL traffic, UEs' DL rate is at ~73000B/s (see attached figures, figure file names start with "am" or "um", and followed by UL traffic frequency, e.g., "_ul_100" shows that UL traffic is 1 packet per 100ms).

When UL traffic is 1 packet per 500ms, UEs' DL rate is at ~65000B/s

When UL traffic is 1 packet per 300ms, UEs' DL rate is at ~60000B/s

When UL traffic is 1 packet per 100ms, UEs' DL rate is at ~30000B/s

And in all above cases when there is UL traffic, DL rate becomes more fluctuated.

However, when I'm using UM mode, I don't see this issue and DL rate is the same for different UL traffic (so UM seems OK, UL and DL are indeed independent).

Script and usage:

I attached my script, mr_bidirection.cc. There are two parameters, "ack" specifies AM or UM (--ack=1 for AM and --ack=0 for UM); "interval_ul" specifies the frequency of UL packet, e.g., --interval_ul=100 makes UEs' UL send one packet per 100ms, --interval_ul=0 means no UL traffic at all.

Sample run: ./waf --run="scratch/mr_bidirectional --ack=1 --interval_ul=100"

Thanks,

Xing

[Xing Xu]

Anyone know if this is normal? Or is there anything I can do to fix this?

[Lorenza Giupponi]

Hi,

when you consider RLC-AM UL and DL are not exactly decoupled. I mean, they are from interference point of view, because as you say you are considering a FDD implementation, but to proceed with transmissions in DL you need to receive STATUS PDU (i.e. RLC ACKs) from UL and viceversa. So, my guess is that in DL now you do not only have to transmit data, but also STATUS PDU to ACK the reception of UL data. STATUS PDUs always have priority on data, and in LENA implementation they are not piggybacked to data, i.e. they are sent alone, without data. So I think this may slow down the transmission of data in DL and consequently reduce the throughput. More data you send in UL, more ACK you will need to send in DL...You should check in traces to see if your reduced throughout is due to that.

Lorenza

--
Posting to this group should follow these guidelines https://www.nsnam.org/wiki/Ns-3-users-guidelines-for-posting
---
You received this message because you are subscribed to the Google Groups "ns-3-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ns-3-users+...@googlegroups.com.
To post to this group, send email to ns-3-...@googlegroups.com.
Visit this group at https://groups.google.com/group/ns-3-users.
For more options, visit https://groups.google.com/d/optout.

[Xing Xu]

Thanks a lot. I have a couple of related questions:

1) Can you let me know how do I check the reason is due to STATUS PDU?

2) If it's the issue for my experiment, isn't this STATUS PDU affect DL way too much, as 10 UL packets per second makes the DL rate from 73000B/s to 30000B/s?

3) Does this also mean that TCP rate would be much lower than UDP (I actually observed this)?

A high level question related to TCP, do you think current implementation cannot simulate TCP, I mean, normally STATUS PDU are piggybacked which would not take DL bandwidth, however currently ACKs would also take DL rate? Because of this unrealistic simulation of TCP DL, If I want to get more realistic TCP DL rate, what should I do? Should I use UDP instead? Do you have other suggestions?

Best,

Xing

[Lorenza Giupponi]

Hi, 

1) in RLC acknowledged mode each PDU has to be acknowledged by means of STATUS PDUs, otherwise is retransmitted.

2) on the specific numbers you should check in traces if they make sense, in principle these functionalities have been widely tested

3) yes, It is to expect that TCP throughput is lower than UDP.

4) STATUS PDUs are not piggybacked, but TCP ACKs are. When you configure TCP and RLC-AM there are two Windows that envolve through ACKs, at both TCP and RLC layers. The fact that RLC ACKs are not piggybacked is a realistic hypothesis, you can check in standards.

Lorenza

Enviado desde mi iPad

[Xing Xu]

Thanks.

For 2), how do I tell if one RLC layer packet is data or STATUS? I attached my D/UlRlcStats for both cases of with UL traffic and without UL traffic, could you please teach me how to understand those? (In general, if there is a document about how to understand these traces, please let me know, I was always using FlowMonitor and analyzed by myself...)

For 4), because you said "LENA implementation", I thought you meant LENA implementation is different to the real LTE system. But let me ask that explicitly, in real LTE system, are RLC ACKs piggybacked or not? For efficiency purpose, it should be piggy-backed, right? Otherwise DL needs to waste bandwidth on UL ACKs (amount of DL wasted bandwidth is equal to the bandwidth required for UL traffic, as each UL packet needs an ACK on DL).

  - Separate not, FYI, I know TCP ACKs are piggy-backed, but in my experiment there is no UL traffic so the TCP ACKs have to be sent on UL and there is no data to piggyback those TCP ACKs.

[Xing Xu]

Clarify my previous post.

For 2), I don't really know how to check traces to see if those DL throughput drop is due to UL ACK, please advise me on that :)

Thanks for your help.

[lorenza....@gmail.com]

Hi Xing,

Those traces that you shared are aggregated, so you cannot get the information you need from there. Please, activate the RLC logs, there you will find all the possible information.

Good luck!

Lorenza

Inviato da iPhone

[Xing Xu]

Thanks.

But can you tell me what is RLC logs and how do I activate it? (Do you mean NS LOG module? I don't know if that can specify network layers?)

Another question I have, is it possible to implement piggy-backed ACKs for link layer (in AM mode)? (Is it easy for me to implement... I guess it is super hard...)

I asked around and seems that real LTE system uses AM mode and the link layer ACKs are piggy-backed. Right now it is hard for me to simulate real TCP traffic, see below reasons, please let me if my below reasons make sense to you, and also let me know if you have better idea of simulating TCP traffic:

 - If I simulate TCP in AM mode, then link layer ACKs consume DL bandwidth, which makes TCP much much slower than UDP, which is not realistic;

 - If I simulate TCP in UM mode, then there are so many link layer packet losses, which leads to network layer packet losses, which affects TCP throughput significantly. 

[Xing Xu]

Actually, I don't think I need to do more experiments to verify this... From RlcStats I attached, you can see that for each UE, 8 PDU got received every 0.25 seconds, but the number of transferred PDU varies from 2 to 5. I think those transmitted PDUs are data, those "not counted as transferred" PDUs are ACKs, and those ACKs' PDU size is only 2 bytes. Does this make sense to you?

And I guess ACKs have higher priority than data, which indeed makes that, we have to waste the same amount of bandwidth of UL on DL. Please let me know what do you think.

You received this message because you are subscribed to a topic in the Google Groups "ns-3-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ns-3-users/Tkkk9q8G6Uo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ns-3-users+...@googlegroups.com.

[Lorenza Giupponi]

Hi,

yes, from the traces you shared, i see in DL only data PDUs when UL traffic is not involved, and also STATUS PDUs when UL traffic is involved, which makes sense.

Lorenza

[Lorenza Giupponi]

Hi,

When you have LTE+TCP, independenly of the link layer, please also consider that the reduced throughput is probably due to the latency introduced by the LTE protocol stack. These delays are increased when you use TCP in comparison to UDP. The reason is that when the TCP ACK is available at the MAC UE, a buffer status report is sent to the eNB, and the eNB has then to send back a DCI with information on the transmission opportunity. Only when this is received and processed the TCP ACK can be scheduled and sent. I estimate approx 15 msec between when the packet is sent and the ACK is received at the remote host. In UDP this latency is lower. So it makes sense to me that TCP is slower than UDP. The point is how much. I guess TCP could also be optimized.

Hope this helps,

Lorenza

[Xing Xu]

Thanks Lorenza, really appreciate your help.

I don't quite follow your DCI thing, but I think the main reason for throughput drop in AM mode is because of "link-layer ACKs are not piggy-backed": each UL PDU will generate an ACK PDU at DL, of even higher priority compared to DL data, and waste the DL bandwidth (transmission opportunity?) From the trace, you saw that these ACK PDUs only have 2 bytes compared to data PDUs have ~2000 bytes, I assume these PDUs consume the same amount of bandwidth resources right (or the correct term, PRB)? Please let me know what do you think about this.

Another question I asked, which you didn't answer, is is it easy to implement piggy-backed ACKs for AM mode? I assume it is super-hard, as I don't know anything about LTE or NS3 simulation (I hacked a lot, but I still view myself as a NS3 user...). Please let me know.

Thanks again for your help! We are doing a research paper, so that's why I was being pushy on this topic. Thanks.

You received this message because you are subscribed to a topic in the Google Groups "ns-3-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ns-3-users/Tkkk9q8G6Uo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ns-3-users+...@googlegroups.com.

[lorenza....@gmail.com]

Hi Xing,

Even in case of DL only traffic, you could see that TCP throughput is quite lower than UDP throughput. The reason is the latency I explained you before. 

I agree that the no piggybacked status is a waste of resources. It is correct what you say, you are occupying 2 bytes for 1 msec transmission. 

About the estimation of effort I did not reply on purpose :-) It is always difficult making development estimations, it should be studied, and I do not know your skills. Of course you need to be knowledgable of LTE. If you are interested in attacking this issue, we can talk off list.

Kind regards,

Lorenza

Inviato da iPhone

[Xing Xu]

You estimate that by, maybe, I need to modify across 3 files, and then I will give up... I have an approaching deadline :)

[Xing Xu]

Hi Lorenza,

I guess it would be hard for me to implement AM mode in piggy-backed way, however, I think it's maybe much easier to calculate / adjust the throughput of piggy-backed way based on current non-piggy-backed way.

My idea is to get the size of TxOpportunity, and if such TxOpportunity is used for STATUS PDU, I output the bytes that I can leverage for DATA in this TxOpportunity. Does this make sense to you?

E.g., if one TxOpportunity size is 2196, however, it is used for STATUS PDU, which is 4 bytes, I'll output 2192 bytes as the DATA for this RNTI, which will be used later to adjust this RNTI's actual TCP throughput.

I found one thing, lots of TxOpportunity have size of 4 bytes, are these for DATA? I count this at function: DoNotifyTxOpportunity

Thanks,

Xing

[Xing Xu]

Hi, Lorenza,

It's Xing again, thanks for your time in advance.

I'm not sure about what you described here. MAC layer doesn't differentiate TCP or UDP right? The only difference between TCP and UDP is that TCP has UL traffic (TCP ACKs), right?

I have another question, if I use independent DL UDP traffic and UL UDP traffic to simulate TCP, I should see similar throughput results for DL, right? (I mean, using UL UDP to simulate TCP ACKs)

However what I found is, using TCP, I will see more link layer retx, but when I'm using bi-directional UDP, there is not much link layer retx (experiments run in the same setting, with inter-cell interference, where interfering UEs have both UL and DL traffic), which makes me think about why TCP performance is different to UDP.

BTW, although I cannot implement piggy-back AM mode, I figured out how to get the size of the Tx Opportunity that is used for STATUS PDU, and I log the bytes that have not been leveraged. In the end I use the log to adjust the TCP throughput, and I see its throughput is similar to UDP for some cases, when there is no inter-cell interference. But when there are inter-cell interference, that is the problem I described above, I see more link layer retx with TCP but not bi-directional UDP.

Let me know if you have any clues.

Thanks,

Xing

[lorenza....@gmail.com]

Hi Xing,

 TCP and UDP are two very different transport protocols, they change the arrival pattern of data, which will impact your performance. TCP pattern is controlled by a congestion window, while UDP is not. TCP data transfers require many RTT, which UDP does not require. So differences go much beyond the bidirectional traffic and I hardly see you can emulate the difference by simulating bidirectional traffic in UDP.

Another effect is the one on end to end delays I described in previous email. 

As for deducing the LTE throughput when you transmit more RLC PDUs in the same txop wrt the case when you transmit only one, as it happens in the simulator, also here I do not think you can make numbers on this, many other aspects may affect, it is more complex. Either you live with the available implementation, or you change it. Personally I would not trust data generated in any other way,

Kind regards,

Lorenza

Inviato da iPhone

[Xing Xu]

Thanks Lorenza.

Sorry, I should give you some recap :) You misunderstood my comparison.

I was NOT comparing TCP to UDP in general (I know both of them in
depth), I was comparing them when both TCP sender and UDP sender have
sent enough data that can congest the enb. If that is the case, then
TCP packets and UDP packets are just "data" at link layer, there is no
difference between them, and if there are enough data for both cases,
their DL throughput should be similar, both equals to enb's link layer
capacity/bandwidth.

How do I make sure both sender send enough data to congest enb? For
UDP, I configured its sending rate to be much higher than enb's link
layer capacity; for TCP, I didn't do anything, but because I'm using
AM mode, there is no network layer packet loss, so TCP's cwnd will
keep increasing, makes the packets buffered at enb (bufferbloat
phenomenon of cellular network), I also confirmed this from trace.

Under this congested circumstance, I was expecting that their DL
throughput should be similar. I said that "TCP ACK on UL should not
affect DL capacity", because anyway there are already enough TCP data
at enb, so UL ACKs, latency, should not matter in my case.

Previously, because TCP has UL traffic (TCP ACKs), which generates
STATUS PDU at DL, so I have to adjust the TCP throughput by assuming
that STATUS PDU (link layer ACKs) are piggybacked, the reason why I'm
logging the extra bytes TCP can leverage: size of Tx Opportunity minus
size of STATUS PDU (BTW, I'm not assuming multiple PDUs, I'm just
assuming each Tx Opportunity can be leveraged efficiently, by
piggybacking actual data on STATUS PDU packet). After the adjustment,
I found that when the inter-cell interference on UL is not severe, my
TCP throughput is similar to UDP.

The new problem I found is that when the inter-cell interference on UL
is severe (DL interference doesn't matter), TCP throughput is still
lower than UDP. I don't know the reason, why UL packet from
neighboring cells can affect DL performance? I then figured out that
seems the reason is that using TCP I saw more link layer ReTx, but for
UDP the link layer ReTx is rare. Do you know the reason?

I guess only I know my experiment in detail, so I think it is really
hard for you to figure out the root cause of the problem. But I wonder
are there any other differences between TCP and UDP that can
potentially make the throughput different?

Three quick, concrete questions:
1) AM mode will treat UDP packet and TCP packet identically, right?
That means both types of packet should be retransmitted in the same
way. (I think so, MAC layer should not differentiate upper layer
protocol);
2) If I'm using ALWAY_AM, then both DL and UL are using AM right? so
UL traffic should also be retransmited? can I configure UL to use UM
but not AM?
3) are there two lte-rlc-am instances for both UL and DL? I'm printing
out ReTx information from lte-rlc-am.cc, I thought all the print-outs
are for DL, but are some of them for UL? How do I differentiate?

Thanks for you time again, my description is long... :)

> You received this message because you are subscribed to a topic in the
> Google Groups "ns-3-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ns-3-users/Tkkk9q8G6Uo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Lorenza Giupponi]

Hi Xing,

yes it is bit hard to follow you. I try to reply to the three questions:

1) AM mode will treat UDP packet and TCP packet identically, right?
That means both types of packet should be retransmitted in the same
way. (I think so, MAC layer should not differentiate upper layer
protocol);

For MAC handles MAC PDUs and treat them the same, independently from the upper layers. Still the behaviour of the upper layers may affect the final results, due to many reasons, as explained in previous posts.

2) If I'm using ALWAY_AM, then both DL and UL are using AM right? so
UL traffic should also be retransmited? can I configure UL to use UM
but not AM?

I am not sure what you mean by ALWAY_AM, I interpret that you are using RLC_AM for all the logical channels. You are using AM on both links. You cannot configure UM on UL and AM on DL.
If you want to have a different behavior in UL wrt DL you should create two different LCs.

3) are there two lte-rlc-am instances for both UL and DL? I'm printing
out ReTx information from lte-rlc-am.cc, I thought all the print-outs
are for DL, but are some of them for UL? How do I differentiate?

Yes, there are 2 instances of the LteRlcAm, as you can check in the LteHelper and in the LteUeRrc. For differentiating them, there are different ways, for instance their pointer in the NS_LOG traces (it depends on where you want differentiate them) or with the node id.

About the other post, which you are also sharing privately with the maintainers team, I am afraid this is a matter of interpretation of your results, in your specific scenario, which we are not familiar with, so we hardly can give hints. Please, try to use the information above to get to the bottom of those results, and if you see anything weird, then let us know. Please, in this case, try to provide scripts of the behaviour you observe, an identification of the code you think is generating an issue. Also, please try not to write privately to the maintainers, they read the list, and if they spend time to reply, it is good that the reply goes to the whole list.

Kind regards,

Lorenza

[Xing Xu]

Thanks again. I am preparing a submission, sorry for being "pushy" on sharing my post...

It would be hard to explain what I'm doing now, otherwise I'll prepare script and file the bug. For example, in my setting, I think TCP and UDP would perform similarly at DL, because they both have enough packet buffered at enb (enb is congested by those traffic), but it would be hard to let other understand, especially in a short post, people may ask why you are comparing TCP to UDP, they are completely different thing... And it would be impossible to ask maintainer to clone my code (I hacked a lot) to generate the same result and analyze on top of it.

Actually, I'm not sure the issue I observed is due to a bug or not (like previously, I thought UL traffic slows down DL traffic, which is probably a bug, however it turns out that it is the correct behavior based on AM implementation), I see TCP result has more variation, due to a lot more link layer ReTx, but I don't know if it is due to a bug or not, that is actually what I'm trying to figure out. I guess the best way for me is just to have some conjectures and ask you guys concrete questions for more insights. (BTW, I don't know LTE, but I work in general networking area, e.g., I know TCP pretty well.)

I guess right now I should first differentiate UL ReTx and DL ReTx, and see if that is the reason of fluctuated and degraded rate of TCP. I have a couple of concrete questions below:

 1) what is the reason of link layer packet loss? low MCS value, or unfortunate packet collision (which means "luck" plays a role)? I asked this because I thought LTE will use different coding rate for different MCS value to control the packet loss rate. However, I observe many more packet losses (many more ReTx) using TCP vs. using UDP, but the entire interference configuration is the same, I think it is strange. (

 2) How do I get the root cause of each link layer packet loss? Should I print out something (or using log) from some particular files? Please let me know.

Thanks for your time. 

[Xing Xu]

I just confirmed that all the ReTx are happening on the DL (logged by enb's lte-lrc-am instance).

[Xing Xu]

I got the reason for TCP's difference behavior, but I'm still trying to understand it.

I found that when I am using TCP, this function is called and lots of PDUs have been put into the retx buffer:

ExpirePollRetransmitTimer

I guess the reason that UDP won't happen this is, when I'm using UDP, txonBuffer is never empty; when I'm using TCP, although AM mode should hide packet loss and thus sender should send enough data, sometimes I still see that txonBuffer is empty, which leads to lots of PDUs put into retx buffer.

Why we put transmitted PDU to retx buffer, when we don't have data to send? Thanks.