link metric for VoIP over Mesh
Steve Song
I was installing a new VT node in the Bo-kaap community on the weekend
and ran into some anomalies in using the batman link metric as a link
quality indicator. More than once I found that even when batmand -cd1
was showing a strong (230-254) link connection, I was unable to ssh
without huge lag into a device only a hop or two away. Has anyone
else found this? Is it perhaps a case of not waiting long enough for
the metric to settle?
I know we've discussed this a bit before but perhaps we could re-cap
what are the best VoIP over Mesh performance metrics. I know David
Rowe has been using ping -s 1400 which forces large packets over the
network. Anyone else have some tried and tested approaches?
Thanks... Steve
elektra
> I was installing a new VT node in the Bo-kaap community on the weekend
> and ran into some anomalies in using the batman link metric as a link
> quality indicator. More than once I found that even when batmand -cd1
> was showing a strong (230-254) link connection, I was unable to ssh
> without huge lag into a device only a hop or two away. Has anyone
> else found this? Is it perhaps a case of not waiting long enough for
> the metric to settle?
>
Did you have this problem when were you using the notebook directly with
the build-in wireless card?
I have seen wireless driver problems like this, resulting in drivers
(namely ath9k and rt73) not being able to perform unicast traffic. The
problem might be caused by bugs in the rate control algorithm, for
example. If the rate control fails, the results of the routing protocol
metrics are OK, as the beacons are measured by sending/receiving
broadcast packets at basic rate. Malfunctions of the rate control in the
driver are thus not affecting the batman metric, and in principle the
link is OK (if the wireless driver weren't broken). However you will not
be able to use the network. The driver may get stuck using a high
bitrate or for some other reason not send/receive any unicast traffic at
all.
I remember having the same problem with ath9k and Ubuntu 10.04, while
setting up Freifunk for a guy who came to the C-Base. We had perfect
link metrics (both devices on the same table!) but we couldn't get more
then a few bytes through at best. Rebooting solved the problem
temporarily. I installed everything to compile a new version of
compat-wireless and updated the driver. At least for the evening all was
working fine.
Certain versions of ath9k seem to have this problem. So far it doesn't
use 802.11n bitrates for ad-hoc, but the developers are working on it.
The driver is a WiP, so not entirely stable. What you can do is reboot
when the problem occurs for the next time - and try to compile and test
different versions of the driver. (Every few weeks I test a new version
of the compat-wireless drivers - usually things get better. My Intel
iw4965 now successfully works flawlessly in the mesh when I do
suspend-resume cycles with my laptop. I can run it for days without
problems - hip, hip, huurrray!
Cheers,
Elektra
David Rowe
I have been wondering about the batman metric for a while. On my mesh
links here in Kilkenny and in Dili I have often experienced relatively
high (e.g. 170-200) batman scores that had poor packet loss, won't
support ssh, and have poor voice quality.
I have also seen situations where large improvements in signal strength
and ping statistics have no impact on the batman score. I guess that's
because the packets batman uses to probe links are small, which doesn't
stress the links much compared to many IP packets (e.g. web traffic)
that are closer to the MTU.
I have been graphing batman scores over time with Munin. The batman
metric does accurately show other wifi traffic messing up links, for
example it jumps around after 3pm here in Kilkenny when Wifi traffic in
my neighbourhood picks up.
The metric is an accurate binary estimate of link quality. I have found
that over 240 means a good link for all data, when it sinks to 0 then
you can be sure the link is non-existent. But in between I haven't
found it useful for telling me if I can make a clean voice call, or if
IP traffic like ssh will work well. So I tend to use ping and ping -s
1400 to measure link quality and adjust nodes.
Since the batman score is used to select the route, this does make me
wonder if it is selecting optimum routes. But if you have a bunch of
solid links I guess it doesn't matter.
This also raises the question of what is a good link? Through
experiment I am defining this as ping -s 1400 with < 10% packet loss.
This means I can do VOIP and ssh no problem. Arguably we can get VOIP
through poorer links than large MTU IP traffic as it uses small packets,
however it might be better to mandate very solid links for VT networks.
Elektra perhaps you can tell us more about the reasoning behind the
batman metric.
Cheers,
David
David Rowe
will be very sensitive to multipath. That's because the metal box
effectively shields the PCB diversity antenna on the motherboard. So
you only have one antenna instead of two. Normally the Wifi driver
selects the antenna with the strongest signal.
In practice this means extreme sensitivity to positioning when used
indoors. I have actually experienced this. I was using an MP01 beta as
an extension to my home IP-PBX network. Every now and again it would
stop working, just drop off the network. I would go and move it 1 inch
and it would come back up. A few weeks later the same thing would
happen.
Even if other objects (like people) move the link will be affected.
Multipath can have a huge effect on signal strength. For example when I
connect a Wifi antenna to my spectrum analyser in my office I can see
the Wifi signal in real time. If I move my arm in the same room the
Wifi signal jumps up and down 20dB.
For a long term indoor deployment with the MP01 Betas its probably best
to operate them out of the box, or at least with the lid off.
- David
Donald Gordon
On Tue, Aug 24, 2010 at 1:51 AM, elektra <onel...@gmx.net> wrote:
> I have seen wireless driver problems like this, resulting in drivers (namely
> ath9k and rt73) not being able to perform unicast traffic. The problem might
> be caused by bugs in the rate control algorithm, for example. If the rate
> control fails, the results of the routing protocol metrics are OK, as the
> beacons are measured by sending/receiving broadcast packets at basic rate.
> Malfunctions of the rate control in the driver are thus not affecting the
> batman metric, and in principle the link is OK (if the wireless driver
> weren't broken). However you will not be able to use the network. The driver
> may get stuck using a high bitrate or for some other reason not send/receive
> any unicast traffic at all.
I was wondering: is there some way batman could be told to use unicast
to work out the link quality, maybe by piggybacking on data packets
that go past to see what the packet loss is really like?
donald
Donald Gordon
On Tue, Aug 24, 2010 at 12:03 PM, David Rowe <da...@rowetel.com> wrote:
> I just remembered something. The MP01 V1.2 Betas in the nice metal case
> will be very sensitive to multipath. That's because the metal box
> effectively shields the PCB diversity antenna on the motherboard. So
> you only have one antenna instead of two. Normally the Wifi driver
> selects the antenna with the strongest signal.
>
> In practice this means extreme sensitivity to positioning when used
> indoors. I have actually experienced this. I was using an MP01 beta as
> an extension to my home IP-PBX network. Every now and again it would
> stop working, just drop off the network. I would go and move it 1 inch
> and it would come back up. A few weeks later the same thing would
> happen.
Ah, that would explain issues I've seen with an MP01 providing phone
service to some friends. I assume it'd be an issue even if the MP01
was connecting to a wifi device that did have unshielded diversity
antennae (i.e. not another metal-boxed MP01)?
donald
David Carman
On Tue, Aug 24, 2010 at 1:55 AM, David Rowe <da...@rowetel.com> wrote:
> I have been wondering about the batman metric for a while. On my mesh
> links here in Kilkenny and in Dili I have often experienced relatively
> high (e.g. 170-200) batman scores that had poor packet loss, won't
> support ssh, and have poor voice quality.
If the batman metric is like OLSR's, it is a running average of the
last 100 OGMs, so it describes link quality over the last 2 minutes.
Ping is a realtime snapshot. In fact, if batman is showing scores >
200, its running average is probably longer than 2 minutes.
> I have been graphing batman scores over time with Munin. The batman
> metric does accurately show other wifi traffic messing up links, for
> example it jumps around after 3pm here in Kilkenny when Wifi traffic in
> my neighbourhood picks up.
Not microwave ovens defrosting supper?
David
elektra
the problem that David has described is probably that the link is asymmetric
and there is no stable alternative link from the other side.
Asymmetric links are quite common in wireless networks, particularly in
densely populated areas. A node on a high side can be seen very well from far
away. At the same time it receives signals and interference from all sides -
it is a exposed node. Asymmetric links with 90% success rate in one direction
and 10% success rate are not uncommon. The same happens if you have another
wireless network on the same channel. It is often a AP indoors or other home
application close to a node, so it will disturb the reception locally when
active.
The Batman metric calculates the success rate of delivered Batman protocol
messages, based on the last 64 expected broadcast packets. By default these
messages are sent once per second, so the metric is an average of the last 64
seconds. Now, if there is a neighbor surfing the web with an accesspoint
operating on the same channel one might experience packetloss every now and
then if someone in this hotspot opens a web page, for example. If you monitor
the batman metric in such a case, it is good most of the time. However there
is no guarantee that the link quality doesn't change anytime because the
interference might come at a random pattern and in short bursts.
Batman supports asymmetric routing, as wireless links are often asymmetric. So
Batman will route from A to X via A-B-X, while it may route from X to A via X-
C-D-A, for example. It will do the best it can to get data delivered, but it
can not route around asymmetric links if there is no alternative route
available.
The metric that Batman uses is called TQ - transmit quality. Since Batman can
and will route asymmetric - if it is the best way to route - the metric is
measuring the success rate of broadcast transmissions towards its next hop
only - it is *not* describing or measuring the success rate of a round trip
transmission.
Routes in a routing table are showing the next hops to a destination, not
from. Now in the wired world this might mean that in 99.999% of the time the
return path is the same. But it doesn't need to be this way. And in the
wireless world this assumption is clearly invalid. There are mesh protocols
that always try to setup symmetric paths, but this is clearly suboptimal.
So if A<->X is a direct asymmetric link with good quality from A->X, Batman
will report a decent metric in A. If the only route to get from X to A is a
miserable direct link X->A, Batman in X will report a low metric, while Batman
in A will still report a decent metric. Looking at the Batman metric only on
one side of a link can lead to false conclusions. By testing with ICMP we can
check how well the bidirectional communication between A and X actually works.
That doesn't mean that the Batman metric reported in A is bogus, or "binary",
as David has concluded.
A local Batman metric is no estimate or guarantee that the receiver side has a
decent transmit routing path towards the node where we look at the Batman
output.
This doesn't mean that the Batman metric will *only* consider the success rate
in one direction. A link with 100% delivery rate of Batman protocol broadcasts
(Originator messages or OGMs for short) in one direction and 0% success rate
in the other direction doesn't get rated as perfect link. In fact it is
unidirectional and therefore disregarded, as the acknowledgements from the
other side do not get through. However if the metric is perfect or almost
perfect (>250) the link is bidirectional and symmetric, as asymmetric links
always get a penalty. Which convinces me that Steves issues are caused by
driver woes.
A link with 100% delivery rate in transmit direction, but only 40% success
rate from the other direction will be seen as a decent TQ link from the
transmitting nodes view. ACKs are very small, so the likelyhood that the
payload packet (large) gets successfully ACKed is good.
But it is important to remember that the local Batman metric is concerned
about getting the data to a destination, not from. If you want to check the
round-trip quality of a certain link you can check the output of the vis
server or check on the Afrimesh map, because the link quality shown there is
calculated from both TQ metrics describing the round trip properties of a
link.
Remember that payload traffic is unicast and gets retransmitted. If the
retransmission fails the data rate will be adapted and yet again
retransmitted. The success rate is therefore much higher than with broadcasts.
David, if you think that the broadcast measurements of Batman are not
accurately describing the link quality you can increase the broadcast data
rate in the Madwifi driver. This way the Batman messages have shorter range and
higher losses. Just uncomment and experiment with:
#option mcast_rate 5500
in /etc/config/wireless of the MP firmware. This will set the broadcast rate to
5.5Mbit. The WiFi default for 802.11b is 1Mbit.
Another nice aspect of increasing the broadcast rate is less airtime being
spent on beacons and probe responses. But Batman might consider some nodes out
of range and the routes might become unstable as it will begin to disregard
them.
Regarding unicasts as link quality measurement strategy:
Unicasts are costly, because of the retransmissions. What is worse: A protocol
would need to perform frequent unicast transmissions to all neighbors. I don't
think this is a feasible approach. We have already seen routing protocols
severely reducing the network capacity due to protocol overhead. I suppose
with unicast link measurements this will get a lot worse.
Another idea would be to send broadcasts with different broadcast data rates.
However there are only few drivers that support this, Broadcom proprietary and
Madwifi are the only ones I know. And it would need a lot more broadcasts to
estimate the bitrates of links than we use now, in order to get results in a
timely manner.
Cheers,
Elektra
Sjur Eivind Usken
these up on the wiki?
cut'n'paste?
would be great to move this knowledge to a permanent place.
sjur
*e-mail is where knowledge goes to die...*
> --
> You received this message because you are subscribed to the Google Groups "village-telco-dev" group.
> To post to this group, send email to village-...@googlegroups.com.
> To unsubscribe from this group, send email to village-telco-...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/village-telco-dev?hl=en.
>
>
David Rowe
> only - it is *not* describing or measuring the success rate of a round trip
> transmission.
OK, thanks for explaining that. Make sense of some of my results. I
can see why people have so much fun designing mesh routing
protocols......
However - we have an urgent practical requirement for a metric that
tells us if we can make a phone call and that ssh will work. We need
this to make MP installations work in the developing world. Batman
metrics are perhaps not an appropriate choice.
Goal: any non-technical person should be able to use this metric to
install a solid link. It needs to run stand alone on a MP. The UI
could be the web interface, command line or telephone.
> But it is important to remember that the local Batman metric is concerned
> about getting the data to a destination, not from. If you want to check the
> round-trip quality of a certain link you can check the output of the vis
> server or check on the Afrimesh map, because the link quality shown there is
> calculated from both TQ metrics describing the round trip properties of a
> link.
Sounds like just what I need! But I don't have a convenient tool to do
that and in a simple MP-MP set up there is no Vis server.
We really need to add this tool to the "to-do" list for the VT camp. We
need it sooner rather than later. To avoid this dragging on forever I
suggest we mandate that the tool must be stable, rigorously tested and
released by October 30. Not alpha or experimental.
> David, if you think that the broadcast measurements of Batman are not
> accurately describing the link quality you can increase the broadcast data
> rate in the Madwifi driver. This way the Batman messages have shorter range and
> higher losses. Just uncomment and experiment with:
Can I adjust the size of the broadcast packets? Large packets may give a
much better indication of link status. The bit and transmission rate
can stay the same. I don't care if it clogs the network a little -
better to have a reliable indication of link quality.
Thanks,
David
David Rowe
I'll be attending ExtremeCom in a few weeks:
Is anyone else from this list going? I will also be in Delhi for a few
days in September if any one wants to say hi.
Cheers,
David
Paul Gardner-Stephen
On Aug 25, 7:23 am, David Rowe <da...@rowetel.com> wrote:
>
> > David, if you think that the broadcast measurements of Batman are not
> > accurately describing the link quality you can increase the broadcast data
> > rate in the Madwifi driver. This way the Batman messages have shorter range and
> > higher losses. Just uncomment and experiment with:
>
> Can I adjust the size of the broadcast packets? Large packets may give a
> much better indication of link status. The bit and transmission rate
> can stay the same. I don't care if it clogs the network a little -
> better to have a reliable indication of link quality.
we end up developing the virtual-circuit software we talked about
where we cram the voice packets for one or more calls into more
regular Batman packets. I have already come to the thought that the
packets should probably just be a fixed large size to accomodate this
(also helps in security, but that's another issue). In that setting
the Batman score would be a direct measure of the call quality. I
like direct measures.
Paul.
> Thanks,
>
> David
elektra
we are actually talking about two kinds of metric here.
1/ SNR - signal to noise ratio
2/ Total throughput, packetloss and latency of a complete routing path between
two destinations.
There are nice tools for 2/ - my all-time favorite is ping from the iputils
package. Here is what I do if I have network problems:
I ping the gateway or destination from my device. If all is fine the problem is
not in the mesh.
If the problem is in the mesh I use ping -R and check the routing path to the
destination I can not reach OK. I ping them one after the other to see which
limb in the chain is broken. Then I go there and help the people to fix their
SNR or turn my antenna to another neighbor which has a good link or route to
the destination.
The metric that any user understands is his experience when he makes a call.
If this doesn't work he knows that something is wrong. Actually he can call
his next-hop neighbor with the MP and see if the call quality is OK.
But ping can only show you whether the route is OK or not. You really start
to think about 2/ if you actually need to look at 1/
your concern at this point is the wireless layer. Batman will try to do the
best to route around broken links. However it is no substitute for links with
good signal to noise ratio.
Layer 1 is a complex beast in our case - the best metric here clearly is
signal to noise ratio. Unfortunately iwconfig will tell you nothing about the
actual noise floor - the value reported merely shows the receiver sensitivity.
If the link isn't working properly this is where you look at. Needless to tell
a HAM that the best way is to go out of the way of noise and boost the signal.
So what are the options?
Increase the transmit power
Increase the antenna gain
Use a directional antenna to increase antenna gain and fitler out unwanted
signals
Increase the sensitivity of the receiver.
Interference from other networks, applications or nodes
and one of these layers is
a excellent tool for testing whether the network actually works between two
MPs is ping. No need to reinvent the wheel.
elektra
don't get confused by the garbled end of the previous mail, I wanted to select
the "save as draft button", not "send" ...
Cheers,
Elektra
David Rowe
Update on the Dili Village Telco:
http://www.rowetel.com/blog/?p=845
We are learning lots from deploying a real world VT network. In
particular a lot of social and non-technical facets of the Village Telco
concept are being teased out. The enthusiasm and progress of the guys
in Dili is amazing. We are really helping people with this technology.
Thanks,
David