[iproute2] tc action mirred question
Xiaofei Wu
I have something to ask.
I construct a network like this:
/A\
B D
\C/
All of the nodes(A, B, C, D) have two wireless cards (wlan0, wlan1). A-B, B-C, A-D, D-C are wireless links.
Node A wnats to transmit packets with node C. Because the wireless links are not very reliable, I want to forward the same packet through A-B-C and A-D-C simultaneously.
How to achieve my purpose？
Stephen Hemminger said,
>Not sure what the best solution would be, but you could investigate
>using the 'tc filter mirred' action. Essentially, the traffic control
>command allows putting filters on output (or input) that can be used
>to do things like mirror packets.
>
On node A,
wlan0, IP address 192.168.1.1/24 ; wlan1, IP address 192.168.2.1/24
I use command 'tc filter add dev wlan0 ... match ip src 192.168.1.0/24 ...
action mirred egress mirror dev wlan1' to mirror packets.
When I use 'tcpdump -i wlan1', I can 'see' the packets 'A(wlan0)->B' (node B will forward them to C). How to forward the mirroring packets 'A(wlan1)' to D (then, node D forwards them to C) ?
Regards,
Wu
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
jamal
> On node A,
> wlan0, IP address 192.168.1.1/24 ; wlan1, IP address 192.168.2.1/24
> I use command 'tc filter add dev wlan0 ... match ip src 192.168.1.0/24 ...
> action mirred egress mirror dev wlan1' to mirror packets.
> When I use 'tcpdump -i wlan1', I can 'see' the packets 'A(wlan0)->B' (node B will forward them to C).
> How to forward the mirroring packets 'A(wlan1)' to D (then, node D forwards them to C) ?
>
>
Is there a trick to this question or is it too basic? ;->
You should repeat the same on wlan1 to mirror to wlan0 i.e on wlan1:
match ip src 192.168.2.0/24 ...
action mirred egress mirror dev wlan0
Note the node C will receive "wrong" src mac addresses on those
interfaces; you may want to correct/edit them first before you send them
out. Look at using the pedit action.
cheers,
jamal
Xiaofei Wu
>> On node A,
>> wlan0, IP address 192.168.1.1/24 ; wlan1, IP address 192.168.2.1/24
>> I use command 'tc filter add dev wlan0 ... match ip src 192.168.1.0/24 ...
>> action mirred egress mirror dev wlan1' to mirror packets.
>> When I use 'tcpdump -i wlan1', I can 'see' the packets 'A(wlan0)->B' (node B will forward them to C).
>> How to forward the mirroring packets 'A(wlan1)' to D (then, node D forwards them to C) ?
>>
>>
>Is there a trick to this question or is it too basic? ;->
I am sorry.
Maybe it is very easy for you. But I didn't find enough documents(or examples) about 'tc' to help me.
I just want to know:
1) Could I forward the mirroring packets to another node ,and then route it to the destination(if I use
iproute2 (ip, tc ...) )? I described my purpose in my last email.
2) After I mirrored the packets, I should use 'ip route' , 'ip rule' to modify route tables. Is this right?
>You should repeat the same on wlan1 to mirror to wlan0 i.e on wlan1:
>match ip src 192.168.2.0/24 ...
>action mirred egress mirror dev wlan0
Would this cause loops?
>Note the node C will receive "wrong" src mac addresses on those
>interfaces; you may want to correct/edit them first before you send them
>out. Look at using the pedit action.
Regards,
Wu
jamal
> I just want to know:
> 1) Could I forward the mirroring packets to another node ,and then route it to the destination(if I use
> iproute2 (ip, tc ...) )? I described my purpose in my last email.
Yes, you can mirror to another node(B/D). To route on that node(B/D),
your dst MAC address has to be correct for that destination node(B/D) to
accept it. You could try to run the destination node in promisc mode
and you may be able to get away without changing dst mac.
> 2) After I mirrored the packets, I should use 'ip route' , 'ip rule' to modify route tables. Is this right?
>
Assuming you are talking about B/D, yes you can do routing there if the
node accepts it..
> >You should repeat the same on wlan1 to mirror to wlan0 i.e on wlan1:
> >match ip src 192.168.2.0/24 ...
> >action mirred egress mirror dev wlan0
>
> Would this cause loops?
>
Ok, so this was the trick question;->
I dont see how the loop would happen - they are different "match" rules
i.e one is for 192.168.2.0/24 and the other is for 192.168.1.0/24
Whether it loops or not is easy for you to verify.
cheers,
jamal
Xiaofei Wu
I am a newbie for 'traffic control' and Linux networking. So I ask some experts here to help me.
Maybe my questions are stupid. But I hope I can get your reply. Thank you!
>> I just want to know:
>> 1) Could I forward the mirroring packets to another node ,and then route it to the destination(if I use
>> iproute2 (ip, tc ...) )? I described my purpose in my last email.
>Yes, you can mirror to another node(B/D). To route on that node(B/D),
>your dst MAC address has to be correct for that destination node(B/D) to
>accept it. You could try to run the destination node in promisc mode
>and you may be able to get away without changing dst mac.
(1) Could I use pedit action to modify the dst MAC, so the destination node D will accept it, then forward it to
node C? (or use other tools to modify the dst MAC, please give me more information)
(2) If I use 'ifconfig wlan0 promisc ... ' on node D, would it route the mirroring packets (the dst MAC is incorrect)
to node C?
>> 2) After I mirrored the packets, I should use 'ip route' , 'ip rule' to modify route tables. Is this right?
>>
>Assuming you are talking about B/D, yes you can do routing there if the
>node accepts it..
Regards,
Wu
jamal
> (1) Could I use pedit action to modify the dst MAC, so the destination node D will accept it,
> then forward it to node C?
Yes, you can achieve it with pedit;
> (or use other tools to modify the dst MAC, please give me more information)
>
it is as usable as u32 is - you have to know your offsets
example, here's something done on an incoming packet:
=-=
#Note:
#dst MAC starts at -14
#src MAC at -8
#ethertype at -2
#
tc filter add dev eth1 parent ffff: protocol ip prio 10 u32 \
match ip src 192.168.2.11/32 flowid 1:2 \
action pedit munge offset -14 u16 set 0x0000 \
munge offset -12 u32 set 0x00000200 \
munge offset -8 u32 set 0x0aaf0100 \
munge offset -4 u32 set 0x0008eb06 pipe \
action mirred egress redirect dev eth0
----
> (2) If I use 'ifconfig wlan0 promisc ... ' on node D, would it route the mirroring packets
> (the dst MAC is incorrect)
> to node C?
It may work.
Go and try running some experiments.
cheers,
jamal
thomas yang
> On Mon, 2009-09-07 at 09:05 -0700, Xiaofei Wu wrote:
>
>> (1) Could I use pedit action to modify the dst MAC, so the destination node D will accept it,
>> then forward it to node C?
>
> Yes, you can achieve it with pedit;
>
>
>> (or use other tools to modify the dst MAC, please give me more information)
>>
>
> it is as usable as u32 is - you have to know your offsets
> example, here's something done on an incoming packet:
> =-=
> #Note:
> #dst MAC starts at -14
> #src MAC at -8
> #ethertype at -2
> #
> tc filter add dev eth1 parent ffff: protocol ip prio 10 u32 \
> match ip src 192.168.2.11/32 flowid 1:2 \
> action pedit munge offset -14 u16 set 0x0000 \
> munge offset -12 u32 set 0x00000200 \
> munge offset -8 u32 set 0x0aaf0100 \
> munge offset -4 u32 set 0x0008eb06 pipe \
> action mirred egress redirect dev eth0
He want to route the mirroring packets.
" - Mirror takes a copy of the packet and sends it to specified
dev ("port" in ethernet switch/bridging terminology)
- redirect
steals the packet and redirects to specified destination dev. "
So,'mirror' is different from 'redirect'. Change the line 'action
mirred egress redirect dev eth0' to 'action mirred egress mirror dev
eth0' .
Both 'mirror' and 'redirect' can transmit the packets to otner node,
but mirror make a copy, then transmit it; redirect steals the packet,
right ?
--
regards,
thomas
jamal
> He want to route the mirroring packets.
>
> " - Mirror takes a copy of the packet and sends it to specified
> dev ("port" in ethernet switch/bridging terminology)
> - redirect
> steals the packet and redirects to specified destination dev. "
>
> So,'mirror' is different from 'redirect'. Change the line 'action
> mirred egress redirect dev eth0' to 'action mirred egress mirror dev
> eth0' .
> Both 'mirror' and 'redirect' can transmit the packets to otner node,
> but mirror make a copy, then transmit it; redirect steals the packet,
> right ?
>
Yes, of course. That was an example on how to use pedit. If you want
to be pedantic then note that no eth1 device is being used in the
original example and neither is itsensible to make changes to the MAC
address on ingress ;->
In any case, please go and run some experiments to test the theories.
cheers,
jamal
thomas yang
> Yes, of course. That was an example on how to use pedit. If you want
> to be pedantic then note that no eth1 device is being used in the
> original example and neither is itsensible to make changes to the MAC
> address on ingress ;->
> In any case, please go and run some experiments to test the theories.
>
I think the idea of the original example is good, 'tc' is very useful.
I will try some experiments to test the theories. : )
------
regards,
thomas
Xiaofei Wu
I did an experiment. It seems that something is wrong.
>> (1) Could I use pedit action to modify the dst MAC, so the destination node D will accept it,
>> then forward it to node C?
>Yes, you can achieve it with pedit;
>it is as usable as u32 is - you have to know your offsets
>example, here's something done on an incoming packet:
=-=
#Note:
#dst MAC starts at -14
#src MAC at -8
#ethertype at -2
#
>
>
A
/ \
B D
\ /
C
A: eth0, IP 192.168.1.242
waln1, IP 192.168.2.200 ,MAC 00 23 cd af d0 74
D: wlan1, IP 192.168.2.11, MAC 00 23 cd af ec da
wlan2, IP 192.168.4.11
On node A,
1) run 'tc qdisc add dev eth0 handle 1: root prio'
2) run 'tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.1.0/24 flowid 1:16 \
action mirred egress mirror dev wlan1'
Node A sent some packets to C. (path: A-B-C)
I can use 'tcpdump -i wlan1 -e' to capture the packets from eth0 (node A), but I can't forward the mirroring packets to D, (then D forwards them to C).
3 ) run 'tc filter del dev eth0 parent 1: protocol ip prio 10 u32'
then,
'tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.1.0/32 flowid 1:16 \
action pedit munge offset -14 u16 set 0x0023 \
munge offset -12 u32 set 0xcdafecda \
munge offset -8 u32 set 0x0023cdaf \
munge offset -4 u32 set 0xd0740800 pipe \
action mirred egress mirror dev wlan1'
After run 'tcpdump -i wlan1 -e', I can not capture any packets.
I change 'mirror' to 'redirect' ('action mirred egress mirror dev wlan1'), also capture nothing.
Why?
BTW,
'uname -a'
Linux fedora 2.6.27.30-170.2.82.fc10.i686 #1 SMP Mon Aug 17 08:38:59 EDT 2009
i686 i686 i386 GNU/Linux
iproute2:
iproute-2.6.27-2.fc10.i386
regards,
wu
jamal
>
> After run 'tcpdump -i wlan1 -e', I can not capture any packets.
Could it be related to the wireless driver? Here's something i tried
on my laptop
---
dogo:/home/hadi# tc qdisc add dev lo handle 1: root prio
dogo:/home/hadi# tc filter add dev lo parent 1: protocol ip prio 10 u32
match ip src 127.0.0.1/24 flowid 1:16 action pedit munge offset -14 u16
set 0x0023 munge offset -12 u32 set 0xcdafecda munge offset -8 u32 set
0x0023cdaf munge offset -4 u32 set 0xd0740800 pipe action mirred egress
mirror dev eth0
---
On window1: tcpdump -n -i eth0
on window2: ping 127.0.0.2
On window1 i see:
----
dogo:/home/hadi# tcpdump -n -i eth0 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
18:05:23.184602 00:23:cd:af:d0:74 > 00:23:cd:af:ec:da, ethertype IPv4
(0x0800), length 98: 127.0.0.2 > 127.0.0.2: ICMP echo request, id 53329,
seq 1, length 64
18:05:23.558949 00:06:dc:44:4b:ed > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 60: arp who-has 10.0.0.34 tell 10.0.0.33
18:05:24.199184 00:23:cd:af:d0:74 > 00:23:cd:af:ec:da, ethertype IPv4
(0x0800), length 98: 127.0.0.2 > 127.0.0.2: ICMP echo request, id 53329,
seq 2, length 64
--------
Try the exact example, if it doesnt work then you have other problems;
cheers,
jamal
Xiaofei Wu
>> After run 'tcpdump -i wlan1 -e', I can not capture any packets.
>Could it be related to the wireless driver?
Maybe. I will check it.
>Here's something i tried on my laptop
....
>
I tried your example.
-on window1 'ping 127.0.0.2'
....
2616 packets transmitted, 0 received, 100% packet loss
-on window2 'tcpdump -n -i eth0 -e' , i see
....
10:15:06.314420 00:23:cd:af:d0:74 > 00:23:cd:af:ec:da, ethertype IPv4 (0x0800), length 98: 127.0.0.2 > 127.0.0.2: ICMP echo request, id 17419, seq 234, length 64
....
-on window3 'tcpdump -i lo -e'
....
10:15:37.332527 00:23:cd:af:d0:74 (oui Unknown) > 00:23:cd:af:ec:da (oui Unknown), ethertype IPv4 (0x0800), length 98: 127.0.0.2 > 127.0.0.2: ICMP echo request, id 17419, seq 265, length 64
....
It seems that I modify the dst MAC, src MAC of the packets, then transmit to 'lo' and mirror the packects to 'eth0'. (On 'lo', '2616 packets transmitted, 0 received, 100% packet loss' .) How to let 'lo' receive the packets?
But I want to only modify the dst MAC, src MAC of the mirroring packets, transmit them to next hop. (not modify the dst,src MAC of the packets to 'lo'). What should I do?
When I change 'lo' to 'eth1' (or wlan1 ...), node A will have two paths (A-B-C, A-D-C) to transmit the "same"(IP header, data) packets to node C simultaneously.
regards,
wu
jamal
>
> It seems that I modify the dst MAC, src MAC of the packets, then transmit to 'lo' and mirror the packects to 'eth0'.
> (On 'lo', '2616 packets transmitted, 0 received, 100% packet loss' .) How to let 'lo' receive the packets?
By not modifying the packets. I am a little suprised that changing the
Mac address on lo has that effect.
Note it should work on ingress as i described because ingress doesnt
queue packets.
>
> But I want to only modify the dst MAC, src MAC of the mirroring packets, transmit them to next hop.
> (not modify the dst,src MAC of the packets to 'lo'). What should I do?
Ok, so modifying then mirroring wont work on ingress;->
One thing you can try is first to mirror lo->eth0, then pedit only
specific flow on eth0 that came from lo.
cheers,
jamal
Xiaofei Wu
I run your example ( mirror lo -> eth0) on Sep. 10th, got almost the same result(in my last email) as yours.
I think interface 'lo' is very special.
When I do the following (eth0 -> lo), the results are very strange.
1> run 'tc qdisc add dev eth0 handle 1: root prio'
2> tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.1.0/32 flowid 1:16 \
action pedit munge offset -14 u16 set 0x0023 \
munge offset -12 u32 set 0xcdafecda \
munge offset -8 u32 set 0x0023cdaf \
munge offset -4 u32 set 0xd0740800 pipe \
action mirred egress mirror dev lo
window1 run ' ping 192.168.1.1'
window2 'tcpdump -i lo -e', I can not capture any packets.
mirror lo -> eth0 ok, eth0 -> lo can not work ???
2'> change 'action mirred egress mirror dev lo' to 'action mirred egress mirror dev eth1' ,
'tcpdump -i eth1 -e' also capture nothing.
Does this mean something wrong with ' action pedit ...' ? ("offset must be on 32 bit boundaries"?)
>> lo -> eth0
>> But I want to only modify the dst MAC, src MAC of the mirroring packets, transmit them to next hop.
>> (not modify the dst,src MAC of the packets to 'lo'). What should I do?
>Ok, so modifying then mirroring wont work on ingress;->
>One thing you can try is first to mirror lo->eth0, then pedit only
>specific flow on eth0 that came from lo.
How to do this. Could you show me the example commands? Thank you.
regards,
wu
jamal
> I run your example ( mirror lo -> eth0) on Sep. 10th, got almost the same result(in my last email) as yours.
> I think interface 'lo' is very special.
>
> When I do the following (eth0 -> lo), the results are very strange.
> 1> run 'tc qdisc add dev eth0 handle 1: root prio'
>
> 2> tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
> match ip src 192.168.1.0/32 flowid 1:16 \
> action pedit munge offset -14 u16 set 0x0023 \
> munge offset -12 u32 set 0xcdafecda \
> munge offset -8 u32 set 0x0023cdaf \
> munge offset -4 u32 set 0xd0740800 pipe \
> action mirred egress mirror dev lo
>
> window1 run ' ping 192.168.1.1'
> window2 'tcpdump -i lo -e', I can not capture any packets.
>
I think you are doing something wrong. Are there really packets
being generated with that source address.
I just did:
----
tc qdisc add dev eth0 handle 1: root prio
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 match ip dst
10.0.0.27 flowid 1:16 action pedit munge offset -14 u16 set 0x0023 munge
offset -12 u32 set 0xcdafecda munge offset -8 u32 set 0x0023cdaf munge
offset -4 u32 set 0xd0740800 pipe action mirred egress mirror dev lo
----
I then ping 10.0.0.27 and i can see the packets on tcpdump lo,
> mirror lo -> eth0 ok, eth0 -> lo can not work ???
>
> 2'> change 'action mirred egress mirror dev lo' to 'action mirred egress mirror dev eth1' ,
> 'tcpdump -i eth1 -e' also capture nothing.
> Does this mean something wrong with ' action pedit ...' ? ("offset must be on 32 bit boundaries"?)
>
Just make sure it all works first. Perhaps you need to run tcpdump with
-n to avoid name lookup or make sure you are not just arping and not
issuing icmp etc.
>
> >> lo -> eth0
> >> But I want to only modify the dst MAC, src MAC of the mirroring packets, transmit them to next hop.
> >> (not modify the dst,src MAC of the packets to 'lo'). What should I do?
>
> >Ok, so modifying then mirroring wont work on ingress;->
> >One thing you can try is first to mirror lo->eth0, then pedit only
> >specific flow on eth0 that came from lo.
>
> How to do this. Could you show me the example commands? Thank you.
>
Add the rule to mirror on lo
Add the rule to pedit for mirrored packet on eth0
cheers,
jamal
Xiaofei Wu
> I think you are doing something wrong. Are there really packets
>being generated with that source address.
I made a mistake. I changed it to 'match ip src 192.168.1.0/24' . Now it works.
I can mirror the packets (A -> B) to D. Node D forwards them to C.
Sometimes the same packets through the path A-B-C, A-D-C are not lost. Node C will receive the
same packets twice. How to handle the duplicate packet on node C ?
Maybe the duplicate packets have side effects to TCP.
regards,
wu
jamal
> I can mirror the packets (A -> B) to D. Node D forwards them to C.
> Sometimes the same packets through the path A-B-C, A-D-C are not lost. Node C will receive the
> same packets twice. How to handle the duplicate packet on node C ?
>
> Maybe the duplicate packets have side effects to TCP.
>
I will leave this to you. This is where you go and do interesting
things, publish your results and maybe write a paper.
I dont think its a big deal to receive duplicate packets.
cheers,
jamal
Xiaofei Wu
>> How to do this. Could you show me the example commands? Thank you.
>>
>Add the rule to mirror on lo
>Add the rule to pedit for mirrored packet on eth0
I did two expriments. One is OK. The result of the other is not the same as I expected. I don't know why.
(1)
A
| |
C
A: eth0 192.168.1.242/24
wlan1 192.168.4.5/24
C: wlan1 192.168.4.202/24
eth0 192.168.1.215/24
On node A, I mirrored packets to wlan1(eth0 -> wlan1), modified dst,src MAC (transmit to wlan1 of node C).
When I run 'ping 192.168.1.215' on node A, one request will get two replies. It's OK.
(2)
A
/ |
B |
\ |
C
A: eth0 192.168.1.242/24
wlan1 192.168.2.5/24
B: wlan1 192.168.2.11/24
wlan2 192.168.4.11/24
C: wlan1 192.168.4.202/24
eth0 192.168.1.215/24
On node A, I run this to mirror, pedit packets.
---
#tc qdisc add dev eth0 handle 1: root prio
#tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.1.0/24 flowid 1:16 \
action mirred egress mirror dev wlan1
#tc qdisc add dev wlan1 handle 1: root prio
#tc filter add dev wlan1 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.1.0/24 flowid 1:16 \
action pedit munge offset -14 u16 set 0x0023 \
munge offset -12 u32 set 0xcdafecda \
munge offset -8 u32 set 0x0023cdaf \
munge offset -4 u32 set 0xd0740800
---
the routing table 0f node B
---
#route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.4.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan2
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan1
0.0.0.0 192.168.4.202 0.0.0.0 UG 0 0 0 wlan2
#cat /proc/sys/net/ipv4/ip_forward
1
---
On node A I run 'ping 192.168.1.215'(IP addr of node C eth0) on node A, one request 'only' get one reply. It's strange.
On node B,
window1: 'tcpdump -i wlan1 -n -e', I can see the mirroring packets.
window2: 'tcpdump -i wlan2 -n -e', I see noting.
It seems that node B didn't forward the mirroring packects. So I did anotner experiment to check it.
I am sure node B can forward packets. But it didn't forward the mirroring packets, why? (something wrong with the mirroring packets?)
regards,
wu
jamal
> On node A I run 'ping 192.168.1.215'(IP addr of node C eth0) on node A, one request 'only' get one reply. It's strange.
> On node B,
> window1: 'tcpdump -i wlan1 -n -e', I can see the mirroring packets.
> window2: 'tcpdump -i wlan2 -n -e', I see noting.
> It seems that node B didn't forward the mirroring packects. So I did anotner experiment to check it.
> I am sure node B can forward packets. But it didn't forward the mirroring packets, why? (something wrong
> with the mirroring packets?)
>
Please put some effort - these are things you can resolve on your
own. Many possibilities on node B:
- run netstats to see forwarding stats
- run ping from node B to .215 to see if you can reach it.
- check ARP tables
- add explicit route to .215
- check rpfilter
cheers,
jamal
Xiaofei Wu
I come across another problem.
network topology:
M
|
A
/ \
B D
\ /
C
node M < ---- > node C
common path: M-A-B-C
the other path: M-A-D-C
With your help I can mirror the outgoing packets(node A wlan0) to wlan1(node A), then transmit it to D. D will route them to C.
There will be another problem.
When the link A-B is not available, there is no packect going out to mirror, node M could not get to node C. (if B is broken, A use ARP to ask the MAC of B's IP addr, but no reply)
So I want to forward the incoming packets( node M -> A(eth0) ) to wlan0(node A) and wlan1(node A) at the same time, route them separately. In this case, if one path is unavailable, it will not influence the other path.
Could iproute2 'tc' do this?
regards,
wu
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
jamal
> When the link A-B is not available,
How do you detect this? Is it a path that is broken or a link?
> Could iproute2 'tc' do this?
you could use iproute2 'ip' which supports multi-nexthops, example
off top of my head (for exact syntax run "ip route help")
----
ip route add blah/24 nexthop via a.b.c.d dev wlan0 \
nexthop via e.f.g.h dev wlan1
----
You will probably need to help it by flushing route cache.
You may need to worry about loops on mirroring when the route flipping
happens and need to optimize for that. Please experiment.
cheers,
jamal