Network Routing Problem
uunet
I have a WAN set up between two sites. Without going into two many details,
users within Site#1 connect through a VPN via Internet an ISDN connection to
Site#2 where they access the host server located at Site#2 (Site#2 has an
ADSL Internet connection). In essence, I've created an IPSec tunnel.
This host server is running SCO OpenServer UNIX 5.0.5
In order to allow the users to see the SCO box and vice server, I've had to
add static routes for all the remote user PC's (5 users are running Windows
95/98) in the form:
route add -host user1-ip gateway-ip
route add -host user2-ip gateway-ip
route add -host user3-ip gateway-ip
route add -host user4-ip gateway-ip
route add -host user5-ip gateway-ip
Works great until SCO mysteriously drops the route from the routing table (I
do a 'netstat -r' and they are gone!). I was under the impression that
static routes are permanent and should never be dropped unless I delete them
from the command shell.
Can anyone shed some light on this perplexing problem? In essence, I want
these routes to remain.
Thanks,
Don
Jeff Liebermann
They should be permanent until you reboot or until some routing
protocol comes along and changes them. Did you add those lines to
some startup file? I use:
/etc/rc2.d/S99routed
Did someone run:
route -flush
which will vaporize all your gateways? Try it watch them all
disappear.
Also, turn off RIP. Kill the "routed" daemon. Something on your net
is belching bogus RIP updates and probably vaporizing the routing
table. Edit the file:
/etc/tcp
and comment out the section that starts routed. This was done in
3.2v5.0.6 but in a rather convuluted manner.
--
Jeff Liebermann 150 Felker St #D Santa Cruz CA 95060
(831)421-6491 pgr (831)426-1240 fax (831)336-2558 home
http://www.LearnByDestroying.com WB6SSY
je...@comix.santa-cruz.ca.us je...@cruzio.com
Don
verified that both were not running. Unfortunately, the routes are still being dropped. Issuing a 'netstat -ra', I see the
following for the dropped route:
User-IP (incomplete) UHLSP- 1 0 net2
I have put the entries for the static routes into the /etc/rc file. Is there anything else I can troubleshoot or do I just chalk
this up to a non standard SCO'ism?
Thanks,
Don
"Jeff Liebermann" <je...@comix.santa-cruz.ca.us> wrote in message news:vu65ot4um08kmeegu...@4ax.com...
Jeff Liebermann
/etc/rc to /etc/rc2.d/S99route
The IP stack must be alive and running before you can issue route
commands. I'm not sure at what point the routes are established when
started from /etc/rc, but if you put them in S99route, they will be
correctly started last.
Your:
netstat -ra
incantation is either heavily edited or broken. It should show a
route to localhost and a default route, neither of which require the
collection of static routes previously discussed. Try:
netstat -rn
instead and see if the output contains at least something like:
=> netstat -rn
Routing tables
Destination Gateway Flags Refs Use Interface
127.0.0.1 127.0.0.1 UH 1 0 lo0
default 192.168.111.33 UG 3 10780 wdn0
192.168.111 192.168.111.1 U 0 3136 wdn0
(Note: This is from my 3.2v4.2 system. Your 3.2v5.0.5 will be
different).
Robert
Thanks for the suggestion, but the routes are being correctly issued in the
current setup. The problem is much more directly related to the fact that
OpenServer seems to be dropping static routes to hosts via specified IP
gateways, in favor of a "Link-Level" address (ie: ARP address to an
interface, rather than an IP address...). Let me explain...
When the static routes are first issued, they correctly point to the IP
gateway that they belong to. As in this example:
host-A 192.168.0.254 UGHS 1 0 e0
As you can see, host-A's gateway address is correctly set, and all flags
are correct. Now, what happens is that eventually the setting changes,
where the gateway IP is completely dropped, and the flags are changed.. As
this example shows:
host-A (incomplete) UHLS 0 0 e0
Now, with 'routed' turned off, this (incomplete) is a new entity. With it
on, we get a lovely little ARP/MAC address of:
host-A 00:50:0e:xx:yy:zz UHLS 0 0 e0
when we do a 'netstat -ra' , as that is the only way to find that entry at
this point. IE: the routing entry for that host DOES EXIST, but it has now
been setup to reply to it's last known interface address (ie: the MAC
address of the gateway it came from), instead of the IP address. But only
when 'routed' is running at a minimum. With NO routing daemons at all, the
entry becomes (incomplete) which is basically it's way of saying it hasn't
got a clue what the MAC/ARP address is.
Now, the situation gets a little more confusing, but the following will help
pinpoint where I believe the problem stems from.
When we are connected to our Cisco 2501's for inter-office communications,
the routing entries do appear as indicated on the first example above. But
these routers are being taken out of service very shortly. So we have opted
to implement a pair of IPSEC Linux gateway routers to replace them across a
pair of dedicated internet routers from WorldCom/UUNET. This is far
cheaper than our current dedicated line between two cities, and from what we
have seen so far, a faster solution as well. However, the VPN Routers show
an MTU of 16260 on the secured interfaces to the internal LANs, whereas the
Cisco's simply provide the default MTU of 1500 for their ethernet
connection. Now, everytime we hook up the VPN Routers to the remote and
local segments, the OpenServers immediately add on 2 additional flags to the
static routes. They are "P" and "N", indicating "PMTU discovery enabled"
and "New MTU found".. They do NOT do this when hooked to the Cisco's.
Now, within a few minutes of use (or amount of traffic passed over time, not
sure), that's when the static routes are dropped and replaced with the
ARP/MAC address values. But these VPN Routers cannot handle just any
traffic showing up at an interface, it must appear with the proper IP
addresses from the OpenServers to pass through the secure tunnel. The
Cisco's basically dont care, and simply shoot anything it sees right across
the link.
This behaviour is consistent across several attempts at implementing the VPN
routers. As soon as they're plugged in, PMTU discovery is turned on, and
shortly after, the route(s) is/are dropped. If we leave the Cisco's in
place, PMTU is never enabled, so everything stays copacetic, and it just
keeps on going. I tried to override the default MTU of the VPN routers to
1500 as well, to provide an identical MTU value between them and the
Cisco's. But the darn OpenServer's still picked up the changes and blew
away the routes again. If I manually issue the static routes again, then
all is restored back into service, until the route(s) get dropped once
again.
So, it appears that the PMTU Discovery mechanism, whether in conjunction
with the ARP protocol, 'routed' daemon, or both, just isn't doing it's job
properly. I can understand the need to drop a route to relearn/implement a
new MTU to "optimize" transmission rates on a circuit, but when you relearn
it, shouldn't it also replace the default gateway address with it's
associated IP address as well? And not simply use the ARP/MAC address value
as it is doing now?
BTW: All of the other servers and hosts (SCO Unix 3.2, Linux 2.2.19, Win9x
clients, etc) work just fine. None of these ever drop routes in favor of
any ARP/MAC addresses, and always keep trying the default gateway IP's as
needed. It's only the OpenServer boxes that do this. The gateways (both
Cisco and VPN Routers) work just fine in transporting the traffic, say from
the Win9x clients in one office to the Linux Internet gateway in the other
office. So it's definitely an OpenServer issue here.
So, the issue is one of the following:
1) Can we fix OpenServer so that when it decides to drop a route during the
MTU optimization process, that it will pickup or relearn the correct IP
address of the default gateway instead of the MAC/ARP address?
2) Can we permanently disable the PMTU Option, so that it will never drop a
route, and only use the static IP gateways that we have setup ? Is it
possible to ONLY use IP routes in gateway values, and disable MAC/ARP
addressing for all routing table entries?
3) Will we have to implement some sort of ARP / Proxy Arp / Reverse Arp
lookup on OpenServer, VPN Gateway, or both so that we can get them to
co-operate together if the PMTU can't be disabled? In other words, let
OpenServer do all the optimization it wants, but at least get it working
properly so that each host route is properly recreated with a valid set of
IP addresses?
Thanks for any insight, input, and help!!
"Jeff Liebermann" <je...@comix.santa-cruz.ca.us> wrote in message
news:gl0hotoljupc959ka...@4ax.com...
Bill Vermillion
Robert <robe...@hotmail.com> wrote:
I may be way off here - but being the fool that I am I press ahead.
>When the static routes are first issued, they correctly point to the IP
>gateway that they belong to. As in this example:
>host-A 192.168.0.254 UGHS 1 0 e0
>As you can see, host-A's gateway address is correctly set, and all
>flags are correct. Now, what happens is that eventually the setting
>changes, where the gateway IP is completely dropped, and the flags
>are changed.. As this example shows:
>host-A (incomplete) UHLS 0 0 e0
>Now, with 'routed' turned off, this (incomplete) is a new entity.
>With it on, we get a lovely little ARP/MAC address of:
>host-A 00:50:0e:xx:yy:zz UHLS 0 0 e0
>when we do a 'netstat -ra' , as that is the only way to find that
>entry at this point. IE: the routing entry for that host DOES
>EXIST, but it has now been setup to reply to it's last known
>interface address (ie: the MAC address of the gateway it came
>from), instead of the IP address. But only when 'routed' is running
>at a minimum. With NO routing daemons at all, the entry becomes
>(incomplete) which is basically it's way of saying it hasn't got a
>clue what the MAC/ARP address is.
Communications is done mac to mac after finding it from the IP
address. However something is definately missing here.
Just a thought. Have you tried binding the IP to the MAC address
via a static arp entry - eg the arp -s function. If so does
that function correct.
Second thought - when everything is working as it should at first -
is the MAC address the same then as it shows above?
Check that out. One way to find it easily is to drop into your
Cisco and and show arp-cache or maybe show ip arp-cache - havne't
used that in a while since I don't have my big router to maintain
anymore.
The reason I ask about this is that at times I have made
a mistreak [hard as it seems to believe from my point of view]
on another system and my arpwatch daemon starts spewing forth
messages every few seconds about the arp address changing on an
address. That happened because two IPs were assigned to the same
address on a router/dsl device.
So check to see if the mac address is changing and if not try
setting a static arp entry. That may give a clue as to the
problem.
>Now, the situation gets a little more confusing, but the following
>will help pinpoint where I believe the problem stems from.
>When we are connected to our Cisco 2501's for inter-office
>communications, the routing entries do appear as indicated on the
>first example above. But these routers are being taken out of
>service very shortly. So we have opted to implement a pair of IPSEC
>Linux gateway routers to replace them across a pair of dedicated
>internet routers from WorldCom/UUNET.
Ah. Then try checking the MAC addresses. You might also go
into the cisco an clear the arp cache. I had one site that had
some machines off of a network I was maintaining and he called
frustrated as he was putting in new servers. He said he'd tried
several times but when he switched to the new machines nothing
happened. As he was on the phone I cleared the arp-cache and
before I could say try it now he said "It's working".
>Now, within a few minutes of use (or amount of traffic passed over
>time, not sure), that's when the static routes are dropped and
>replaced with the ARP/MAC address values. But these VPN Routers
>cannot handle just any traffic showing up at an interface, it must
>appear with the proper IP addresses from the OpenServers to pass
>through the secure tunnel. The Cisco's basically dont care, and
>simply shoot anything it sees right across the link.
I still think it has to do with the MAP/IP mapping.
I'm still trying to learn a lot of this but a gut feeling says it
has to do with this and not the MTU. You may be mis-leading
yourself.
Bill
--
Bill Vermillion - bv @ wjv . com
Robert
Thanks for the input... But I think I have left out a few important
details, which I will outline below...
"Bill Vermillion" <bi...@wjv.com> wrote in message
news:GIrHq...@wjv.com...
> In article <dBDi7.19980$Jp4.1...@weber.videotron.net>,
> Robert <robe...@hotmail.com> wrote:
>
> I may be way off here - but being the fool that I am I press ahead.
>
> >When the static routes are first issued, they correctly point to the IP
> >gateway that they belong to. As in this example:
>
> >host-A 192.168.0.254 UGHS 1 0 e0
>
> >As you can see, host-A's gateway address is correctly set, and all
> >flags are correct. Now, what happens is that eventually the setting
> >changes, where the gateway IP is completely dropped, and the flags
> >are changed.. As this example shows:
>
> >host-A (incomplete) UHLS 0 0 e0
>
> >Now, with 'routed' turned off, this (incomplete) is a new entity.
> >With it on, we get a lovely little ARP/MAC address of:
>
> >host-A 00:50:0e:xx:yy:zz UHLS 0 0 e0
>
> >when we do a 'netstat -ra' , as that is the only way to find that
> >entry at this point. IE: the routing entry for that host DOES
> >EXIST, but it has now been setup to reply to it's last known
> >interface address (ie: the MAC address of the gateway it came
> >from), instead of the IP address. But only when 'routed' is running
> >at a minimum. With NO routing daemons at all, the entry becomes
> >(incomplete) which is basically it's way of saying it hasn't got a
> >clue what the MAC/ARP address is.
>
> Communications is done mac to mac after finding it from the IP
> address. However something is definately missing here.
Correct... something is missing. And it's the reverse ARP. IE: getting the
proper IP address from the MAC address.
>
> Just a thought. Have you tried binding the IP to the MAC address
> via a static arp entry - eg the arp -s function. If so does
> that function correct.
No, I haven't. But there is a reason for this. Which will clarify the rest
of the situation below. First part is the fact that I am REPLACING the
Cisco's with these VPN Router boxes. So I'd have to bind the VPN Router MAC
address to the IP address first AFTER doing the switch to them. But then
that would make the original Cisco's unuseable if we had to switch back
until I cleared those static entries and put in the other ones.
>
> Second thought - when everything is working as it should at first -
> is the MAC address the same then as it shows above?
When everything is working out, I have the proper static IP gateways for the
remote hosts, and the correct MAC for the router in the tables. The ARP
cache does NOT show these remote hosts either. When we switch over, the
same router name now adopts the new MAC address, but the remote host routes
get whacked, and when that happens, thats when they appear in the ARP cache.
We never have duplicate IP's on the segment, but we do have changes in the
MAC addresses when we switch over from the Cisco's to the VPN Routers. But
that settles down pretty quickly. I've cleared the router addresses
manually, and via a 'ping' (which forces a reload/relearn by the way, but it
sometimes takes a bit of time before the ARP cache expires the existing
entry first).
>
> Check that out. One way to find it easily is to drop into your
> Cisco and and show arp-cache or maybe show ip arp-cache - havne't
> used that in a while since I don't have my big router to maintain
> anymore.
It's 'show ip arp-cache', and I can't do that when they're disconnected as
indicated above. Remember, these Cisco's are being replaced. Sorry if I
didn't make that clear the first time I stated my problem.
>
> The reason I ask about this is that at times I have made
> a mistreak [hard as it seems to believe from my point of view]
> on another system and my arpwatch daemon starts spewing forth
> messages every few seconds about the arp address changing on an
> address. That happened because two IPs were assigned to the same
> address on a router/dsl device.
Yep, I've seen this happen many a time. When we do the switchovers, I
basically UNPLUG the Cisco's physically, then plug in the VPN Routers. This
prevents the ARP daemon's from going nuts. Same goes for the vice-versa.
>
> So check to see if the mac address is changing and if not try
> setting a static arp entry. That may give a clue as to the
> problem.
>
> >Now, the situation gets a little more confusing, but the following
> >will help pinpoint where I believe the problem stems from.
>
> >When we are connected to our Cisco 2501's for inter-office
> >communications, the routing entries do appear as indicated on the
> >first example above. But these routers are being taken out of
> >service very shortly. So we have opted to implement a pair of IPSEC
> >Linux gateway routers to replace them across a pair of dedicated
> >internet routers from WorldCom/UUNET.
>
> Ah. Then try checking the MAC addresses. You might also go
> into the cisco an clear the arp cache. I had one site that had
> some machines off of a network I was maintaining and he called
> frustrated as he was putting in new servers. He said he'd tried
> several times but when he switched to the new machines nothing
> happened. As he was on the phone I cleared the arp-cache and
> before I could say try it now he said "It's working".
Ahhh, I see some of the confusion here. Let me clarify. The VPN Routers
that are plugged into the dedicated UUNET routers are the brand-new setup.
IE: I have the VPN Routers being our internet gateways, which I then have
plugged into the UUNET routers. THIS PART IS CONSTANT, and never gets
changed. The existing Cisco routers on the dedicated line are ANOTHER pair,
using another commlink. So what get's switched around is: OLD CISCO's on
ISDN vs VPN Boxes/New Routers on DSL. The only connection that gets
switched is the LAN connections for the INTERNAL LAN support (ie: unplug
Ethernet0 from the old 2501's, and plug in the Eth0 on the VPN boxes, or
vice-versa). The comms side of the setup never gets touched. So, the only
arp cache that can get confused is the one on the OpenServers, as they're
the ones seeing a change in MAC address vs IP address. And that's where my
problem is.
>
> >Now, within a few minutes of use (or amount of traffic passed over
> >time, not sure), that's when the static routes are dropped and
> >replaced with the ARP/MAC address values. But these VPN Routers
> >cannot handle just any traffic showing up at an interface, it must
> >appear with the proper IP addresses from the OpenServers to pass
> >through the secure tunnel. The Cisco's basically dont care, and
> >simply shoot anything it sees right across the link.
>
> I still think it has to do with the MAP/IP mapping.
>
> I'm still trying to learn a lot of this but a gut feeling says it
> has to do with this and not the MTU. You may be mis-leading
> yourself.
You could be right Bill. I do believe that it's something to do with the
ARP values as well. But what I was trying to point out was that this
problem ONLY OCCURS when the OpenServer's enable PMTU discovery on a host
route. That's the only time the static routes get dropped, in favor of an
ARP address. And that the OpenServer boxes are the only ones that do this.
None of the other SCO Unix nor Linux machines nor client (Win9x boxes) have
this problem. Everyone else is happy with either router (Cisco 2501 or VPN
Router) in place. So maybe the problem is coming from somewhere else, but
this is what I have seen, and can consistently reproduce as a problem.
Thanks for your input tho.. I'll keep on looking.
Robert
Bill Vermillion
Robert <robe...@hotmail.com> wrote:
>Hi Bill,
>
> Thanks for the input... But I think I have left out a few important
>details, which I will outline below...
It's those damn details again. :-(
[snippety snip snip snip]
>> Communications is done mac to mac after finding it from the IP
>> address. However something is definately missing here.
>Correct... something is missing. And it's the reverse ARP. IE:
>getting the proper IP address from the MAC address.
And it appeared to me that the MAC was changing and the IP was
still assocaited with the old MAC. Since the router was no longer
on the net all you got was a MAC address.
>> Just a thought. Have you tried binding the IP to the MAC address
>> via a static arp entry - eg the arp -s function. If so does
>> that function correct.
>No, I haven't. But there is a reason for this. Which will clarify
>the rest of the situation below. First part is the fact that I am
>REPLACING the Cisco's with these VPN Router boxes. So I'd have to
>bind the VPN Router MAC address to the IP address first AFTER doing
>the switch to them. But then that would make the original Cisco's
>unuseable if we had to switch back until I cleared those static
>entries and put in the other ones.
Then clear the arp tables in the local machines - or the one that
only shows a MAC address. That is making the assumption that the IP
is still in that table.
>> Second thought - when everything is working as it should at first -
>> is the MAC address the same then as it shows above?
>We never have duplicate IP's on the segment, but we do have changes
>in the MAC addresses when we switch over from the Cisco's to the
>VPN Routers. But that settles down pretty quickly. I've cleared
>the router addresses manually, and via a 'ping' (which forces a
>reload/relearn by the way, but it sometimes takes a bit of time
>before the ARP cache expires the existing entry first).
And do the individual machines still show the IP bound to an old
MAC.
>It's 'show ip arp-cache', and I can't do that when they're
>disconnected as indicated above. Remember, these Cisco's are being
>replaced. Sorry if I didn't make that clear the first time I stated
>my problem.
And if they are disconnected then they wouldn't be affecting the
network.
>> Ah. Then try checking the MAC addresses. You might also go
>> into the cisco an clear the arp cache. I had one site that had
>> some machines off of a network I was maintaining and he called
>> frustrated as he was putting in new servers. He said he'd tried
>> several times but when he switched to the new machines nothing
>> happened. As he was on the phone I cleared the arp-cache and
>> before I could say try it now he said "It's working".
>Ahhh, I see some of the confusion here. Let me clarify. The VPN
>Routers that are plugged into the dedicated UUNET routers are the
>brand-new setup. IE: I have the VPN Routers being our internet
>gateways, which I then have plugged into the UUNET routers. THIS
>PART IS CONSTANT, and never gets changed. The existing Cisco
>routers on the dedicated line are ANOTHER pair, using another
>commlink. So what get's switched around is: OLD CISCO's on ISDN vs
>VPN Boxes/New Routers on DSL.
I was assuming T1's changing to DSL. Because you had said cost.
Locally DSL's with dedicated IP's are more costly and ISDN - BUT -
several of the larger local ISP have been purchased and their modem
banks are gone and now they are outsourced. With [for example]
on bank of over 30,000 modems you you can't really run bonded ISDN
because you can't even guarantee you'll come in on the same T3 [one
to each box of about 600+ modems] let alone the same PRI within
that T3. I may be wrong but think when you bonded ISDN' the pairs
have to be on the same PRI. So all the ISPs locally now only have
single channel [64K] ISDN.
>The only connection that gets switched is the LAN connections for
>the INTERNAL LAN support (ie: unplug Ethernet0 from the old 2501's,
>and plug in the Eth0 on the VPN boxes, or vice-versa). The comms
>side of the setup never gets touched. So, the only arp cache that
>can get confused is the one on the OpenServers, as they're the ones
>seeing a change in MAC address vs IP address. And that's where my
>problem is.
And what happens if you clear the apr tables on the OpenServer?
>> I still think it has to do with the MAP/IP mapping.
>> I'm still trying to learn a lot of this but a gut feeling says it
>> has to do with this and not the MTU. You may be mis-leading
>> yourself.
>You could be right Bill. I do believe that it's something to do
>with the ARP values as well. But what I was trying to point out was
>that this problem ONLY OCCURS when the OpenServer's enable PMTU
>discovery on a host route.
That MTU seemed awfully high to me - but then again the stuff I'm
working with is either 1500 on the ethernets or 4470 on the ATMs
> That's the only time the static routes get dropped, in favor of an
>ARP address. And that the OpenServer boxes are the only ones that
>do this. None of the other SCO Unix nor Linux machines nor client
>(Win9x boxes) have this problem. Everyone else is happy with either
>router (Cisco 2501 or VPN Router) in place. So maybe the problem is
>coming from somewhere else, but this is what I have seen, and can
>consistently reproduce as a problem.
Do you have any routing protocols turned on in the OpenServer
machine? That might be causing this.
I'm out of ideas.
Robert
More info below...
"Bill Vermillion" <bi...@wjv.com> wrote in message news:GIsEu...@wjv.com...
> In article <EXMi7.26455$Jp4.1...@weber.videotron.net>,
> Robert <robe...@hotmail.com> wrote:
> >Hi Bill,
> >
> > Thanks for the input... But I think I have left out a few important
> >details, which I will outline below...
>
> It's those damn details again. :-(
>
> [snippety snip snip snip]
>
> >> Communications is done mac to mac after finding it from the IP
> >> address. However something is definately missing here.
>
> >Correct... something is missing. And it's the reverse ARP. IE:
> >getting the proper IP address from the MAC address.
>
> And it appeared to me that the MAC was changing and the IP was
> still assocaited with the old MAC. Since the router was no longer
> on the net all you got was a MAC address.
Valid assumption, but the sequence of events showed me that this wasn't the
case. I do get a MAC address, and it is from the correct device... so the
packets will be sent to the correct interface.. But when this happens, the
VPN Router ignores it... It only wants to see a packet destined to the
correct IP address.. not to the interface...
>
> >> Just a thought. Have you tried binding the IP to the MAC address
> >> via a static arp entry - eg the arp -s function. If so does
> >> that function correct.
>
> >No, I haven't. But there is a reason for this. Which will clarify
> >the rest of the situation below. First part is the fact that I am
> >REPLACING the Cisco's with these VPN Router boxes. So I'd have to
> >bind the VPN Router MAC address to the IP address first AFTER doing
> >the switch to them. But then that would make the original Cisco's
> >unuseable if we had to switch back until I cleared those static
> >entries and put in the other ones.
>
> Then clear the arp tables in the local machines - or the one that
> only shows a MAC address. That is making the assumption that the IP
> is still in that table.
I have done so, several times. But when the MAC address shows up, the IP
address is dropped, so my intent was to see if the OpenServer box would
pickup the IP address from the new MAC address. It didn't. So this
assumption is sadly incorrect. I either have the IP address of the gateway
in those static host routes, *OR* I have the MAC address as the
destination.. never both.
>
> >> Second thought - when everything is working as it should at first -
> >> is the MAC address the same then as it shows above?
>
> >We never have duplicate IP's on the segment, but we do have changes
> >in the MAC addresses when we switch over from the Cisco's to the
> >VPN Routers. But that settles down pretty quickly. I've cleared
> >the router addresses manually, and via a 'ping' (which forces a
> >reload/relearn by the way, but it sometimes takes a bit of time
> >before the ARP cache expires the existing entry first).
>
> And do the individual machines still show the IP bound to an old
> MAC.
On the static host routes in OpenServer, there are no ARP entries (until the
static route is dropped/replaced). For the gateway address itself in
OpenServer, immediately following the switchover, yes the old MAC address is
there. Which either clears itself automatically after 30 seconds, or when I
do a manual ARP removal. The individual host routes are on the other side
of the gateway, so those machines themselves only see the gateway MAC
address. And yes, they clear up on their own just fine.
>
> >It's 'show ip arp-cache', and I can't do that when they're
> >disconnected as indicated above. Remember, these Cisco's are being
> >replaced. Sorry if I didn't make that clear the first time I stated
> >my problem.
>
> And if they are disconnected then they wouldn't be affecting the
> network.
Correct!!! Nor would they be able to affect the ARP cache on the
OpenServers... (or any other machine for that matter... LoLL)
LoLLL.. I only wish the budget was there for T1's... And yes, your
explanation on bonded channels for ISDN is correct. But the old commlink
ISDN was a dedicated line between the two cities, using a matched pair of
ISDN modems.. They bonded just fine. In our case, our new ISDN circuit is
dedicated hosting via UUNET.. so they've provided us with a bonded setup at
128Kb... and it's working just fine too.. The home office got a commercial
DSL link at 640Kb upstream, and 2.2Mb downstream... we eventually plan on
adding other offices into this scenario (but not until I can get this
headache fixed....)
>
>
> >The only connection that gets switched is the LAN connections for
> >the INTERNAL LAN support (ie: unplug Ethernet0 from the old 2501's,
> >and plug in the Eth0 on the VPN boxes, or vice-versa). The comms
> >side of the setup never gets touched. So, the only arp cache that
> >can get confused is the one on the OpenServers, as they're the ones
> >seeing a change in MAC address vs IP address. And that's where my
> >problem is.
>
> And what happens if you clear the apr tables on the OpenServer?
Nothing. They come back with the correct MAC addresses, as expected. But
the VPN router wont route them until they see a valid IP address first.
>
> >> I still think it has to do with the MAP/IP mapping.
>
> >> I'm still trying to learn a lot of this but a gut feeling says it
> >> has to do with this and not the MTU. You may be mis-leading
> >> yourself.
>
> >You could be right Bill. I do believe that it's something to do
> >with the ARP values as well. But what I was trying to point out was
> >that this problem ONLY OCCURS when the OpenServer's enable PMTU
> >discovery on a host route.
>
> That MTU seemed awfully high to me - but then again the stuff I'm
> working with is either 1500 on the ethernets or 4470 on the ATMs
True enough. It was a high MTU to ensure that the entire frame could get
encapsulated and encrypted across the IPSEC link without fragmenting it. It
was left to the physical Ethernet transmission MTU to fragment as necessary
the actual transmission. But as mentioned before, I cranked it down to
match the Ethernet MTU of 1500 to no avail... It still decided to enable
PMTU Discovery.
>
> > That's the only time the static routes get dropped, in favor of an
> >ARP address. And that the OpenServer boxes are the only ones that
> >do this. None of the other SCO Unix nor Linux machines nor client
> >(Win9x boxes) have this problem. Everyone else is happy with either
> >router (Cisco 2501 or VPN Router) in place. So maybe the problem is
> >coming from somewhere else, but this is what I have seen, and can
> >consistently reproduce as a problem.
>
> Do you have any routing protocols turned on in the OpenServer
> machine? That might be causing this.
Nope. All dead.. gone... disabled. nothing left... it's really a basic
server now... But I'm going to re-enable 'routed' (and keep forwarding
turned off)... and I'm thinking of trying to mesh ARP processing between the
two. ARP was enabled on the physical interface (Eth0) of the VPN Router,
but not on the IPSEC route tunnel. It is now, but with all the routing
daemons disabled on OpenServer, it's not rediscovering the proper host
routes MAC/ARP address when it drops it (hence the [incomplete] display when
I check it). Before, with 'routed' running (no forwarding), it at least
picked up the proper MAC addresses automatically.
>
> I'm out of ideas.
Thanks for your help Bill. I know I'm close to resolving this, just wish I
had a copy of OpenServer here to load up onto a machine to try it out with
offline. But hopefully, if you are ARP/MAC saavy, we can figure this out.
I'm still convinced that if I can get PMTU discovery disabled, then it will
have no reason to drop the static host routes, and the problem will be
solved, and everything will work just as it does on all the other machines.
But if that proves impossible, then trying to get a proper ARP/MAC address
resolution fix into this scenario would probably work just as well.
Jean-Pierre Radley
| Why not just add those hosts to /etc/hosts? Works for me.
Adding a host to etc/hosts just helps to quickly get its IP number. It
has zilch to do with routing packets to that host or to any other host.
--
JP
Jeff Liebermann
The contents of /etc/hosts has nothing to do with establishing a route
to a remote computer. You could put anything you want into /etc/hosts
and you still will not accomplish anything useful for routeing. What
/etc/hosts does is a crude form of DNS (Don't kNow Service). If you
have a computer name that constantly needs to be converted into an IP
address, put it in /etc/hosts and you won't need to go to the
nameserver every time that name needs to be resolved. It's also
required for local network machine names that are NOT known to a DNS
server.
On the other foot, routeing is a bit more complex. Run:
netstat -rn
to see your router table. The way it works (assuming static routes)
is that you start with a list of pre-defined routes to specific
machines and specific networks. For example, if the local network is
a class C (256 IP's) at 192.168.111.XXX, everything destined for that
block of IP's will stay local and not go out via some router or
gateway. If there's a remote block of machines at 192.168.222.xxx,
there will be a gateway shown in the router table, that says:
"everything for 192.168.222.XXX shall go via that gateway"
If there are additional blocks of IP's, each will have a gateway which
designates which freeway offramp to take to get to the 192.168.222.xxx
block of IP's.
Near the bottom is the default route, which points to a gateway to
take if the destination IP is NOT local, and NOT one of the static
routes. Usually, it points to an ISP and the internet.
--