Machine not responding SYN, but IP and MAC are correct

[Victor Preston]

I really have not made any progress with this. I have a machine that does not respond to TCP requests from my router. Using WIreshark I can see that what's going on is that the router sends a SYN to begin a TCP transaction and my Linux machine (running Ubuntu 16.04) fails to return a SYN,ACK, so the request fails. There's only the one router, and one hardware interface on the linux machine. If I run "arp" on the linux machine it shows the correct MAC for the router. Also if I run "arping" pointing to the router's IP it returns with the correct MAC. Furthermore when I look at the packets intercepted by wireshark the ll header has the correct MAC address for the ip. It looks like a "martian packet" log IS created, but, again, it has the right MAC/IP in it. So what's going on? Why does my linux machine think this is a martian packet when all the values are correct?

[tho...@habets.se]

Things that come to mind:
* Firewall blocks TCP (`iptables -L -n -v`, and with `-t nat` and `-t mangle`)
* routing not set up (check with `ip route get 1.2.3.4`)
* `rp_filter` rejects the packet source interface[1]
* When you run wireshark, run with promisc mode off to ensure you
still see the SYN. You should if the MAC is correct.
* Any bridge interfaces? Try sniffing on the `any` pseudointerface.
Maybe the SYNACK is going elsewhere.


[1]
If set to `1` Linux is strict about this:

```
$ sudo sysctl -a | grep [.]rp_filter
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.wlan0.rp_filter = 0
```

[Victor Preston]

On Tuesday, June 12, 2018 at 1:30:23 PM UTC-4, Victor Preston wrote:

I really have not made any progress with this. I have a machine that does not respond to TCP requests from my router. Using WIreshark I can see that what's going on is that the router sends a SYN to begin a TCP transaction and my Linux machine (running Ubuntu 16.04) fails to return a SYN,ACK, so the request fails. There's only the one router, and one hardware interface on the linux machine. If I run "arp" on the linux machine it shows the correct MAC for the router. Also if I run "arping" pointing to the router's IP it returns with the correct MAC. Furthermore when I look at the packets intercepted by wireshark the ll header has the correct MAC address for the ip. It looks like a "martian packet" log IS created, but, again, it has the right MAC/IP in it. So what's going on? Why does my linux machine think this is a martian packet when all the values are correct?

Thomas very nicely sent  suggestions, which I will attempt to answer:

Things that come to mind: 
* Firewall blocks TCP (`iptables -L -n -v`, and with `-t nat` and `-t mangle`)  

                     Firewall is off, UFW is disabled, 

                     iptables -L -n -v reports nothing

                     iptables -L -t nat reports MASQUERADE all anywhere

                     iptables -L -t mangle reports CHECKSUM udp anywhere udp:bootpc CHECKSUM

* routing not set up (check with `ip route get 1.2.3.4`) 

                     ip route <router's ip>:

                             local <router's ip> dev lo src <router's ip> cache(local)

                     ip route <ip of other machine on theNAT that WILL connect>:

                             <ip of other machine on theNAT that WILL connect>  dev eno1(that's the iface on the linux machine in question)  src <ip of the linux machine in question>   cache

  
* `rp_filter` rejects the packet source interface[1] 

                     I tried running with all rp_filter values set to 0, but it did not help

* When you run wireshark, run with promisc mode off to ensure you 
still see the SYN. You should if the MAC is correct. 

                    I would think that promiscuous mode would show MORE than disabling it, but I tried this and still do not see an ACK

* Any bridge interfaces? Try sniffing on the `any` pseudointerface. 

Maybe the SYNACK is going elsewhere. 

                    I don't know of any, and I tried running wireshark on the 'any' interface, and looked carefully and it does not appear anywhere

So the only thing I see that MIGHT be relevant is that the route to the router refers to the loopback device, which certainly would not work. How could that come to be? And how do I set it correctly - it should be on the other side of the eno1 interface - that;s where it comes in, why doesn't it reply on the interface it comes in on?

[tho...@habets.se]

On Thu, 14 Jun 2018 07:31:21 -0700 (PDT), Victor Preston
<vbpr...@gmail.com> said:
> * routing not set up (check with `ip route get 1.2.3.4`)
> ip route <router's ip>:
> local <router's ip> dev lo src <router's ip>
> cache(local)
> ip route <ip of other machine on theNAT that WILL
> connect>:
> <ip of other machine on theNAT that WILL
> connect> dev eno1(that's the iface on the linux machine in question) src
> <ip of the linux machine in question> cache

so "router" is the one seeing the SYN but isn't sending back SYNACK?

> * `rp_filter` rejects the packet source interface[1]
> I tried running with all rp_filter values set to 0,
> but it did not help

Confirmed with sysctl that they were actually set?

> * When you run wireshark, run with promisc mode off to ensure you
> still see the SYN. You should if the MAC is correct.
> I would think that promiscuous mode would show MORE
> than disabling it, but I tried this and still do not see an ACK

Yes the point would be to see fewer packets. This is to confirm that
the MAC address is correct on the other machine's ARP table, since if
you don't use promisc then the packet will be filtered before
wireshark/tcpdump can see it (indeed before any part of the kernel sees it).

> * Any bridge interfaces? Try sniffing on the `any` pseudointerface.
> Maybe the SYNACK is going elsewhere.
> I don't know of any, and I tried running wireshark on
> the 'any' interface, and looked carefully and it does not appear anywhere

You shouldn't have to look too carefully, just do:
sudo tcpdump -nlpi any host 1.2.3.4
(1.2.3.4 being the remote end)

> So the only thing I see that MIGHT be relevant is that the route to the
> router refers to the loopback device,

Looking up your own address, even one attached to eth0 or another
physical device, and seeing 'dev lo' is normal, and fine.

Is that what you mean?


You refer to the device as a router? If it's sold as a router that has
a linux kernel, then there could be any amount of special weirdness
introduced by the vendor. Is it vanilla linux or some product?

[Victor Preston]

On Tuesday, June 12, 2018 at 1:30:23 PM UTC-4, Victor Preston wrote:

I really have not made any progress with this. I have a machine that does not respond to TCP requests from my router. Using WIreshark I can see that what's going on is that the router sends a SYN to begin a TCP transaction and my Linux machine (running Ubuntu 16.04) fails to return a SYN,ACK, so the request fails. There's only the one router, and one hardware interface on the linux machine. If I run "arp" on the linux machine it shows the correct MAC for the router. Also if I run "arping" pointing to the router's IP it returns with the correct MAC. Furthermore when I look at the packets intercepted by wireshark the ll header has the correct MAC address for the ip. It looks like a "martian packet" log IS created, but, again, it has the right MAC/IP in it. So what's going on? Why does my linux machine think this is a martian packet when all the values are correct?

> * routing not set up (check with `ip route get 1.2.3.4`) 
>                      ip route <router's ip>: 
>                              local <router's ip> dev lo src <router's ip> 
> cache(local) 
>                      ip route <ip of other machine on theNAT that WILL 
> connect>: 
>                              <ip of other machine on theNAT that WILL 
> connect>  dev eno1(that's the iface on the linux machine in question)  src 
> <ip of the linux machine in question>   cache 

so "router" is the one seeing the SYN but isn't sending back SYNACK? 

   No, it's the other way around, the router sends the ACK, but the server never responds with SYN/ACK

> * `rp_filter` rejects the packet source interface[1] 
>                      I tried running with all rp_filter values set to 0, 
> but it did not help 

Confirmed with sysctl that they were actually set? 

   Yes, it was actually set, so I tried turning it off. When that did not help I put it back.

> * When you run wireshark, run with promisc mode off to ensure you 
> still see the SYN. You should if the MAC is correct. 
>                     I would think that promiscuous mode would show MORE 
> than disabling it, but I tried this and still do not see an ACK 

Yes the point would be to see fewer packets. This is to confirm that 
the MAC address is correct on the other machine's ARP table, since if 
you don't use promisc then the packet will be filtered before 
wireshark/tcpdump can see it (indeed before any part of the kernel sees it). 

That makes no sense to me, but it might be where the problem is. I went through this odd experience where the server could not access the web. It turned out that there's some kind of bug in the server's hardware (but NOT the interface card) that causes it to not respond to ARP requests, so the router could never learn it's correct MAC, so the router could not extablish a connection. I fixed that by manually entering the correct MAC address for this server on the router. Now the server can connect to the internet just fine. However, the problem I'm having is when remote machines try to establish a TCP connection with the server - i.e., it's going the other way, with a remote browser trying to connect to a webserver on my machine instead of a browser on the server connecting to Google (or whatever) .What happens is that through my ISP I have a dedicated, static IP, which is the router. The router then forwards packets to the server. So what the server sees is packets from the router.Although I don't see how, maybe there's some problem with the ARP table on the server. I've looked at the ARP table on the router for the server (it's labeled PERM, of course, since I apparently need to put it in manually) and on the server for the router and they are both correct. And it seems to me the only information available to sysctl is the MAC's and IP's in the packet itself, and those are correct too.

> * Any bridge interfaces? Try sniffing on the `any` pseudointerface. 
> Maybe the SYNACK is going elsewhere. 
>                     I don't know of any, and I tried running wireshark on 
> the 'any' interface, and looked carefully and it does not appear anywhere 

You shouldn't have to look too carefully, just do: 
sudo tcpdump -nlpi any host 1.2.3.4 
(1.2.3.4 being the remote end) 

When I do this I don't see any bridges, but I DO see a very weird thing:

In amongst other calls ar two ARP requests. I'll describe the second one first because it does what I would expect., i.e., Who-Has <router IP> tell <server's ip>

                     followed by ARP reply saying <router's ip> is-at <router's correct MAC address>

The other one (which actually happens first) is an ARP request who-has <router's ip> tell (this happens to be a TIVO machine on the net)

                     followed by ARP reply saying  <router's ip> is at -- AND  HERE is the SERVER's MAC address! Even if the TIVO machines messages are being forwarded to the server I don't think this should ever happen.And yet the TIVO functions correctly!

[tho...@habets.se]

On Thu, 14 Jun 2018 13:25:43 -0700 (PDT), Victor Preston
<vbpr...@gmail.com> said:
> so "router" is the one seeing the SYN but isn't sending back SYNACK?

> No, it's the other way around, the router sends the ACK, but the server
> never responds with SYN/ACK

You mean SYN, not ACK, right?

But which machine is which, here? Which one are you running commands
on? Which one are you sniffing on?

> .What happens is that through my ISP I have a dedicated, static
> IP, which is the router. The router then forwards packets to the server. So
> what the server sees is packets from the router.Although I don't see how,
> maybe there's some problem with the ARP table on the server. I've looked at
> the ARP table on the router for the server (it's labeled PERM, of course,
> since I apparently need to put it in manually)

Something is wrong, and this is not a good fix. I'm not surprised to
hear that it merely moves the problem, not fixes it.

I'd suggest removing that entry and trying to make it works as it's
supposed to.

If you think there's something wrong with the server hardware, then
that's what you should be fixing. (though I don't know what you
described could be caused by hardware)

> The other one (which actually happens first) is an ARP request who-has
> <router's ip> tell (this happens to be a TIVO machine on the net)
> followed by ARP reply saying <router's ip> is at --
> AND HERE is the SERVER's MAC address! Even if the TIVO machines messages
> are being forwarded to the server I don't think this should ever happen.And
> yet the TIVO functions correctly!

You might have some rogue or malfunctioning proxy-arp going on. Unplug
everything not related to what you're troubleshooting.

There are too many variables here, and too many unknowns, to
troubleshoot remotely with no access to the devices, network map,
device list, or device history.

I've given you some ideas, but I don't think I can help you more, really.

[Victor Preston]

You may indeed be right.  (and yes, I DID mean 'SYN' and not "ACK") I resorted to manually adding an entry to the router's ARP table because I spent a few months trying to get my machine to respond to the router's ARP requests and couldn't do it. I gave up when I saw an article that said some Dell servers had a hardware problem that caused this. However I had also tried buying a replacement interface card and using that instead of the one the machine came with and it exhibited the same problem, But thinking about it now it seems to me that the ARP request (since it's primarily concerned with the MAC address, which is defined by the interface hardware) really ought to be implemented by the interface. So why didn't that work? It sounds like maybe I should try that again and take your advice about disconnecting everything else. It's not clear to me what might be a "rogue or malfunctioning proxy device" as I have nothing that I know of being a proxy. But then I DO have devices I know little about on the network (TIVO, Sonos...). In any case, thank you very much fo all your ideas!

[Victor Preston]

Hi Thomas,

I just wanted to express my appreciation of all your help. I DID just now solve the problem, partly due to you forcing me to go back and figure out the original cause of the problem. Naturally, once I figured it out it was very simple and easy to fix!

What happened was that sometime ago I installed OpenVPN in an effort to be able to access my server from coffee shops, etc., but I found out that my ISP is blocking the relevant ports so that was not possible. Therefore I uninstalled it. The install process added a "tun0" virtual interface to the server, which the uninstall process did not remove. When messages came in from the router's IP the route rules directed the return packets to the local machine on the "tun0" interface, so the router was never receiving the ARP replies back. I was able to make that problem go away by manually entering the correct IP/MAC entry in the router's ARP table. However, that also meant that messages inbound from the WAN, which the router redirected to the machine in question, and the rules perceived as from the router, so THEY were directed to the "tun0" interface as well, so they were failing. Once I cleaned that up everything worked perfectly!

Once again, thank you very much for pointing me in the right direction!

Victor Preston

VPS Software, Inc.

617-230-8350

--
You received this message because you are subscribed to the Google Groups "Synscan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to synscan+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tho...@habets.se]

On Wed, 20 Jun 2018 09:08:46 -0400, Victor Preston <vbpr...@gmail.com> said:
> Once again, thank you very much for pointing me in the right direction!

Great that it worked out, and thanks for following up.