Still seeing a dead network issue on 10.7

[Neil Lee]

Hi all -

I just discovered sshuttle and have been experimenting with it on Mac OS X 10.7.2. Everything works fine except once I disconnect sshuttle I can't reach anything on the internal network or internet. I've tried toggling wifi as the github page recommends and added the scopedroute setting to com.apple.Boot.plist as well, and rebooted.

Still, I lose all web access. Any suggestions on what I could try to do to troubleshoot this?

[Gert Van Gool]

Since my OS X knowledge is limited, I don't know the commands.
But what would be helpful is a dump of your firewall config after
sshuttle stopped. It feels it wasn't cleaned up properly.

-- Gert

Mobile: +32 498725202
Twitter: @gvangool
Web: http://gertvangool.be

[Aaron Bull Schaefer]

I'm seeing this on 10.7.2 as well, but only when using sshuttle via
the GUI interface (CLI works fine)...although it doesn't seem to drop
connectivity completely, just DNS lookups start to fail and restarting
the machine has been the only way I've found to fix it. Everything
looks as expected in the network preferences and toggling the
connection and/or renewing my DHCP lease makes no difference.

It doesn't matter whether I send DNS requests though a server or not,
as soon as I disconnect using the GUI, DNS lookups will fail. It just
seems like something in the cleanup process fails. I haven't seen
anything too strange in the system.log, but will continue to
investigate to see if I can provide more details. I just wanted to
confirm that I am also seeing some weirdness on Lion.

--
Aaron Bull Schaefer

[Neil Lee]

I should add that like Aaron I'm reporting this via the GUI - using the command line version seems to be okay. You were right about the firewall - flushing the rules brings back connectivity.

Here's what I got from "ipfw list" before I flushed the rules:

12300 check-state

12300 skipto 12301 log tcp from any to 127.0.0.0/8

12300 fwd 127.0.0.1,12300 log tcp from any to any not ipttl 42 setup keep-state

65535 allow ip from any to any

Hope this is of use!

[Neil Lee]

FYI - to get access back without having to reboot, run this in the terminal:

sudo ipfw -f flush

[Avery Pennarun]

Hi, the diagnosis here is obviously correct - those firewall rules
should get cleared when sshuttle exits and they obviously aren't being
cleared - but I don't understand why. They absolutely do get cleared
for me (although I haven't tried on Lion in particular).

How are you killing the connection in the GUI? Can you try out the
latest github.com/apenwarr/master branch, as I just made some slight
changes to it yesterday?

Definitely want to get this fixed up.

Thanks,

Avery

[Jimmy Tang]

I just pulled the head and tried it, its still not flushing the routes when sshuttle exits. also on lion, the macos ui doesn't build as apple changed somethings with gcc (removal of ppc stuff and moved to llvm etc...) this somewhat fixes it, it's a pretty messy hack to get it to detect if its lion or not.

diff --git a/ui-macos/bits/runpython.do b/ui-macos/bits/runpython.do

index a53247f..50782d1 100644

--- a/ui-macos/bits/runpython.do

+++ b/ui-macos/bits/runpython.do

@@ -1,13 +1,18 @@

 exec >&2

 redo-ifchange runpython.c

 ARCHES=""

+BUILDPLATFORM=$(uname -r)

 printf "Platforms: "

-for d in /usr/libexec/gcc/darwin/*; do

-    PLAT=$(basename "$d")

-    [ "$PLAT" != "ppc64" ] || continue  # fails for some reason on my Mac

-    ARCHES="$ARCHES -arch $PLAT"

-    printf "$PLAT "

-done

+if [ "$BUILDPLATFORM" = "11.2.0" ]; then

+       ARCHES="-arch i686"

+else

+       for d in /usr/libexec/gcc/darwin/*; do

+           PLAT=$(basename "$d")

+           [ "$PLAT" != "ppc64" ] || continue  # fails for some reason on my Mac

+           ARCHES="$ARCHES -arch $PLAT"

+           printf "$PLAT "

+       done

+fi

[Jimmy Tang]

attached is an aggregated set of changes for the newer version of redo and the check for building on lion. I was just thinking maybe its worth just dropping the ppc support and just default to building i686 and x86_64 for ui-macos/bits/runpython.c wrapper program.

[Aaron Bull Schaefer]

On Jan 4, 11:33 am, Avery Pennarun <apenw...@gmail.com> wrote:
> How are you killing the connection in the GUI?  Can you try out the
> latest github.com/apenwarr/master branch, as I just made some slight
> changes to it yesterday?

For me, it doesn't matter how I kill the connection with the GUI,
either by clicking the chicken and then clicking 'Disconnect
<servername> (all traffic)', or by going into the Preferences and de-
selecting the checkbox next to the server name, or even by quitting
the app. I always see those same firewall rules still left in place;
and using the CLI always tears the firewall rules down as expected.

I am running the latest git, including the commits you pushed
yesterday and still see this behavior. The Log Messages tab gets
cleared on disconnect, so I see regular debug messages when it's
enabled, but nothing that might be of help about the disconnection
process. Let me know if there's something else you'd like for me to
run that might help.

--
Aaron Bull Schaefer

[Avery Pennarun]

On Wed, Jan 4, 2012 at 3:34 PM, Jimmy Tang <jcf...@gmail.com> wrote:>
I just pulled the head and tried it, its still not flushing the routes
when> sshuttle exits. also on lion, the macos ui doesn't build as
apple changed> somethings with gcc (removal of ppc stuff and moved to
llvm etc...) this> somewhat fixes it, it's a pretty messy hack to get
it to detect if its lion> or not.

I don't like the idea of detecting specific OS versions - it will be
an ongoing maintenance nightmare. Can you find a way (eg. with the
existence of a file or directory) to *actually* detect which build
architectures are available?

Thanks,

Avery

[Avery Pennarun]

On Wed, Jan 4, 2012 at 3:50 PM, Aaron Bull Schaefer
<aarons...@gmail.com> wrote:
> On Jan 4, 11:33 am, Avery Pennarun <apenw...@gmail.com> wrote:
>> How are you killing the connection in the GUI?  Can you try out the
>> latest github.com/apenwarr/master branch, as I just made some slight
>> changes to it yesterday?
>
> For me, it doesn't matter how I kill the connection with the GUI,
> either by clicking the chicken and then clicking 'Disconnect
> <servername> (all traffic)', or by going into the Preferences and de-
> selecting the checkbox next to the server name, or even by quitting
> the app. I always see those same firewall rules still left in place;
> and using the CLI always tears the firewall rules down as expected.

Can you actually 'kill -9' the sshuttle sub-process and tell me what
happens? Specifically, the child process of the GUI process, not the
actual GUI process itself, or any of the sub-sub processes.

Thanks,

Avery

[Aaron Bull Schaefer]

On Jan 4, 2:50 pm, Avery Pennarun <apenw...@gmail.com> wrote:
> Can you actually 'kill -9' the sshuttle sub-process and tell me what
> happens?  Specifically, the child process of the GUI process, not the
> actual GUI process itself, or any of the sub-sub processes.

If I `kill -9` the sub-process, the firewall rules remain and the GUI
reports 'State: ERROR - try reconnecting' with a red chicken. The GUI
still gives me the option to disconnect, which makes no difference in
flushing out the rules. These are the processes that were running:

UID PID PPID CMD
501 37568 213 /usr/local/libexec/sshuttle/ui-macos/Sshuttle
VPN.app/Contents/MacOS/Sshuttle -psn_0_8562730
501 37692 37568 python /usr/local/libexec/sshuttle/ui-macos/Sshuttle
VPN.app/Contents/Resources/sshuttle/main.py python -r linode 0/0
0 37695 37692 sudo -p [local sudo] Password: python /usr/local/
libexec/sshuttle/ui-macos/Sshuttle VPN.app/Contents/Resources/sshuttle/
main.py python --firewall 12300 0
0 37696 37695 python /usr/local/libexec/sshuttle/ui-macos/Sshuttle
VPN.app/Contents/Resources/sshuttle/main.py python --firewall 12300 0

$ kill -9 37692
$ sudo ipfw list
12300 check-state
12300 skipto 12301 tcp from any to 127.0.0.0/8
12300 fwd 127.0.0.1,12300 tcp from any to any not ipttl 42 setup keep-

state
65535 allow ip from any to any

$ sudo ipfw -f flush
Flushed all rules.
$ sudo ipfw list

65535 allow ip from any to any

--
Aaron Bull Schaefer

[Avery Pennarun]

On Wed, Jan 4, 2012 at 6:34 PM, Aaron Bull Schaefer
<aarons...@gmail.com> wrote:
> On Jan 4, 2:50 pm, Avery Pennarun <apenw...@gmail.com> wrote:
>> Can you actually 'kill -9' the sshuttle sub-process and tell me what
>> happens?  Specifically, the child process of the GUI process, not the
>> actual GUI process itself, or any of the sub-sub processes.
>
> If I `kill -9` the sub-process, the firewall rules remain and the GUI
> reports 'State: ERROR - try reconnecting' with a red chicken. The GUI
> still gives me the option to disconnect, which makes no difference in
> flushing out the rules. These are the processes that were running:
>
>  UID   PID  PPID CMD
>  501 37568   213 /usr/local/libexec/sshuttle/ui-macos/Sshuttle
> VPN.app/Contents/MacOS/Sshuttle -psn_0_8562730
>  501 37692 37568 python /usr/local/libexec/sshuttle/ui-macos/Sshuttle
> VPN.app/Contents/Resources/sshuttle/main.py python -r linode 0/0
>    0 37695 37692 sudo -p [local sudo] Password:  python /usr/local/
> libexec/sshuttle/ui-macos/Sshuttle VPN.app/Contents/Resources/sshuttle/
> main.py python --firewall 12300 0
>    0 37696 37695 python /usr/local/libexec/sshuttle/ui-macos/Sshuttle
> VPN.app/Contents/Resources/sshuttle/main.py python --firewall 12300 0
>
> $ kill -9 37692

This is where, on Linux, I would run 'strace -p 37696' in another
window while I ran 'kill -9 37692'. We need to know why 37696 is
dying without clearing out the firewall rules, even though you're not
killing it explicitly. It must be dying due to some error, but I
don't know what error that is.

Do you know how to use dtrace? It's supposedly 'better' than strace,
but it's never done anything useful for me. Supposedly it should be
able to tell you things like what syscall was running when it died,
and that's what I need to know. Weird that it doesn't happen for me.

Have fun,

Avery

[Jimmy Tang]

I would agree that detecting OS's is a bad idea, but anyway in Lion as far as I know has dropped ppc support, as a result recent xcode releases have also dropped xcode support.

what used to be in /usr/libexec/gcc/darwin/ is now in /usr/llvm-gcc-4.2/ with no hint of ppc related libraries or tools. lion will pretty much install the latest xcode (and update it) for the user from the app-store. the only sane solution I can think of is to do something like drop ppc and ppc64 support and only build a i386 and x86_64 binary or write a bunch of tests to check if gcc supports ppc, ppc64, i386 and x86_64. doing these types of tests is heading into autoconf hell :P 

[Avery Pennarun]

On Thu, Jan 5, 2012 at 5:41 AM, Jimmy Tang <jcf...@gmail.com> wrote:
> what used to be in /usr/libexec/gcc/darwin/ is now in /usr/llvm-gcc-4.2/
> with no hint of ppc related libraries or tools. lion will pretty much
> install the latest xcode (and update it) for the user from the app-store.
> the only sane solution I can think of is to do something like drop ppc and
> ppc64 support and only build a i386 and x86_64 binary or write a bunch of
> tests to check if gcc supports ppc, ppc64, i386 and x86_64. doing these
> types of tests is heading into autoconf hell :P

Sure, I understand that if you're building on Lion, you shouldn't
expect PPC support. But the build script in sshuttle already very
nearly auto-selects which platforms it needs to build for; it's just
doing it wrong, apparently. It should just need a tiny bit of
tweaking, right?

Avery

[Avery Pennarun]

Please try the following patch to see if it fixes the MacOS ipfw clearing
problems.

---
firewall.py | 7 ++++++-
1 files changed, 6 insertions(+), 1 deletions(-)
mode change 100644 => 100755 main.py

diff --git a/firewall.py b/firewall.py
index 4fd8c79..291b7fd 100644
--- a/firewall.py
+++ b/firewall.py
@@ -1,4 +1,4 @@
-import re, errno, socket, select, struct
+import re, errno, socket, select, signal, struct
import compat.ssubprocess as ssubprocess
import helpers, ssyslog
from helpers import *
@@ -398,6 +398,11 @@ def main(port, dnsport, syslog):
sys.stdout.write('READY\n')
sys.stdout.flush()

+ # don't disappear if our controlling terminal or stdout/stderr
+ # disappears; we still have to clean up.
+ signal.signal(signal.SIGHUP, signal.SIG_IGN)
+ signal.signal(signal.SIGPIPE, signal.SIG_IGN)
+
# ctrl-c shouldn't be passed along to me. When the main sshuttle dies,
# I'll die automatically.
os.setsid()
diff --git a/main.py b/main.py
old mode 100644
new mode 100755
--
1.7.3.1

[Aaron Bull Schaefer]

On Jan 5, 6:17 pm, Avery Pennarun <apenw...@gmail.com> wrote:
> Please try the following patch to see if it fixes the MacOS ipfw clearing
> problems.

YES! That patch fixes the problem for me...thank you for such a quick
resolution! I started to look at dtrace, but have never used it before
and couldn't figure out anything that would be helpful :-)

--
Aaron Bull Schaefer

[Avery Pennarun]

Me neither. Piece of junk, I'm afraid. I eventually ended up having
to go with a hunch :)

I'll push it to github tomorrow.

Have fun,

Avery

[Wayne Scott]

Nice deduction, Sherlock!

-Wayne

[Avery Pennarun]

I extracted out the parts of your patch that apply to redo, and
applied those. I have not applied the Lion patch until we can get one
that's less gross. :)

Have fun,

Avery