OS X Lion Issues

[David Cramer]

Couldn't find anywhere with any information on this, and I'm not too
familiar w/ how sshuttle works, but here's the error that I've hit
after the Lion upgrade:


Connected.
sysctl: oid 'net.inet.ip.scopedroute' is read only

WARNING: ONE-TIME NETWORK DISRUPTION:
=====================================
sshuttle has changed a MacOS kernel setting to work around
a bug in MacOS 10.6. This will cause your network to drop
within 5-10 minutes unless you restart your network
interface (change wireless networks or unplug/plug the
ethernet port) NOW, then restart sshuttle. The fix is
permanent; you only have to do this once.

sysctl: oid 'net.inet.ip.scopedroute' is read only
client: fatal: cleanup: ['python', '/Users/dcramer/Development/
sshuttle/main.py', 'python', '--firewall', '12300', '0'] returned 1

[David Cramer]

[za...@foursquare.com]

Yeah, Apple has changed the read/write permissions of /etc/sysctl.conf so that sshuttle has trouble writing to it. Unfortunately, there appears to be much more at work here since even when I make the file writeable, run sshuttle with sudo, or comment out the lines that cause sshuttle to write to the file it still fails to create the tunnel correctly. I'm hoping there's a fix in the works for this :-\

[Clint Ecker]

Yeah the two of us here @ Ars tried the same thing and didn't much of
anywhere. It claims to be connected, but then nothing really works :(

[Ian Atha]

This change is caused by the following difference between
xnu-1504.9.37 (the kernel shipped with the last Snow Leopard build)
and xnu-1699.22.73 (the kernel shipped with Mac OS X Lion):

< SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RW,
> SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RD | CTLFLAG_LOCKED,
> PE_parse_boot_argn("net.inet.ip.scopedroute",
> &ip_doscopedroute, sizeof (ip_doscopedroute));

(See this for yourself via """diff -B <(curl "http://
www.opensource.apple.com/source/xnu/xnu-1504.9.37/bsd/netinet/ip_input.c?txt")
<(curl "http://www.opensource.apple.com/source/xnu/xnu-1699.22.73/bsd/
netinet/ip_input.c?txt") | grep -i scopedroute""").

In the flag declaration for "net.inet.ip.scopedroute", CTLFLAG_RW has
been with CTLFLAG_RD (which means read-only).

The usage of PE_parse_boot_argn indicates that Lion only sets
"net.inet.ip.scopedroute" during boot, and doesn't allow sysctl (the
userspace utility) to modify that flag. (See """http://
www.opensource.apple.com/source/xnu/xnu-1699.22.73/pexpert/gen/bootargs.c""".)

By editing """/Library/Preferences/SystemConfiguration/
com.apple.Boot.plist""" and making it look like
<dict>
<key>Kernel Flags</key>
<string>net.inet.ip.scopedroute=0</string>
</dict>

you can set scopedroute at boot time.

Sshuttle still doesn't work, even if you comment out the comment that
attempts to set net.inet.ip.scopedroute via userspace manipulation
(via the sysctl binary and /etc/sysctl.conf). Specifically, it
connects, reports "Connected" but attempting to route packages to one
of the to-be-tunneled subnets causes "request timeouts".

-- Ian Atha.

[Ian Atha]

In case your reader mangled up my email: http://pastebin.com/NzAARKVG

On Jul 21, 5:18 pm, Ian Atha <thatha7...@gmail.com> wrote:
> This change is caused by the following difference between
> xnu-1504.9.37 (the kernel shipped with the last Snow Leopard build)
> and xnu-1699.22.73 (the kernel shipped with Mac OS X Lion):
>
> < SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RW,
>
> > SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RD | CTLFLAG_LOCKED,
> >            PE_parse_boot_argn("net.inet.ip.scopedroute",
> >                &ip_doscopedroute, sizeof (ip_doscopedroute));
>

> (See this for yourself via """diff -B <(curl "http://www.opensource.apple.com/source/xnu/xnu-1504.9.37/bsd/netinet/ip_inpu...")

> <(curl "http://www.opensource.apple.com/source/xnu/xnu-1699.22.73/bsd/
> netinet/ip_input.c?txt") | grep -i scopedroute""").
>
> In the flag declaration for "net.inet.ip.scopedroute", CTLFLAG_RW has
> been with CTLFLAG_RD (which means read-only).
>
> The usage of PE_parse_boot_argn indicates that Lion only sets
> "net.inet.ip.scopedroute" during boot, and doesn't allow sysctl (the

> userspace utility) to modify that flag.  (See """http://www.opensource.apple.com/source/xnu/xnu-1699.22.73/pexpert/gen/bootar...""".)

[Ian Atha]

Looks like sshuttle doesn't work on my particular configuration due to
other network-level hacks. For SShuttle to work on Mac OS X Lion:

1) Run

sudo defaults write "/Library/Preferences/SystemConfiguration/
com.apple.Boot" "Kernel Flags" "" # net.inet.ip.scopedroute=0
`defaults read "/Library/Preferences/SystemConfiguration/
com.apple.Boot" "Kernel Flags"`"

2) Restart.

3) Comment out line 236 in firewall.py (or run "git clone -b
macos_10_7_only_hack git://github.com/thatha/sshuttle.git")

-ian atha.

On Jul 21, 5:22 pm, Ian Atha <thatha7...@gmail.com> wrote:
> In case your reader mangled up my email:http://pastebin.com/NzAARKVG
>
> On Jul 21, 5:18 pm, Ian Atha <thatha7...@gmail.com> wrote:
>
>
>
>
>
>
>
> > This change is caused by the following difference between
> > xnu-1504.9.37 (the kernel shipped with the last Snow Leopard build)
> > and xnu-1699.22.73 (the kernel shipped with Mac OS X Lion):
>
> > < SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RW,
>
> > > SYSCTL_INT(_net_inet_ip, OID_AUTO, scopedroute, CTLFLAG_RD | CTLFLAG_LOCKED,
> > >            PE_parse_boot_argn("net.inet.ip.scopedroute",
> > >                &ip_doscopedroute, sizeof (ip_doscopedroute));
>

> > (See this for yourself via """diff -B <(curl "http://www.opensource.apple.com/source/xnu/xnu-1504.9.37/bsd/netinet/......")

> > <(curl "http://www.opensource.apple.com/source/xnu/xnu-1699.22.73/bsd/
> > netinet/ip_input.c?txt") | grep -i scopedroute""").
>
> > In the flag declaration for "net.inet.ip.scopedroute", CTLFLAG_RW has
> > been with CTLFLAG_RD (which means read-only).
>
> > The usage of PE_parse_boot_argn indicates that Lion only sets
> > "net.inet.ip.scopedroute" during boot, and doesn't allow sysctl (the

> > userspace utility) to modify that flag.  (See """http://www.opensource.apple.com/source/xnu/xnu-1699.22.73/pexpert/gen......""".)

[Ian Atha]

The correct command is:

sudo defaults write "/Library/Preferences/SystemConfiguration/
com.apple.Boot" "Kernel Flags" "net.inet.ip.scopedroute=0 `defaults
read "/Library/Preferences/SystemConfiguration/
com.apple.Boot" "Kernel Flags"`" (http://pastebin.com/jWHJEJJ5)

[George Guimarães]

Thanks a lot for the workaround, Ian.

It worked here

[Avery Pennarun]

On Thu, Jul 21, 2011 at 11:15 PM, Ian Atha <thath...@gmail.com> wrote:
> Looks like sshuttle doesn't work on my particular configuration due to
> other network-level hacks. For SShuttle to work on Mac OS X Lion:
>
> 1) Run
>
> sudo defaults write "/Library/Preferences/SystemConfiguration/
> com.apple.Boot" "Kernel Flags" "" # net.inet.ip.scopedroute=0
> `defaults read "/Library/Preferences/SystemConfiguration/
> com.apple.Boot" "Kernel Flags"`"
>
> 2) Restart.

Nice! Now, probably we're not supposed to *actually* need to use that
command. A little strange that they made it unchangeable like that...
maybe it's because of that bug I filed where if you change it, your
network stops working after 10 minutes thanks to a helpful ARP bug.
So rather than fix the ARP bug, they just made it unchangeable?
That's some quality engineering there. Sigh.

I wonder if there's some way to make the ipfw fwd stuff work *without*
disabling the scopedroute feature.

I suppose we now have to add yet another workaround to sshuttle to
make it do the above commands *instead* of the sysctl commands, if it
turns out the sysctl is non-writable. Patches to do something like
that are welcome. Patches to fix things so we don't even need that
are even more welcome :)

Have fun,

Avery

[Clint Ecker]

On Jul 22, 11:28 am, Avery Pennarun <apenw...@gmail.com> wrote:

> On Thu, Jul 21, 2011 at 11:15 PM, Ian Atha <thatha7...@gmail.com> wrote:
> > Looks like sshuttle doesn't work on my particular configuration due to
> > other network-level hacks. For SShuttle to work on Mac OS X Lion:
>
> > 1) Run
>
> > sudo defaults write "/Library/Preferences/SystemConfiguration/
> > com.apple.Boot" "Kernel Flags" "" # net.inet.ip.scopedroute=0
> > `defaults read "/Library/Preferences/SystemConfiguration/
> > com.apple.Boot" "Kernel Flags"`"
>
> > 2) Restart.

Has anyone else run this and had it work? I ran the following command:

https://gist.github.com/7acd5c9d563f5bc74b6f

Which resulted in my com.apple.Boot.plist looking like the following:

http://i.clintecker.com.s3.amazonaws.com/Screen%20Shot%202011-07-28%20at%203.01.09%20PM.png

When I reboot, I check the value of that flag and I see:

https://gist.github.com/36677340493ed8256cfc

Anything I'm missing here?

[Ian Atha]

What does your "uname -a" look like?

[Clint Ecker]

On Jul 28, 3:44 pm, Ian Atha <i...@atha.io> wrote:
> What does your "uname -a" look like?

"Darwin Brutus.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18
12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64"

[Christopher]

Same as Clint, this doesn’t work for me on my new MacBook Air, even
with the “Kernel Flags” Boot.plist entry in place. Weirrrrd. Y’all
want some -vv output from sshuttle or anything like that?

—cbowns

[Avery Pennarun]

On Fri, Jul 29, 2011 at 5:30 PM, Christopher <cbo...@gmail.com> wrote:
> Same as Clint, this doesn’t work for me on my new MacBook Air, even
> with the “Kernel Flags” Boot.plist entry in place. Weirrrrd. Y’all
> want some -vv output from sshuttle or anything like that?

I doubt we'll learn anything more exciting from that than "it doesn't
work." It really is caused by a kernel bug; too bad Apple doesn't
seem to care about kernel bugs. (Perhaps rationally so, if almost
nobody uses 'ipfw fwd' rules, but it's weird that they don't work *at
all* without this ugly sysctl hack.)

I guess I might have to give in and buy a copy of Lion this weekend.
Blah. Normally I wait a year before installing new versions of OSes
:) But luckily I have a separate work Macbook now so I'll be able to
continue testing 10.6 even if I install 10.7 at home.

Have fun,

Avery

[Clint Ecker]

On Jul 29, 6:14 pm, Avery Pennarun <apenw...@gmail.com> wrote:
> I guess I might have to give in and buy a copy of Lion this weekend.
> Blah.  Normally I wait a year before installing new versions of OSes
> :)  But luckily I have a separate work Macbook now so I'll be able to
> continue testing 10.6 even if I install 10.7 at home.

FWIW You are technically allowed to virtualize OS X now with Lion.
I'm not sure if any of the desktop vm vendors have added support for
it yet though.

clint

[Luis]

Hello guys, I have the same uname -a
"Luis-Gonzalezs-MacBook-Pro:~ lgonzalezsa$ uname -a
Darwin Luis-Gonzalezs-MacBook-Pro.local 11.0.0 Darwin Kernel Version

11.0.0: Sat Jun 18 12:56:35 PDT 2011; root:xnu-1699.22.73~1/
RELEASE_X86_64 x86_64"

After reboot I have this:

"Luis-Gonzalezs-MacBook-Pro:~ lgonzalezsa$ sysctl -a | grep -i
net.inet.ip.scopedroute
net.inet.ip.scopedroute: 1"

I'm running git repository using:

"git clone -b macos_10_7_only_hack git://github.com/thatha/sshuttle.git"

after run sshuttle with the previous condition, I do not make it work

I miss you sshuttle!!!

[Ian Atha]

This is very odd, guys...

bash-3.2# uname -a
Darwin haecceity.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18

12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64

bash-3.2# defaults read
"//Library/Preferences/SystemConfiguration/com.apple.Boot.plist"
{
"Kernel Flags" = "net.inet.ip.scopedroute=0 ";
}

bash-3.2# sysctl -a | grep scopedroute
kern.bootargs: net.inet.ip.scopedroute=0
net.inet.ip.scopedroute: 0
net.inet6.ip6.scopedroute: 1

Luis-- what's your "kern.bootargs"?

-ian.

[Ian Atha]

Hm, it looks like the "defaults" invocation I gave out, has the tendency to convert your property files from XML to the opaque binary property list format, that's not necessarily understood by the kernel.

I've updated the README at https://github.com/thatha/sshuttle/blob/macos_10_7_only_hack/README.md with the correct commands.

As root, try running:

rm /Library/Preferences/SystemConfiguration/com.apple.Boot.plist.lockfile

echo '<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>Kernel Flags</key><string>net.inet.ip.scopedroute=0</string></dict></plist>' > /Library/Preferences/SystemConfiguration/com.apple.Boot.plist  

Restart, and confirm that kern.bootargs is set correctly, along with scopedroute for IPv4:

haecceity:sshuttle thatha$ sysctl -a | grep bootargs

kern.bootargs: net.inet.ip.scopedroute=0

haecceity:sshuttle thatha$ sysctl -a | grep net.inet.ip.scopedroute

kern.bootargs: net.inet.ip.scopedroute=0

net.inet.ip.scopedroute: 0

That should re-enable the workaround on Mac OS X Lion, and sshutle should operate normally.

Thanks to hitch17 for helping me debug this.

[Clint Ecker]

On Aug 1, 7:24 pm, Ian Atha <i...@atha.io> wrote:
> That should re-enable the workaround on Mac OS X Lion, and sshutle should
> operate normally.
>
> Thanks to hitch17 for helping me debug this.

Works great! You are a genius and a gentleman :)

[Christopher]

Alternately, keep the defaults command as before (I liked it because
it preserved existing flags), and use `plutil` to convert the plist:

plutil -convert xml1 <filepath>

i.e.

sudo plutil -convert xml1 /Library/Preferences/SystemConfiguration/
com.apple.Boot.plist

—cbowns

[Luis]

Thank you so much!!!
It works perfectly now!

I have my sshuttle working again!

Luis