How to implement client exit when on diverse wwan connections

[Brian Call]

Hi all,

Two functions (IsOnWwan/IsOnAndroidWap) in Simian's preflight.py attempt to determine whether a client is using a mobile data connection, but they are quite old. 

https://github.com/google/simian/blob/433c2ef1e9d89abc8aeb72b2707809dec55aa4fa/src/simian/mac/client/preflight.py#L421

The functions only check for a ppp0 connection or android bt/wifi tether. Now-ubiquitous iPhone tethering and mobile hotspots both aren't addressed in these checks (because it's hard to do properly). How do people solve this problem these days? I want to turn on Apple Software Updates for our many remote users, but I have to make sure we don't obliterate their mobile data plans when we promote Apple updates.

If there is not an existing solution out there, I've put some thought into augmenting the current functionality by adding functions that would allow admins to define a list of SSIDs and/or a list of ASNs that will induce a client exit for simian auto-runs.

Just curious how google or other orgs using simian actually deal with this scenario these days. Thanks.

Brian

[Justin McWilliams]

We have iOS WAP and MiFi mobile hotspot detection logic implemented in our internal instance of Simian, which actually work quite reliably.  We have SSID detection as well, using nmcli, but it's not wired up to a user-friendly way to block particular names.  That shouldn't be too difficult to add to settings.cfg.  I'll try to slot it in for submitting to GitHub later this week.

The only other coverage we have is server-side client exit requests based on a predefined list of netblocks known to be associated with similarly undesirable networks for updates (i.e. GoGo InFlight WiFi, our shuttle buses, etc.) which you can manage at "/admin/ip_blacklist": https://github.com/google/simian/blob/master/src/simian/mac/munki/handlers/reports.py#L66

BTW, I quickly started a doc on this all: https://github.com/google/simian/wiki/Network-Detection-Backoff

- Justin

--
You received this message because you are subscribed to the Google Groups "Simian Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simian-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Randolph]

We tried to do this detection via an automatic/learning method when we
first developed Simian.
We couldn't find or discover any algorithms that were less disruptive
than just pulling the updates anyway, especially since users may be
bandwidth and/or total bytes transported constrained.

The summary of seemingly all lightweight methods is:
1)do lightweight discovery process
2)match discovered item in a list that confirms/denies constrained link

So this includes IP blocks/ASNs, SSIDs, get MAC addresses of default gw?, etc.

On Sat, Oct 10, 2015 at 1:54 PM, Brian Call <ca...@puppetlabs.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups
> "Simian Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to simian-discus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
John Randolph -- Google NYC -- Privacy Infrastructure Engineering

[Brian Call]

John & Justin,

Thanks for the explanation of the thought that's been put into this so far. And thanks for adding that info to the docs, Justin! Providing a space for us to collect useful blacklist IP ranges is a great idea. Looking forward to seeing the code for the 'coming soon' items.

Thanks again.

Brian

[Justin McWilliams]

Here's the module that detects tethering, complete with iOS and MiFi:

  https://github.com/google/simian/commit/d931432103b934e71a51d795fe06f6e3e42e07a1

Here's where it's plugged into preflight, replacing old methods (only submitted separately due to Git mistake):

  https://github.com/google/simian/commit/716b02d61d7ad5417b3c54d49926d48fcfea7ff0

Please give it a whirl and let me know if everything works as expected!  If so, we can plumb network_detect.GetNetworkName() up and expose to Simian admins as well.

- Justin

--

[Brian Call]

Hey Justin,

Thanks for making this available so quickly! 'Make test' revealed that requests is missing as a dependency for network_detect.py. I was able to get around this for testing by installing requests in the simian virtual env following installation on the client. I'd submit a PR to fix, but I haven't quite wrapped my head around the Simian build process yet :P I'll report back on functionality after some testing on a few of our wireless devices.

Speaking of PRs, any thoughts on thoughts around allowing users to specify their own Facter command in settings.cfg?

https://github.com/google/simian/pull/25

I understand that it's not really a 'user setting' like most of the other items in settings.cfg, so maybe it doesn't really belong there. If you feel like settings.cfg is not the right place for this knob, I'd be happy to instead contribute some documentation to help users edit simianfacter to properly execute different versions/configurations of Facter. 

Thanks!

-Brian

[Justin McWilliams]

Brian,

Apologies I don't have a Mac nearby to test this, so I overlooked the "requests" dependency; I developed and released from my Linux machine in effort to get it into your hands quickly.  I **think** this will fix you up, but please do confirm: https://github.com/google/simian/commit/8e024ee977df509d84c8d2736694a23fe62b0420

Also apologies that I totally overlooked the pull request you previously sent for facter; I'll try to review that soon. 

- Justin

--

[Brian Call]

Hi Justin,

Building simian works fine now with requests added to setup.py. Thank you. 

This new functionality is really promising, but it doesn't work out of the box for iPhone WAP detection, at least not for my iPhone 6 – that's all I've had a chance to test with so far. It seems likely the problem is that we can't rely on iPhones having consistent MAC addresses. IsOnIosWap checks that the ARP lookup of the default gateway (iPhone) MAC starts with ac:de:48. Unfortunately my iPhone's MAC doesn't contain anything close to this string. I'm no expert in mac address alloction/use by manufacturers, but I can't find any resource online that indicates ac:de:48 is an iPhone-specific string either. I'll get some more MAC data from my iPhone-toting colleagues tomorrow to see if this bears out, but it seems likely it will.

I changed this function to return False if the default gateway != 172.20.10.1, removed the MAC address check, and everything worked as expected:

def IsOnIosWap():

  """Checks if the wireless connection is to an iOS WAP tether.

  Returns:

    Boolean. True if iOS WAP is connected, False otherwise.

  """

  # iOS WAP looks like a 172.20.10/28 network with a default gateway of 172.20.10.1

  gateway = GetNetworkGateway('172.20.10/28')

  if not gateway:

    return False

  ip = GetDefaultGateway()

  if ip != '172.20.10.1':

    return False

  else:

    return True

I think for my use case this is specific enough. Seems unlikely that a user would find themselves on a non-iPhone network out in the wild that would return True for this check. But it's not impossible.

While using an external service isn't ideal, It would be relatively easy to write a supplemental function that uses an ASN lookup service to help verify that a client should exit. (e.g. http://www.team-cymru.org/IP-ASN-mapping.html#dns OR https://www.shadowserver.org/wiki/pmwiki.php/Services/IP-BGP)

Clearly a feature like this shouldn't be on by default for everyone but.... it's an idea. John mentioned that you were not able to successfully implement a more intelligent way of blocking these types of clients, so I'd imagine you considered this option or something similar but decided against it. I'd be happy for you to tell me why this is a stupid idea if you've already explored it :)

Anyway, thanks again. I'll keep playing with this and reporting back.

[Justin McWilliams]

Brian, 

You're correctly identified the challenge here.  We've built this detection based on characteristics of a sample of devices, but the target moves over time.  Unfortunately our MDM solution doesn't report MAC addresses, so it's not trivial for me to see if all iPhone 6 devices share a common OUI.  While the OUI may not be restricted to iPhones, it's simply all the more unlikely that a device is connected to that specific gateway which also has that specific OUI.  I'll try to get my hands on some iPhone 6's (and maybe some 5's as well, since we built this quite a while back), to see if adding a new OUI will help.

As far as APN detection, yes it's worth exploring.  In our environment, we leverage crankd to execute all network_detect checks whenever a network interface change is detected.  In addition, we also call to an App Engine hosted service which detects if the client is connected to a blacklisted netblock (similar to what is built into Simian), a corporate netblock, a corporate VPN, etc.  All of this detection is all cached on disk for Simian and other management tools to leverage (so they're not each firing off all checks on every execution).  Theoretically, this solution could be extended to detect APNs, and we could blacklist particular ones.  Also theoretically, assuming such detection logic was implemented, plugging it into the existing blacklisting functionality in Simian would likely be very minimal effort (i.e. IsExitFeedbackIpAddress could simply call a new IsIpAddressInBlacklistedApn method).

John was primarily hinting that *automatically* detecting if a connection is slow based on dynamic heuristics is extremely nontrivial, and thus assessments are unreliable, and such a method opens the potential to unnecessarily and indefinitely block clients from functioning as intended.  For example, it's very difficult to reliably understand the health of a network by merely downloading a small payload in order to measure speed/latency/etc. (and downloading a larger payload is counter to the point of such detection anyways).  On the other hand, while applying static blacklists of IPs/APNs/etc. comes with maintenance burden and some gaps in coverage over time, it's also far more predictable, and balances on the side of security (patching).

We're happy to help guide, if you want to dig deeper!

- Justin

[Gregory Neagle]

Maybe a better approach is to allow users to “opt-in” when not on the “corporate” network. IOW, assume that non-Corp is bandwidth constrained, and have the user specifically green-light getting updates over the current network. You’d then somehow remember the network for the future (which yes, gets hard to do since not all 192.168.x.x networks are the same…) But if you remembered wireless SSIDs, that _might_ be doable.

I am just throwing things at the wall and am probably underestimating the difficulty here.

-Greg

[Justin McWilliams]

That may work for some enterprises.  Many probably have devices which stay primarily on premises.  In our environment, we see many devices that never see the corp network, via VPN ever, after initial deployment, but we still want to manage them.

That said, beyond the UI/UX challenge of opt-in, this wouldn't be too difficult to plug into Simian's preflight.

// mobile

[Gregory Neagle]

So it sounds like you have three choices:

1) Run roughshod and just use the available bandwidth.

2) Make unreliable guesses about network suitability.

3) Put the responsibility in your user’s hands and trust your users to make good choices.

-Greg

[Justin McWilliams]

On Tue, Oct 13, 2015 at 1:31 PM, Gregory Neagle <gregn...@mac.com> wrote:

So it sounds like you have three choices:

Or, ideally, a blend of several options.  Given the clients only *request* to abort and the server must approve, we can always have the server veto requests.

We do this for some limited cases, and the client only exits if the server confirms it's OK: 

- https://github.com/google/simian/blob/master/src/simian/mac/munki/handlers/reports.py#L116

- https://github.com/google/simian/blob/master/src/simian/mac/client/preflight.py#L395

We'd probably want to beef up this logic to be a bit more aggressive, if going a full opt-in route.  Ideally grace periods/etc. would be easily configurable for admins.

[Justin McWilliams]

On Tue, Oct 13, 2015 at 12:58 AM, Brian Call <ca...@puppetlabs.com> wrote:

Hi Justin,

Building simian works fine now with requests added to setup.py. Thank you. 

This new functionality is really promising, but it doesn't work out of the box for iPhone WAP detection, at least not for my iPhone 6 – that's all I've had a chance to test with so far. It seems likely the problem is that we can't rely on iPhones having consistent MAC addresses. IsOnIosWap checks that the ARP lookup of the default gateway (iPhone) MAC starts with ac:de:48. Unfortunately my iPhone's MAC doesn't contain anything close to this string. I'm no expert in mac address alloction/use by manufacturers, but I can't find any resource online that indicates ac:de:48 is an iPhone-specific string either. I'll get some more MAC data from my iPhone-toting colleagues tomorrow to see if this bears out, but it seems likely it will.

I changed this function to return False if the default gateway != 172.20.10.1, removed the MAC address check, and everything worked as expected:

I collected a sample of about 20 iPhone MACs, ranging from iPhone5(s) to iPhone6(s)... and every single one had a unique OUI!   Then I ran into: http://appleinsider.com/articles/14/06/09/mac-address-randomization-joins-apples-heap-of-ios-8-privacy-improvements

So we might be stuck looking exclusively at the gateway, as you've adjusted, unless we can look for another characteristic (i.e. Android uses an uncommon netmask).  In any case, feel free to send a pull request for that change if you want, because what we have doesn't help for iOS 8 or newer.

[Brian Call]

I agree with Greg that an algorithm will never be perfect when applied to all possible connections in the real world, and that a user switch may be useful. However, I think we can improve these current checks enough to be acceptable, depending one's environment. We provide a couple different models of MiFis to employees who need to travel and we set them up with predictable SSIDs. When on the road, employees should connect through our company-owned devices, but the additional android/ios checks provide further safeguards should they need to hop on a personal mobile data connection. If a client wwan check provides a false negative and dings someone's mobile data plan on rare occasion, we feel like that is an acceptable incidental cost. Perfect? No. But close enough for our needs at this point and far better than nothing.

Now on to iOS–

I used nmap to run a port scan on my iPhone today while connected via Personal Hotspot and noticed that it has a unique port open: 62078, labeled 'iphone-sync'. I did the same with a few of my colleagues' phones and that port was open on all of them. A little internet digging revealed this post explaining that port 62078 is used for the undocumented lockdownd daemon on iOS: http://apple.stackexchange.com/questions/139447/how-to-interface-with-ios-lockdownd

Info around this service and port is very piecemeal online, but signs point to this being a reliable check for iPhone tethering when added to the existing logic. Before moving forward with this I want to double-check that this port is open on a freshly-restored phone, which I'll try to test tomorrow.

I was also suspicious of iOS 8's MAC randomization which you linked to, but I couldn't find anything that indicated it was used aside from in WiFi scanning mode, so I'm not sure it's actually in play with what we're looking at. Either way, we can be confident there isn't a reliable way to check a MAC address and determine whether it's an iPhone in 2015 :)

Also thanks for the PR feedback, I'll adjust and resubmit tomorrow. 

-Brian

[Justin McWilliams]

Nice find with the port (62078).  With the default gateway and that, we'll likely avoid the vast majority of false positives (and hopefully false negatives, though it seems prudent to repeatedly test this when new iOS updates are released).

-Brian

--

You received this message because you are subscribed to the Google Groups "Simian Discuss" group.

[John Randolph]

are you really seeing totally random MAC addresses for an iPhone
acting as a tether nat/gw, or are the leading bytes always AC:DE:48?
that OUI is used over and over in RFCs to demonstrate MAC address byte
formatting and is marked as 'private', but it isn't clear what that
means exactly.

On Tue, Oct 13, 2015 at 12:58 AM, Brian Call <ca...@puppetlabs.com> wrote:

[Justin McWilliams]

I asked 19 colleagues and friends to share their MAC, and got 18 different responses for the leading bytes*.  It's worth noting they merely shared the MAC reported in the settings page, and they did *not* tether and use ARP.

Checking the MAC of the default gateway while tethering would be prudent, but Brian indicated the code (as is) didn't function as intended (which hints that the default gateway MAC did not match AC:DE:48).

* Examples, all of which OUI lookup maps to Apple Inc.:

    '18:f6:43', '4c:7c:5f', '70:48:0f', 'b4:18:d1', 'd4:f4:6f', 'dc:2b:2a', 'e0:b5:2d'

[Brian Call]

I tested this on four iPhones yesterday. Not the biggest sample size, but I tethered to each one and determined the MAC of the iPhone gateway via ARP. No consistency observed between MACs. 

Also, maybe we actually are seeing iOS8+ MAC randomization here, because I don't see the same MAC that ARP reports (6e:7c:5f) in Settings on the iPhone (4c:7c:5f)

[Justin McWilliams]

FYI --- I've added support for blacklisting particular WiFi SSIDs, and bootstrapped it with a short list of ones I'm familiar with; let the crowdsourcing begin!  If there's contention, we can turn this into a configuration option.

https://github.com/google/simian/commit/5541a63901144fbc8582a3a8e87fd5ffe4bc611b

On Wed, Oct 14, 2015 at 2:53 PM, Brian Call <ca...@puppetlabs.com> wrote:

I tested this on four iPhones yesterday. Not the biggest sample size, but I tethered to each one and determined the MAC of the iPhone gateway via ARP. No consistency observed between MACs. 

Also, maybe we actually are seeing iOS8+ MAC randomization here, because I don't see the same MAC that ARP reports (6e:7c:5f) in Settings on the iPhone (4c:7c:5f)

--