CBL on iOS does not sync with some HTTP proxy settings

[Ian Ragsdale]

Hello, all. I'm currently debugging a really fun problem where the CBL framework fails to sync for a client in a corporate environment. I spent some time debugging, and I've tracked the issue down to the use of the SCNetworkReachability framework to decide whether to attempt to sync.

The problem is that the SCNetworkReachability library doesn't take proxy settings into consideration, so it attempts to resolve the host and verify that there is a valid path directly, which fails. So, in this particular environment, CBL never attempts to sync, even though the proxy should allow the sync to work correctly.

So, I'm going to have to attempt to make this work somehow, and ideally whatever fix I make can be contributed back, so I'd like to make sure the approach I take is acceptable. So here's my proposal.

In the absence of any proxy settings, the SCNetworkReachability check seems like the right thing to do, it just cannot be the final word when there is a proxy available. So, I think the right thing to do is to modify [CBLReachablity reachability] to be proxy-aware.

My thinking is when the network state changes, the CBLReachability instance should ask the system if the given host would go through any proxies using CFNetworkCopyProxiesForURL. If there ARE proxies for that URL, it should make an immediate test request to the replication host with a relatively short timeout. It would treat any response other than a timeout as a positive reachability state, and a timeout as a negative reachability state. 

I'm not sure if the normal WiFi/Cellular flags can be trusted if SCNetworkReachablity says that the server is not accessible. If they cannot, I think that if the CBLReachabliity instance determined that the server was available via a proxy, it can assume that the phone is on WiFi and not cellular - I don't really see any other likely scenario where the phone would be behind a proxy like that.

Does this make sense? Does anybody see any obvious problems with this?

- Ian

[Andrew W. Donoho]

On May 15, 2014, at 21:49 , Ian Ragsdale <ian.ra...@gmail.com> wrote:

Does this make sense? Does anybody see any obvious problems with this?

Ian,

I have a somewhat intimate relationship with the Reachbility library. Are you sure it ever leaves the machine? In my experience, it solely depends upon local state of the radios, etc. Touching a host, the recommended way to start the radios, does leave the machine. But that isn't a Reachability function. The first thing that happens, of course, is name resolution. That is normally where things start to fall apart with my connectivity.

Hence, I'm not sure I'm buying your analysis of the data nor your identification of the culprit class.

Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew...@gmail.com, +1 (512) 666-7596, twitter.com/adonoho

Download Retweever here: <http://Retweever.com>

No risk, no art.
No art, no reward.
-- Seth Godin

[Ian Ragsdale]

Thanks, Andrew, I'd hope by now you could give me a bit more credit than that. :)

The logs from CBL are quite clear. The reachability check is failing, but the server is definitely reachable using Safari - I can hit URLs on it and see the expected response. CBL is very clearly not ever attempting a connection due to the reachability check returning a failure. So, I am quite positive that in this situation, a negative response from SCNetworkReachability is NOT a clear indication that a successful request cannot be made, because the proxy server will allow it to succeed.

I do not particularly care WHY it reports a failure, nor was I attempting to posit a particular reason. In this environment, the server actually cannot be resolved by DNS, but again, that is irrelevant because it can connect to the proxy server, and only the proxy server needs to resolve the server.

So, I'm quite positive that when a proxy server is involved, SCNetworkReachability does not accurately predict whether or not a successful request to the server can be made. So, lets assume for the moment that I'm not an idiot and that I've diagnosed the issue correctly. In that case, do we think there are other likely problems with this? Seems like an actual check is the most accurate way to see if the server can be reached. The first thing that comes to mind is that I should probably look for a specific response, otherwise a proxy server returning an error would also appear to be a success.

- Ian

[Andrew W. Donoho]

On May 16, 2014, at 11:11 , Ian Ragsdale <ian.ra...@gmail.com> wrote:

> Thanks, Andrew, I'd hope by now you could give me a bit more credit than that. :)
>
> The logs from CBL are quite clear. The reachability check is failing, but the server is definitely reachable using Safari - I can hit URLs on it and see the expected response. CBL is very clearly not ever attempting a connection due to the reachability check returning a failure. So, I am quite positive that in this situation, a negative response from SCNetworkReachability is NOT a clear indication that a successful request cannot be made, because the proxy server will allow it to succeed.
>
> I do not particularly care WHY it reports a failure, nor was I attempting to posit a particular reason. In this environment, the server actually cannot be resolved by DNS, but again, that is irrelevant because it can connect to the proxy server, and only the proxy server needs to resolve the server.
>
> So, I'm quite positive that when a proxy server is involved, SCNetworkReachability does not accurately predict whether or not a successful request to the server can be made. So, lets assume for the moment that I'm not an idiot and that I've diagnosed the issue correctly. In that case, do we think there are other likely problems with this? Seems like an actual check is the most accurate way to see if the server can be reached. The first thing that comes to mind is that I should probably look for a specific response, otherwise a proxy server returning an error would also appear to be a success.

Ian,



You've obviously taken offense. I cast no aspersions on your character but did attempt to change what appears to be a misunderstanding of how reachability functions. Please don't attribute malice to pedantry.

The evidence you present is fully consistent with my interpretation of how reachability works. It will never bring up the radios when you call it. It really only checks their status after they are on and reports when they are turned off or change state.

Now, CBL may not choose to bring up the radios in response to a reachability. That is their choice. And the fix is pretty trivial. Just touch the server with NSURLConnection.

[Ian Ragsdale]

Sorry, Andrew, sometimes I take offense when I feel like someone is implying I haven't done my due diligence. But I'm always happy to have my assumptions challenged. :)

I'm also not at all challenging your interpretation of how reachability works. In this case, I'm quite sure the radios are already active, but the reachability check will fail either way, because of the DNS check you mentioned - I had already tested that a DNS lookup for our host does in fact fail. Nonetheless, a request through the proxy will succeed, because the client doesn't actually contact our server, but hits the proxy instead.

It's a good point that actually making a request will activate the radios even if they would not otherwise be active, and that's the kind of feedback I was looking for. Sounds like the right thing to do is probably to do a check for reachability on the zero address (as Apple does in their sample code for reachabilityForInternetConnection), which should tell us if the radio is active, and then if the radio is active, we can then make a request to verify that the server is actually reachable. That way, we won't unnecessarily tax the battery.

Any other problems you can think of with this approach? Jens, is this a fix you'd consider pulling in if I submitted a pull request?

- Ian

[Andrew W. Donoho]

On May 16, 2014, at 13:00 , Ian Ragsdale <ian.ra...@gmail.com> wrote:

Sounds like the right thing to do is probably to do a check for reachability on the zero address (as Apple does in their sample code for reachabilityForInternetConnection), which should tell us if the radio is active, and then if the radio is active, we can then make a request to verify that the server is actually reachable. That way, we won't unnecessarily tax the battery.

Ian,

I've had a long conversation with Kevin, the author of Apple's Reachability code. (BTW, I have substantially improved it and added a few features, such as touching the of hosts, etc.) I have a blog post to that effect.

Kevin would advise you to not listen on 0.0.0.0. You should listen on the address of a real host that you really want to connect with. (The fact that your DNS is frakked is a different question.) Checking for reachability on any address tells you nothing if you haven't tried to bring up the radios. You must attempt to bring them up first and then check reachability. The check alone does nothing of value.

[Ian Ragsdale]

Hmm, that's quite annoying, as their current sample code does that exact thing: https://developer.apple.com/Library/ios/samplecode/Reachability/Listings/Reachability_Reachability_m.html.

Do you know _why_ he suggests not to do that?

In this case, I think that not bringing up the radios is the exact thing I WANT to accomplish. I'd suggest that CBL would probably NOT want to activate the radios on it's own, but continue to check opportunistically. Making sure that the radios up should be in charge of the app developer. I just would like CBL to actually try to sync in an environment where Reachability says no, but the existence of a proxy would allow it to work anyway.

However, perhaps I'm thinking too generally. Perhaps a simple flag for the library that tells it that I will handle checking for reachability or a pluggable implementation for CBLReachability would make more sense? I'm sure this is a bit of an esoteric case, and the problem I'm solving here will not be one for the vast majority of people. Or I can just continue to make a fork, I just prefer to share improvements back when possible.

- Ian

[Ian Ragsdale]

Now that I've had a chance to spend some time trying to resolve the issue, I've got a working patch, so I thought I'd revisit the issue in case anybody was interested. I had to do two things to get CBL to function behind a proxy without external DNS access.

First, I had to modify the Reachability check in order to allow it to attempt the sync in the first place. I'm still undecided about what the correct thing to do is, so for the moment I've modified my copy to use "localhost" instead of the actual hostname of the remote server. I'm sure this is not optimal, but for the moment it allows the sync process to begin. Having put a bit more thought into it, I think the most correct thing to do is probably to check the proxy settings and use the proxy server for the reachability check instead of the sync host if one is defined, but the current check seems to work OK for now.

Second, I had to change the way the CBLSocketChangeTracker class was configuring its proxy settings. The existing version looks like it should work correctly (it's hard to say given the sparse documentation of the CFHTTPStream stuff), but in my testing it does not correctly use the proxy. Instead, a DNS error is thrown upon attempting to connect. Passing in the entire settings dictionary returned from CFNetworkCopySystemProxySettings does work correctly for my use case, so it appears that it is not necessary to manually call CFNetworkCopyProxiesForURL to before passing in the settings. Patch here:

https://github.com/boxer/couchbase-lite-ios/commit/8213589b779e5c41b0c490f7e8611b92ecd53463

We'll be testing this out in our customer's environment for the next few weeks, will report back if we run into any troubles with this solution. Don't know if it makes sense for the general case, but for us it resolved a complete inability to sync in this environment.

Andrew, thanks again for your clarifications re: the reachability library.

- Ian

[Jens Alfke]

Hi guys, sorry not to chime in earlier but I was out of town all weekend, and then immediately out sick with a cold :(

My understanding of the problem is:

• The device is on a WiFi network that’s firewalled so no outside hosts are reachable (the router won’t route to any external address ranges.)
• There is an HTTP proxy server on this network that can be used to make requests of outside servers.
• The device is aware of the proxy, as evidenced by being able to use Safari to view outside sites.
• But Couchbase Lite thinks the remote db server is unreachable and won’t try to replicate.

It does sound as though SCNetworkReachability is ignoring the system proxy settings, which is surprising to me.  But maybe that’s because it doesn’t know what particular protocol I’m going to use to contact the host, so it can’t assume that an HTTP-only proxy would provide access?

It sounds like the solution is to have my reachability code check the system proxy settings and check the reachability of the proxy server instead if there is one. The confusing aspect of this is that the proxy settings themselves are probably location-based, since the device can move between LANs with different proxies, so I probably need a second layer of notification to find out when the system proxy settings change...

—Jens

[Ian Ragsdale]

Hi Jens, hope you're feeling better.

That is an excellent summary of the issue, leaving out the change I had to make in order for it to correctly take advantage of the proxy settings (see the patch I linked in a separate email). I also think your proposed solution sounds like the correct one, and one I was planning on attempting to implement if I get a bit more time.

One thing to keep in mind that I learned about when I was debugging is the kCFProxyTypeAutoConfigurationURL type, which isn't a specific proxy host but a URL to a file containing a JavaScript function that will return the proper proxy settings for any given URL. So, when examining the proxy settings I think you need to check for either a URL under the kCFProxyTypeAutoConfigurationURLKey or a hostname under the kCFProxyHostNameKey, and then choose one to use for the reachability check. They certainly don't make this simple. :/

- Ian