CFStreamCopyRunLoopsAndModes Vulverability

[Todd Orler]

Hi,

I'm passing along an investigation from a colleague, since I've posted here before.  I can manage the liaison work.

Jens, please check out the links.  Does this ring any bells?

Thanks for your time.

-Todd

---------------------------

We are getting regular crash reports for our iOS app, with no reproducibility. The common thread is that all of them have this callstack for the crashed thread:

CoreFoundation	__CFTypeCollectionRetain + 101
2	CoreFoundation	__CFArrayCreateCopy0 + 380
3	CoreFoundation	_CFWriteStreamCopyRunLoopsAndModes + 48
4	CoreFoundation	boundPairReadClose + 40
5	CoreFoundation	_CFStreamClose + 76
6	CFNetwork	CoreStreamBase::_streamInterface_Close() + 52
7	CFNetwork	HTTPNetStreamInfo::closeRequestResources() + 82
8	CFNetwork	HTTPNetConnection::transmitRequest(HTTPNetStreamInfo*, __CoreWriteStream*, CFStreamError*, unsigned char) + 1370
9	CFNetwork	HTTPNetStreamInfo::_httpRequestPayloadCallBack(__CoreReadStream*, unsigned long, void*) + 122
10	CFNetwork	CoreReadStreamClient::coreStreamEventsAvailable(unsigned long) + 36
11	CFNetwork	CoreStreamBase::_callClientNow(CoreStreamClient*) + 42
12	CFNetwork	CoreStreamBase::_streamSetEventAndScheduleDelivery(unsigned long, unsigned char) + 122
13	CFNetwork	CoreStreamBase::_streamInterface_SignalEvent(unsigned long, CFStreamError const*) + 34
14	CFNetwork	CoreReadStreamFromCFReadStream::_readStreamClientCallBack(__CFReadStream*, unsigned long, void*) + 76
15	CoreFoundation	_signalEventSync + 118
16	CoreFoundation	_cfstream_shared_signalEventSync + 334

There is no consistency as to what is going on on any other thread at the time of the crash.

With a bunch of Googling, I found this thread in the AFNetworking project which exhibits exactly the same randomness and callstack:

https://github.com/AFNetworking/AFNetworking/issues/907

The discussion contains an analysis of a weakness in Apple's implementation of CFStreamCopyRunLoopsAndModes in CFNetwork. They also came up with a solution (a hack really) which appears to work, detailed here:

https://github.com/AFNetworking/AFNetworking/pull/935

So I went through all of the code in our project looking for a similar pattern, and the only one I came up with was in CBLMultiStreamWriter.m. The close() function follows the same pattern and possibly is exposed to the same weakness on iOS.

This is an extremely speculative set of logical leaps, I admit. Without repro steps it is hard to even know if we have solved it. I can confirm that from user descriptions of what they were doing at the time of the crash, those things would involve CBLMultiStreamWriter::close being called.

We are going to try out duplicating the fix from AFNetworking to see if the crashes stop being reported (we currently get multiple reports a day of this crash). If we run for a week without getting more reports, then maybe we have learned something. But I also wanted to raise the issue here, to see if anyone else has seen in or has other input. It is also possible that Apple may fix the threading hole in iOS 8, but we don't have enough use there yet to know (we have not yet caught a crash here in iOS8, but that isn't terribly meaningful yet).

[Jens Alfke]

Thanks, Todd — that looks like a pretty serious issue, and I think your colleague is correct in connecting the dots between that AFNetworking crash and what Couchbase Lite is doing. I think they and we both use "bound pairs" of streams, because that's the way to create an HTTP request whose body is an output stream you can write to. I've filed an issue in CBL-iOS, #424.

Sadly no one involved in AFNetworking seems to have filed a bug report with Apple, or contacted Apple DTS. (Or if they did, they didn't add anything about it to the thread.) I've posted a question to the macnetworkprog mailing list, and if it's not a known issue I'll file a Radar.

—Jens

[Todd Orler]

Thanks for the quick response, and awesome support, as always, Jens!  We will let you know if our local fix (hack) causes the crash reports to stop coming.  And please let us know if you hear anything from Apple on this.

Thanks,

-Todd